* Blacken .py files in blib2to3
This is in preparation for adding type annotations to blib2to3 in
order to compiling it with mypyc (#1009, which I can rebase on top of
this).
To enforce that it stays blackened, I just cargo-culted the existing
test code used for validating formatting. It feels pretty clunky now,
though, so I can abstract the common logic out into a helper if that
seems better. (But error messages might be less clear then?)
* Tidy up the tests
* add test for special unicode symbol which usual re can not process correctly
add regex lib which supports unicode 12.1.0 standard
replace re usage in project in favor to regex
* #455 fix dependency
mypy 0.730 fixed a bug involving nonexistent attributes accessed on
modules, which caused an error since COLONEQUAL never got added to
token.pyi. Add it.
* Parse `:=` properly
* never unwrap parenthesis around `:=`
* When checking for AST-equivalence, use `ast` instead of `typed-ast` when running on python >=3.8
* Assume code that uses `:=` is at least 3.8
Based on the feedback in
https://github.com/python/black/pull/845#issuecomment-490622711
- Remove TokenizerConfig, and add a field to Grammar instead.
- Pass the Grammar to the tokenizer.
- Rename `ASYNC_IS_RESERVED_KEYWORD` to `ASYNC_KEYWORDS` and
`ASYNC_IS_VALID_IDENTIFIER` to `ASYNC_IDENTIFIERS`.
Fixes#593
I looked into this bug with @ambv and @carljm, and we reached the
conclusion was that it's not possible for the tokenizer to determine if
async/await is a keyword inside all possible generators without breaking
the grammar for older versions of Python.
Instead, we introduce a new tokenizer mode for Python 3.7+ that will
cause all async/await instances to get parsed as a reserved keyword,
which should fix async/await inside generators.
... to stop it from spamming the log when black is used as a library in another
python application.
When used indirectly by black the logger initiated in `driver.py` will emit
thousands of debug messages making the debug level of the root logger virtually
useless. By getting a named logger instead the verbosity of logging from this
module can easily be controlled by setting its log level.
Fixes#715
This is a new syntax added in python3.7, so black can't verify that reformatting will not change the ast unless black itself is run with 3.7. We'll need to change the error message black gives in this case. @ambv any ideas?
Fixes#125.
There is no need to keep the pickled grammar files in git. PR #203 will
move them into a user-specific cache directory any way.
See: https://github.com/ambv/black/issues/192
Signed-off-by: Christian Heimes <christian@python.org>
Since black.py is not marked as executable, the shebang in black.py serves
no purpose. black should be invoked through its entry point any way.
token.py is an internal module without a __name__ == '__main__' block or
other executable code. It contains just list of constants and small
helper functions.
Signed-off-by: Christian Heimes <christian@python.org>
The blib2to3/ code is PSF licensed as that is where the code originated.
This change just drops a proper copy of that license file into the
directory tree to make that clear.
This support isn't *exactly* right per PEP 3131 as the regex engine is a bit
too limited for that and I didn't want to spend time on Other_ID_Start and
Other_ID_Continue unless they're actually needed.
Hopefully this doesn't slow it down too much.
