* Parse `:=` properly
* never unwrap parenthesis around `:=`
* When checking for AST-equivalence, use `ast` instead of `typed-ast` when running on python >=3.8
* Assume code that uses `:=` is at least 3.8
Based on the feedback in
https://github.com/python/black/pull/845#issuecomment-490622711
- Remove TokenizerConfig, and add a field to Grammar instead.
- Pass the Grammar to the tokenizer.
- Rename `ASYNC_IS_RESERVED_KEYWORD` to `ASYNC_KEYWORDS` and
`ASYNC_IS_VALID_IDENTIFIER` to `ASYNC_IDENTIFIERS`.
Fixes#593
I looked into this bug with @ambv and @carljm, and we reached the
conclusion was that it's not possible for the tokenizer to determine if
async/await is a keyword inside all possible generators without breaking
the grammar for older versions of Python.
Instead, we introduce a new tokenizer mode for Python 3.7+ that will
cause all async/await instances to get parsed as a reserved keyword,
which should fix async/await inside generators.
... to stop it from spamming the log when black is used as a library in another
python application.
When used indirectly by black the logger initiated in `driver.py` will emit
thousands of debug messages making the debug level of the root logger virtually
useless. By getting a named logger instead the verbosity of logging from this
module can easily be controlled by setting its log level.
Fixes#715
This is a new syntax added in python3.7, so black can't verify that reformatting will not change the ast unless black itself is run with 3.7. We'll need to change the error message black gives in this case. @ambv any ideas?
Fixes#125.
There is no need to keep the pickled grammar files in git. PR #203 will
move them into a user-specific cache directory any way.
See: https://github.com/ambv/black/issues/192
Signed-off-by: Christian Heimes <christian@python.org>
Since black.py is not marked as executable, the shebang in black.py serves
no purpose. black should be invoked through its entry point any way.
token.py is an internal module without a __name__ == '__main__' block or
other executable code. It contains just list of constants and small
helper functions.
Signed-off-by: Christian Heimes <christian@python.org>
The blib2to3/ code is PSF licensed as that is where the code originated.
This change just drops a proper copy of that license file into the
directory tree to make that clear.
This support isn't *exactly* right per PEP 3131 as the regex engine is a bit
too limited for that and I didn't want to spend time on Other_ID_Start and
Other_ID_Continue unless they're actually needed.
Hopefully this doesn't slow it down too much.
