* Make most of blib2to3 directly typed and mypyc-compatible
This used a combination of retype and pytype's merge-pyi to do the
initial merges of the stubs, which then required manual tweaking to
make actually typecheck and work with mypyc.
Co-authored-by: Sanjit Kalapatapu <sanjitkal@gmail.com>
Co-authored-by: Michael J. Sullivan <sully@msully.net>
* Make black able to compile and run with mypyc
The changes made fall into a couple categories:
* Fixing actual type mistakes that slip through the cracks
* Working around a couple mypy bugs (the most annoying of which being
that we need to add type annotations in a number of places where
variables are initialized to None)
Co-authored-by: Sanjit Kalapatapu <sanjitkal@gmail.com>
Co-authored-by: Michael J. Sullivan <sully@msully.net>
The old behavior would detect the existence of a `# fmt: on` in a leaf
node's comment prefix and immediately mark the node as formatting-on,
even if a subsequent `# fmt: off` in the same comment prefix would turn
it back off. This change modifies that logic to track the state through
the entire prefix and take the final state.
Note that this does not fully solve on/off behavior, since any _comment_
lines between the off/on are still formatted. We may need to add
virtual leaf nodes to truly solve that. I will leave that for a separate
commit/PR.
Fixes#1005
Fixes#1042 (and probably #1044 which looks like the same thing).
The issue with the "obviously unnecessary" parentheses that #850 removed is that sometimes they're necessary to help Black fit something in one line. I didn't see an obvious solution that still removes the parens #850 was intended to remove, so let's back out this change for now in the interest of unblocking a release.
This PR also adds a test adapted from the failing example in #1042, so that if we try to reapply the #850 change we don't break the same case again.
* Blacken .py files in blib2to3
This is in preparation for adding type annotations to blib2to3 in
order to compiling it with mypyc (#1009, which I can rebase on top of
this).
To enforce that it stays blackened, I just cargo-culted the existing
test code used for validating formatting. It feels pretty clunky now,
though, so I can abstract the common logic out into a helper if that
seems better. (But error messages might be less clear then?)
* Tidy up the tests
It turns out we also need to handle invisible *left* parens added at
the *start* of a line. Refactor `contains_unsplittable_type_ignore` to
handle this more cleanly.
* add test for special unicode symbol which usual re can not process correctly
add regex lib which supports unicode 12.1.0 standard
replace re usage in project in favor to regex
* #455 fix dependency
In #1040 I had convinced myself that the type ignore logic didn't
need anything like the ignored_ids from the type comment logic, but I
was wrong, and we do.
We hit these cases in practice a bunch.
The code introduced in #1027 to detect whether a type comment appeared
after a regular comment in a Line would spuriously misfire when a leaf
was in the comments dict but had an empty list of comments. This can
occur as an artifact of how comments on trailing commas are handled,
it seems.
(This was discovered trying to test black out on mypy.)
Type comments only apply if they are the first comment on the line,
which means that allowing them to be pushed behind a regular comment
when joining lines is a semantic change (and, indeed, one that black
catches and fails on).
* Parse `:=` properly
* never unwrap parenthesis around `:=`
* When checking for AST-equivalence, use `ast` instead of `typed-ast` when running on python >=3.8
* Assume code that uses `:=` is at least 3.8
Modified maybe_remove_trailing_comma to remove trailing commas for
typedarglists (in addition to arglists), and updated line split logic
to ensure that all lines in a function definition that contain only one
arg have a trailing comma.
Based on the feedback in
https://github.com/python/black/pull/845#issuecomment-490622711
- Remove TokenizerConfig, and add a field to Grammar instead.
- Pass the Grammar to the tokenizer.
- Rename `ASYNC_IS_RESERVED_KEYWORD` to `ASYNC_KEYWORDS` and
`ASYNC_IS_VALID_IDENTIFIER` to `ASYNC_IDENTIFIERS`.
Fixes#593
I looked into this bug with @ambv and @carljm, and we reached the
conclusion was that it's not possible for the tokenizer to determine if
async/await is a keyword inside all possible generators without breaking
the grammar for older versions of Python.
Instead, we introduce a new tokenizer mode for Python 3.7+ that will
cause all async/await instances to get parsed as a reserved keyword,
which should fix async/await inside generators.
... to stop it from spamming the log when black is used as a library in another
python application.
When used indirectly by black the logger initiated in `driver.py` will emit
thousands of debug messages making the debug level of the root logger virtually
useless. By getting a named logger instead the verbosity of logging from this
module can easily be controlled by setting its log level.
Fixes#715
