AWS Lambda and some other virtualized environment may not permit access
to /dev/shm on Linux and as such, trying to use ProcessPoolExecutor will
fail.
As using parallelism is only a 'nice to have' feature of black, if it fails
we gracefully fallback to a monoprocess implementation, which permits black
to finish normally.
Co-authored-by: Allan Simon <asimon@yolaw.fr>
* Re-indent the contents of docstrings when indentation changes
Keeping the contents of docstrings completely unchanged when
re-indenting (from 2-space intents to 4, for example) can cause
incorrect docstring indentation:
```
class MyClass:
"""Multiline
class docstring
"""
def method(self):
"""Multiline
method docstring
"""
pass
```
...becomes:
```
class MyClass:
"""Multiline
class docstring
"""
def method(self):
"""Multiline
method docstring
"""
pass
```
This uses the PEP 257 algorithm for determining docstring indentation,
and adjusts the contents of docstrings to match their new indentation
after `black` is applied.
A small normalization is necessary to `assert_equivalent` because the
trees are technically no longer precisely equivalent -- some constant
strings have changed. When comparing two ASTs, whitespace after
newlines within constant strings is thus folded into a single space.
Co-authored-by: Luka Zakrajšek <luka@bancek.net>
* Extract the inner `_v` method to decrease complexity
This reduces the cyclomatic complexity to a level that makes flake8 happy.
* Blacken blib2to3's docstring which had an over-indent
Co-authored-by: Luka Zakrajšek <luka@bancek.net>
Co-authored-by: Zsolt Dollenstein <zsol.zsol@gmail.com>
This pull request's main intention is to wraps long strings (as requested by #182); however, it also provides better string handling in general and, in doing so, closes the following issues:
Closes#26Closes#182Closes#933Closes#1183Closes#1243
Fixes#1360, where an invalid config file causes a return/exit code of 1. This
change means this case is caught earlier, treated like any other bad
parameters, and results in an exit code of 2.
Co-authored-by: Toby Fleming <tobywf@users.noreply.github.com>
`split("\n")` includes a final empty element `""` if the final line
ends with `\n` (as it should for POSIX-compliant text files), which
then became an extra `"\n"`.
`splitlines()` solves that, but there's a caveat, as it will split
on other types of line breaks too (like `\r`), which may not be
desired.
Fixes#526.
`type: ignore` shouldn't block collapsing a line, since it will still
apply fine to the merged line. This prevents an issue where a reformat
causes it to shift lines and then be merged on a subsequent pass.
There is a downside to this, which is that it can cause a `type:
ignore` to apply to more code than was originally intended. There
might be a way to apply this in a more limited situation, but I'm not
sure what it is.
Fixes#1061.
* Make most of blib2to3 directly typed and mypyc-compatible
This used a combination of retype and pytype's merge-pyi to do the
initial merges of the stubs, which then required manual tweaking to
make actually typecheck and work with mypyc.
Co-authored-by: Sanjit Kalapatapu <sanjitkal@gmail.com>
Co-authored-by: Michael J. Sullivan <sully@msully.net>
* Make black able to compile and run with mypyc
The changes made fall into a couple categories:
* Fixing actual type mistakes that slip through the cracks
* Working around a couple mypy bugs (the most annoying of which being
that we need to add type annotations in a number of places where
variables are initialized to None)
Co-authored-by: Sanjit Kalapatapu <sanjitkal@gmail.com>
Co-authored-by: Michael J. Sullivan <sully@msully.net>
The old behavior would detect the existence of a `# fmt: on` in a leaf
node's comment prefix and immediately mark the node as formatting-on,
even if a subsequent `# fmt: off` in the same comment prefix would turn
it back off. This change modifies that logic to track the state through
the entire prefix and take the final state.
Note that this does not fully solve on/off behavior, since any _comment_
lines between the off/on are still formatted. We may need to add
virtual leaf nodes to truly solve that. I will leave that for a separate
commit/PR.
Fixes#1005
Fixes#1042 (and probably #1044 which looks like the same thing).
The issue with the "obviously unnecessary" parentheses that #850 removed is that sometimes they're necessary to help Black fit something in one line. I didn't see an obvious solution that still removes the parens #850 was intended to remove, so let's back out this change for now in the interest of unblocking a release.
This PR also adds a test adapted from the failing example in #1042, so that if we try to reapply the #850 change we don't break the same case again.
* Blacken .py files in blib2to3
This is in preparation for adding type annotations to blib2to3 in
order to compiling it with mypyc (#1009, which I can rebase on top of
this).
To enforce that it stays blackened, I just cargo-culted the existing
test code used for validating formatting. It feels pretty clunky now,
though, so I can abstract the common logic out into a helper if that
seems better. (But error messages might be less clear then?)
* Tidy up the tests
It turns out we also need to handle invisible *left* parens added at
the *start* of a line. Refactor `contains_unsplittable_type_ignore` to
handle this more cleanly.
* add test for special unicode symbol which usual re can not process correctly
add regex lib which supports unicode 12.1.0 standard
replace re usage in project in favor to regex
* #455 fix dependency
In #1040 I had convinced myself that the type ignore logic didn't
need anything like the ignored_ids from the type comment logic, but I
was wrong, and we do.
We hit these cases in practice a bunch.
The code introduced in #1027 to detect whether a type comment appeared
after a regular comment in a Line would spuriously misfire when a leaf
was in the comments dict but had an empty list of comments. This can
occur as an artifact of how comments on trailing commas are handled,
it seems.
(This was discovered trying to test black out on mypy.)
Type comments only apply if they are the first comment on the line,
which means that allowing them to be pushed behind a regular comment
when joining lines is a semantic change (and, indeed, one that black
catches and fails on).
* Parse `:=` properly
* never unwrap parenthesis around `:=`
* When checking for AST-equivalence, use `ast` instead of `typed-ast` when running on python >=3.8
* Assume code that uses `:=` is at least 3.8
Modified maybe_remove_trailing_comma to remove trailing commas for
typedarglists (in addition to arglists), and updated line split logic
to ensure that all lines in a function definition that contain only one
arg have a trailing comma.
Based on the feedback in
https://github.com/python/black/pull/845#issuecomment-490622711
- Remove TokenizerConfig, and add a field to Grammar instead.
- Pass the Grammar to the tokenizer.
- Rename `ASYNC_IS_RESERVED_KEYWORD` to `ASYNC_KEYWORDS` and
`ASYNC_IS_VALID_IDENTIFIER` to `ASYNC_IDENTIFIERS`.
Fixes#593
I looked into this bug with @ambv and @carljm, and we reached the
conclusion was that it's not possible for the tokenizer to determine if
async/await is a keyword inside all possible generators without breaking
the grammar for older versions of Python.
Instead, we introduce a new tokenizer mode for Python 3.7+ that will
cause all async/await instances to get parsed as a reserved keyword,
which should fix async/await inside generators.
... to stop it from spamming the log when black is used as a library in another
python application.
When used indirectly by black the logger initiated in `driver.py` will emit
thousands of debug messages making the debug level of the root logger virtually
useless. By getting a named logger instead the verbosity of logging from this
module can easily be controlled by setting its log level.
Fixes#715
Fixes#305
- Run separate jobs for mypy, self-formatting, flake8, and test runs.
- Don't run flake8 in 3.8 because it is broken (and we can't really expect flake8 to always keep up with 3.8 development).
- Fix unused variable in test
This is a new syntax added in python3.7, so black can't verify that reformatting will not change the ast unless black itself is run with 3.7. We'll need to change the error message black gives in this case. @ambv any ideas?
Fixes#125.
Fixes#452
I ended up making a couple of other normalizations to numeric literals
too (lowercase everything, don't allow leading or trailing . in floats,
remove redundant + sign in exponent). I don't care too much about those,
so I'm happy to change the behavior there.
For reference, here is Python's grammar for numeric literals:
https://docs.python.org/3/reference/lexical_analysis.html#numeric-literals
