It was causing stability issues because the first pass
could cause a "magic trailing comma" to appear, meaning
that the second pass might get a different result. It's
not critical.
Some things format differently (with extra parens)
It turns out "simple_stmt" isn't that simple: it can contain multiple
statements separated by semicolons. Invisible parenthesis logic for
arithmetic expressions only looked at the first child of simple_stmt.
This causes instability in the presence of semicolons, since the next
run through the statement following the semicolon will be the first
child of another simple_stmt.
I believe this along with #2572 fix the known stability issues.
Since power operators almost always have the highest binding power in expressions, it's often more readable to hug it with its operands. The main exception to this is when its operands are non-trivial in which case the power operator will not hug, the rule for this is the following:
> For power ops, an operand is considered "simple" if it's only a NAME, numeric CONSTANT, or attribute access (chained attribute access is allowed), with or without a preceding unary operator.
Fixes GH-538.
Closes GH-2095.
diff-shades results: https://gist.github.com/ichard26/ca6c6ad4bd1de5152d95418c8645354b
Co-authored-by: Diego <dpalma@evernote.com>
Co-authored-by: Felix Hildén <felix.hilden@gmail.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Closes#2360: I'd like to make passing SRC or `--code` mandatory and the arguments mutually exclusive. This will change our (partially already broken) promises of CLI behavior, but I'll comment below.
This PR is intended to have no change to semantics.
This is in preparation for #2784 which will likely introduce more logic
that depends on `current_line.depth`.
Inlining the subtraction gets rid of offsetting and makes it much easier
to see what the result will be.
Fixes#2506
``XDG_CACHE_HOME`` does not work on Windows. To allow for users to set a custom cache directory on all systems I added a new environment variable ``BLACK_CACHE_DIR`` to set the cache directory. The default remains the same so users will only notice a change if that environment variable is set.
The specific use case I have for this is I need to run black on in different processes at the same time. There is a race condition with the cache pickle file that made this rather difficult. A custom cache directory will remove the race condition.
I created ``get_cache_dir`` function in order to test the logic. This is only used to set the ``CACHE_DIR`` constant.
Fixes#2742.
This PR adds the ability to configure additional python cell magics. This
will allow formatting cells in Jupyter Notebooks that are using custom (python)
magics.
Black would now echo the location that it determined as the root path
for the project if `--verbose` is enabled by the user, according to
which it chooses the SRC paths, i.e. the absolute path of the project
is `{root}/{src}`.
Closes#1880
*blib2to3's support was left untouched because: 1) I don't want to touch
parsing machinery, and 2) it'll allow us to provide a more useful error
message if someone does try to format Python 2 code.
error: cannot format <string>: ('EOF in multi-line statement', (2, 0))
▲ before ▼ after
error: cannot format <string>: Cannot parse: 2:0: EOF in multi-line statement
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
We were no longer using it since GH-2644 and GH-2654. This should hopefully
make using Black easier to use as there's one less compiled dependency.
The core team also doesn't have to deal with the surprisingly frequent fires
the regex packaging setup goes through.
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
* Treat functions/classes in blocks as if they're nested
One curveball is that we still want two preceding newlines before blocks
that are probably logically disconnected. In other words:
if condition:
def foo():
return "hi"
# <- aside: this is the goal of this commit
else:
def foo():
return "cya"
# <- the two newlines spacing here should stay
# since this probably isn't related
with open("db.json", encoding="utf-8") as f:
data = f.read()
Unfortunately that means we have to special case specific clause types
instead of just being able to just for a colon leaf. The hack used here
is to check whether we're adding preceding newlines for a standalone or
dependent clause. "Standalone" being a clause that doesn't need another
clause to be valid (eg. if) and vice versa.
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
This removes all but one usage of the `regex` dependency. Tricky bits included:
- A bug in test_black.py where we were incorrectly using a character range. Fix also submitted separately in #2643.
- `tokenize.py` was the original use case for regex (#1047). The important bit is that we rely on `\w` to match anything valid in an identifier, and `re` fails to match a few characters as part of identifiers. My solution is to instead match all characters *except* those we know to mean something else in Python: whitespace and ASCII punctuation. This will make Black able to parse some invalid Python programs, like those that contain non-ASCII punctuation in the place of an identifier, but that seems fine to me.
- One import of `regex` remains, in `trans.py`. We use a recursive regex to parse f-strings, and only `regex` supports that. I haven't thought of a better fix there (except maybe writing a manual parser), so I'm leaving that for now.
My goal is to remove the `regex` dependency to reduce the risk of breakage due to dependencies and make life easier for users on platforms without wheels.
Fixes https://github.com/psf/black/issues/2627 , a non-Python cell magic such as `%%writeline` can legitimately contain "incorrect" indentation, however this causes `tokenize-rt` to return an error. To avoid this, `validate_cell` should early detect cell magics (just like it detects `TransformerManager` transformations).
Test added too, in the shape of a "badly indented" `%%writefile` within `test_non_python_magics`.
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Marco Edward Gorelli <marcogorelli@protonmail.com>
In Python 3.10 the exception generated by creating a process pool on
a Python build that doesn't support this is now `NotImplementedError`
Commit history before merge:
* Fix process pool fallback on Python 3.10
* Update CHANGES.md
* Update CHANGES.md
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
The implementation of the new backtracking logic depends heavily on deepcopying the current state of the parser before seeing one of the new keywords, which by default is an very expensive operations. On my system, formatting these 3 files takes 1.3 seconds.
```
$ touch tests/data/pattern_matching_*; time python -m black -tpy310 tests/data/pattern_matching_* 19ms
All done! ✨🍰✨
3 files left unchanged.
python -m black -tpy310 tests/data/pattern_matching_* 2,09s user 0,04s system 157% cpu 1,357 total
```
which can be optimized 3X if we integrate the existing copying logic (`clone`) to the deepcopy system;
```
$ touch tests/data/pattern_matching_*; time python -m black -tpy310 tests/data/pattern_matching_* 1ms
All done! ✨🍰✨
3 files left unchanged.
python -m black -tpy310 tests/data/pattern_matching_* 0,66s user 0,02s system 147% cpu 0,464 total
```
This still might have some potential, but that would be way trickier than this initial patch.
* Improve Python 2 only syntax detection
First of all this fixes a mistake I made in Python 2 deprecation PR
using token.* to check for print/exec statements. Turns out that
for nodes with a type value higher than 256 its numeric type isn't
guaranteed to be constant. Using syms.* instead fixes this.
Also add support for the following cases:
print "hello, world!"
exec "print('hello, world!')"
def set_position((x, y), value):
pass
try:
pass
except Exception, err:
pass
raise RuntimeError, "I feel like crashing today :p"
`wow_these_really_did_exist`
10L
* Add octal support, more test cases, and fixup long ints
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
`DEPRECATION: Python 2 support will be removed in the first stable releaseexpected in January 2022` - > `DEPRECATION: Python 2 support will be removed in the first stable release expected in January 2022`
* Prepare for Python 2 depreciation
- Use BlackRunner and .stdout in command line test
So the next commit won't break this test. This is in its own commit so
we can just revert the depreciation commit when dropping Python 2
support completely.
* Deprecate Python 2 formatting support
If the individual failures are verbose, it's useful to have
the summary at the end. Otherwise, it can be really difficult
to figure out which projects have an issue.
* re-implement simple CORS middleware for blackd
* remove aiohttp-cors from setup.py
* Remove aiohttp-cors from Pipfile.lock
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
re. import, the ipynb code was assuming that typing-extensions would
always be available, but that's not the case! There's an environment
marker on the requirement meaning it won't get installed on 3.10 or
higher. The test suite didn't catch this issue since aiohttp pulls in
typing-extensions unconditionally.
This also introduces a script so we can reference the latest version in
the example pre-commit configuration in the docs without forgetting to
update it when doing a release!
Commit history before merge:
* document jupyter hook
* note minimum version
* add check for pre-commit version
* use git tag
* curl api during ci
* parse version from changes file
* fixup script
* rename variables
* Tweak the docs & magical script
* fix couple of typos
* pin additional dependencies in hook
* Add types-PyYAML to lockfile
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
Implementation stolen from PR davidhalter/parso#162. Thanks parso!
I could add support for these newer syntactical constructs in the
target version detection logic, but until I get diff-shades up
and running I don't feel very comfortable adding the code.
This fixes a bug where a trailing comma would be added to a
parenthesized return annotation changing its type to a tuple.
Here's one case where this bug shows up:
```
def spam() -> (
this_is_a_long_type_annotation_which_should_NOT_get_a_trailing_comma
):
pass
```
The root problem was that the type annotation was treated as if it was
a parameter & import list (is_body=True to linegen::bracket_split_build_line)
where a trailing comma is usually fine. Now there's another check in the
aforementioned function to make sure the body it's operating on isn't
a return annotation before truly adding a trailing comma.
* Add CPython repository into primer runs
- CPython tests is probably the best repo for black to test on as the stdlib's unittests should use all syntax
- Limit to running in recent versions of the python runtime - e.g. today >= 3.9
- This allows us to parse more syntax
- Exclude all failing files for now
- Definitely have bugs to explore there - Refer to #2407 for more details there
- Some test files on purpose have syntax errors, so we will never be able to parse them
- Add new black command arguments logging in debug mode; very handy for seeing how CLI arguments are formatted
CPython now succeeds ignoring 16 files:
```
Oh no! 💥💔💥
1859 files would be reformatted, 148 files would be left unchanged.
```
Testing
- Ran locally with and without string processing - Very little runtime difference BUT 3 more failed files
```
time /tmp/tb/bin/black --experimental-string-processing --check . 2>&1 | tee /tmp/black_cpython_esp
...
Oh no! 💥💔💥
1859 files would be reformatted, 148 files would be left unchanged, 16 files would fail to reformat.
real 4m8.563s
user 16m21.735s
sys 0m6.000s
```
- Add unittest for new covienence config file flattening that allows long arguments to be broke up into an array/list of strings
Addresses #2407
---
Commit history before merge:
* Add new `timeout_seconds` support into primer.json
- If present, will set forked process limit to that value in seconds
- Otherwise, stay with default 10 minutes (600 seconds)
* Add new "base_path" concept to black-primer
- Rather than start at the repo root start at a configured path within the repository
- e.g. for cpython only run black on `Lib`
* Disable by default - It's too much for GitHub Actions. But let's leave config for others to use
* Minor tweak to _flatten_cli_args
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
The fix for #1688 in #1761 breaks help("modules") introspection and also leads
to unhappy results when inadvertently importing blackd from Python. Basically
the sys.exit(-1) causes the whole Python REPL to exit -- not great to suffice.
Commit history before merge:
* Change sys.exit to Raise.
* Add #2440 to changelog.
* Fix lint error from prettier
* Remove exception chain for more helpful user message.
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
we don't accidentally add backslashes to them when normalizing quotes
because that's invalid syntax!
The problem this commit fixes is that matches would eat too much
blocking important matches to occur. For example, here's one f-string
body:
{a}{b}{c}
I know there's no risk of introducing backslashes here, but the regex
already goes sideways with this. Throwing this example at regex101
I get:
{a}{b}{c} # The As and Bs are the two matches, and the upper
---- ---- # case letters are the groups with those matches.
aAaa bbBb
... we've missed the middle expression (so if any backslashes in a
more complex example were introduced there we wouldn't bail out
even though we should -- hence the bug). As it stands the regex
needs somesort of extra character (or the start/end of the body)
around the expressions but that isn't always the case as shown
above.
The fix implemented here is to turn the "eat a surrounding non-curly
bracket character" groups ie. `(?:[^{]|^)` and `(?:[^}]|$)` into
negative lookaheads and lookbehinds. This still guarantees the
already specified rules but without problematically eating extra
characters ^^
Fixes#2359.
This commit now makes Black exit with an user-friendly error message if a
.gitignore file couldn't be parsed -- a massive improvement over an opaque
traceback!
To summarise, based on what was discussed in that issue:
due to not being able to parse automagics (e.g. pip install black)
without a running IPython kernel, cells with syntax which is parseable
by neither ast.parse nor IPython will be skipped cells with multiline
magics will be skipped trailing semicolons will be preserved, as they
are often put there intentionally in Jupyter Notebooks to suppress
unnecessary output
Commit history before merge (excluding merge commits):
* wip
* fixup tests
* skip tests if no IPython
* install test requirements in ipynb tests
* if --ipynb format all as ipynb
* wip
* add some whole-notebook tests
* docstrings
* skip multiline magics
* add test for nested cell magic
* remove ipynb_test.yml, put ipynb tests in tox.ini
* add changelog entry
* typo
* make token same length as magic it replaces
* only include .ipynb by default if jupyter dependencies are found
* remove logic from const
* fixup
* fixup
* re.compile
* noop
* clear up
* new_src -> dst
* early exit for non-python notebooks
* add non-python test notebook
* add repo with many notebooks to black-primer
* install extra dependencies for black-primer
* fix planetary computer examples url
* dont run on ipynb files by default
* add scikit-lego (Expected to change) to black-primer
* add ipynb-specific diff
* fixup
* run on all (including ipynb) by default
* remove --include .ipynb from scikit-lego black-primer
* use tokenize so as to mirror the exact logic in IPython.core.displayhooks quiet
* fixup
* 🎨
* clarify docstring
* add test for when comment is after trailing semicolon
* enumerate(reversed) instead of [::-1]
* clarify docstrings
* wip
* use jupyter and no_jupyter marks
* use THIS_DIR
* windows fixup
* perform safe check cell-by-cell for ipynb
* only perform safe check in ipynb if not fast
* remove redundant Optional
* 🎨
* use typeguard
* dont process cell containing transformed magic
* require typing extensions before 3.10 so as to have TypeGuard
* use dataclasses
* mention black[jupyter] in docs as well as in README
* add faq
* add message to assertion error
* add test for indented quieted cell
* use tokenize_rt else we cant roundtrip
* fmake fronzet set for tokens to ignore when looking for trailing semicolon
* remove planetary code examples as recent commits result in changes
* use dataclasses which inherit from ast.NodeVisitor
* bump typing-extensions so that TypeGuard is available
* bump typing-extensions in Pipfile
* add test with notebook with empty metadata
* pipenv lock
* deprivative validate_cell
* Update README.md
* Update docs/getting_started.md
* dont cache notebooks if jupyter dependencies arent found
* dont write to cache if jupyter deps are not installed
* add notebook which cant be parsed
* use clirunner
* remove other subprocess calls
* add docstring
* make verbose and quiet keyword only
* 🎨
* run second many test on directory, not on file
* test for warning message when running on directory
* early return from non-python cell magics
* move NothingChanged to report to avoid circular import
* remove circular import
* reinstate --ipynb flag
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
toml unfortunately has a lack of maintainership issue right now. It's
evident by the fact toml only supports TOML v0.5.0. TOML v1.0.0 has
been recently released and right now Black crashes hard on its usage.
tomli is a brand new parse only TOML library. It supports TOML
v1.0.0. Although TBH we're switching to this one mostly because
pip is doing the same.
*The upper bound was included at the library maintainer's request.
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Taneli Hukkinen <3275109+hukkin@users.noreply.github.com>
Commit history before merge:
* Accept empty stdin (close#2337)
* Update tests/test_black.py
* Add changelog
* Assert Black reformats an empty string to an empty string (#2337) (#2346)
* fix
* Add STDIN test to primer
- Check that out STDIN black support stays working
- Add asyncio.subprocess STDIN pip via communicate
- We just check we format python code from primer's `lib.py`
Fixes#2310
`black.strings.get_string_prefix` used to lowercase the extracted
prefix before returning it. This is wrong because 1) it ignores the
fact we should leave R prefixes alone because of MagicPython, and 2)
there is dedicated prefix casing handling code that fixes issue 1.
`.lower` is too naive.
This was originally fixed in 20.8b0, but was reintroduced since 21.4b0.
I also added proper prefix normalization for docstrings by using the
`black.strings.normalize_string_prefix` helper.
Some more test strings were added to make sure strings with capitalized
prefixes aren't treated differently (actually happened with my original
patch, Jelle had to point it out to me).
Not sure the fix is right. Here is what I found: issue is connected
with line
first.prefix = prefix[comment.consumed :]
in `comments.py`. `first.prefix` is a prefix of the line, that ends
with `# fmt: skip`, but `comment.consumed` is the length of the
`" # fmt: skip"` string. If prefix length is greater than 14,
`first.prefix` will grow every time we apply formatting.
Fixes#2254
See if we pass all our repos with experimental string processing enabled.
Django probably needed:
- Ignores >= 3.8 only
We could support PEP440 version specifiers, but that would introduce the packaging module as a dependency that I'd like to avoid ... Or I could implement a poor persons version or vendor
Commit history before merge:
* [primer] Enable everything
* Add exclude extend to django CLI args for primer
* Change default timeout to from 5 to 10 mins for a primer project
* Skip string normalization for Django
* Limit Django to >= 3.8 due to := operator
Closes#1246: This PR adds a new option (and automatically a toml entry, hooray for existing configuration management 🎉) to require a specific version of Black to be running.
For example: `black --required-version 20.8b -c "format = 'this'"`
Execution fails straight away if it doesn't match `__version__`.
PR #2286 did not fix the edge-cases (e.g. when the string is just long
enough to cause a line to be 89 characters long). This PR corrects that
mistake.
Behavior other than output shouldn't depend on the verbose/quiet option. As far as I can tell this currently has no visible effect, since code after this function is called handles an empty list gracefully.
* Add optional uvloop import
- If we find `uvloop` in the env for black, blackd or black-primer lets try and use it
- Add a uvloop extra install
Fixes#2257
Test:
- Add ci job to install black[uvloop] and run a primer run with uvloop
- Only with latest python (3.9)
- Will be handy to compare runtimes as a very unoffical benchmark
* Remove tox install
* Add to CHANGES/news
Resolves#2168 by disabling the insertion of a " " when the docstring is entirely empty.
Note that this PR is focussed only on the case of empty docstrings. In particular this does not make any changes to the behaviour that a " " is inserted if a non-empty docstring begins with the quoting character. That is, black still prefers:
""" "something" """
to:
""""something" """
and that:
""""Something""""
is not a legal docstring.
* Solved Problem with non-alphabetical .gitignore files
When .gitignore file in the user's project directory contained non-alphabetical
characters(Japanese, Korean, Chinese, etc), Nothing works and printed this
weird message in the console('cp949' is the encoding for Korean characters
in this case). It even blocks VSCode's formatting from working. This commit
solves the problem.
Traceback (most recent call last):
File "c:\users\username\anaconda3\envs\project-name\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "c:\users\username\anaconda3\envs\project-name\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\username\anaconda3\envs\project-name\Scripts\black.exe\__main__.py", line 7, in <module>
File "c:\users\username\anaconda3\envs\project-name\lib\site-packages\black\__init__.py", line 1056, in patched_main
main()
File "c:\users\username\anaconda3\envs\project-name\lib\site-packages\click\core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "c:\users\username\anaconda3\envs\project-name\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "c:\users\username\anaconda3\envs\project-name\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "c:\users\username\anaconda3\envs\project-name\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "c:\users\username\anaconda3\envs\project-name\lib\site-packages\click\decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "c:\users\username\anaconda3\envs\project-name\lib\site-packages\black\__init__.py", line 394, in main
stdin_filename=stdin_filename,
File "c:\users\username\anaconda3\envs\project-name\lib\site-packages\black\__init__.py", line 445, in get_sources
gitignore = get_gitignore(root)
File "c:\users\username\anaconda3\envs\project-name\lib\site-packages\black\files.py", line 122, in get_gitignore
lines = gf.readlines()
UnicodeDecodeError: 'cp949' codec can't decode byte 0xb0 in position 13: illegal multibyte sequence
* Made .gitignore File Reader Detect Its Encoding
* Revert "Made .gitignore File Reader Detect Its Encoding"
This reverts commit 6c3a7ea42b5b1e441cc0026c8205d1cee68c1bba.
* Revert "Solved Problem with non-alphabetical .gitignore files"
This reverts commit b0100b5d91c2f5db544a60f34aafab120f0aa458.
* Made .gitignore Reader Open the File with Auto Encoding Detecting
https://docs.python.org/3.8/library/tokenize.html#tokenize.open
* Revert "Made .gitignore Reader Open the File with Auto Encoding Detecting"
This reverts commit 50dd80422938649ccc8c7f43aac752f9f6481779.
* Made .gitignore Reader Use UTF-8
* Updated CHANGES.md for #2229
* Updated CHANGES.md for #2229
* Update CHANGES.md
* Update CHANGES.md
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
Commit history before merge:
Black now respects .gitignore files in all levels, not only root/.gitignore file
(apply .gitignore rules like git does).
* Fix: typo
* Fix: respect .gitignore files in all levels.
* Add: CHANGELOG note.
* Fix: TypeError: unsupported operand type(s) for +: 'NoneType' and 'PathSpec'
* Update docs.
* Fix: no parent .gitignore
* Add a comment since the if expression is a bit hard to understand
* Update tests - conver no parent .gitignore case.
* Use main's Pipfile.lock instead
The original changes in Pipfile.lock are whitespace only. The changes
turned the JSON's file indentation from 4 to 2. Effectively this
happened: `json.dumps(json.loads(old_pipfile_lock), indent=2) + "\n"`.
Just using main's Pipfile.lock instead of undoing the changes because
1) I don't know how to do that easily and quickly, and 2) there's a
merge conflict.
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
* Merge remote-tracking branch 'upstream/main' into i1730 …
conflicts for days ay?
It appears sqlalchemy has recently reformatted their project with
Black 21.5b1.
Most of our dependencies have a lower bound and creating a test
environment with the oldest acceptable dependencies runs the full
Black test suite just fine. The only exception to this is aiohttp-cors.
It's unbounded and the oldest version 0.1.0 until 0.4.0 breaks the
test suite in such an old environment.
Failure with 0.1.0:
```
tests/test_blackd.py:10: in <module>
import blackd
testenv/lib/python3.8/site-packages/blackd/__init__.py:12: in <module>
import aiohttp_cors
testenv/lib/python3.8/site-packages/aiohttp_cors/__init__.py:29: in <module>
from .urldispatcher_router_adapter import UrlDistatcherRouterAdapter
testenv/lib/python3.8/site-packages/aiohttp_cors/urldispatcher_router_adapter.py:27: in <module>
class UrlDistatcherRouterAdapter(RouterAdapter):
testenv/lib/python3.8/site-packages/aiohttp_cors/urldispatcher_router_adapter.py:32: in UrlDistatcherRouterAdapter
def route_methods(self, route: web.Route):
E AttributeError: module 'aiohttp.web' has no attribute 'Route'
```
For 0.2.0:
```
tests/test_blackd.py:10: in <module>
import blackd
testenv/lib/python3.8/site-packages/blackd/__init__.py:12: in <module>
import aiohttp_cors
testenv/lib/python3.8/site-packages/aiohttp_cors/__init__.py:27: in <module>
from .cors_config import CorsConfig
testenv/lib/python3.8/site-packages/aiohttp_cors/cors_config.py:24: in <module>
from .urldispatcher_router_adapter import UrlDistatcherRouterAdapter
testenv/lib/python3.8/site-packages/aiohttp_cors/urldispatcher_router_adapter.py:27: in <module>
class UrlDistatcherRouterAdapter(AbstractRouterAdapter):
testenv/lib/python3.8/site-packages/aiohttp_cors/urldispatcher_router_adapter.py:32: in UrlDistatcherRouterAdapter
def route_methods(self, route: web.Route):
E AttributeError: module 'aiohttp.web' has no attribute 'Route'
```
For 0.3.0:
```
ERROR: Cannot install aiohttp-cors==0.3.0 and aiohttp==3.6.0 because these package versions have conflicting dependencies.
The conflict is caused by:
The user requested aiohttp==3.6.0
aiohttp-cors 0.3.0 depends on aiohttp<=0.20.2 and >=0.18.0
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
```
* Move string-related utility to functions to strings.py, const.py
* Move Leaf/Node-related functionality to nodes.py
* Move comment-related functions to comments.py
* Move caching to cache.py and Mode/TargetVersion/Feature to mode.py
* Move some leftover functions to nodes.py, comments.py, strings.py
* Add missing files to source list for test runs
* Move line-related functionality into lines.py, brackets into brackets.py
* Move transformers to trans.py
* Move file handling, output, parsing, concurrency, debug, and report
* Move two more functions to nodes.py
* Add CHANGES
* Add numeric.py
* Add linegen.py
* More docstrings
* Include new files in tests
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Closes#2164.
Changes behavior of how .gitignore is handled. With this change, the rules in .gitignore are only used as a fallback if no exclusion rule is explicitly passed on the command line or in pyproject.toml. Previously they were used regardless if explicit exclusion rules were specified, preventing any overriding of .gitignore rules.
Those that depend only on .gitignore for their exclusion rules will not be affected. Those that use both .gitignore and exclude will find that exclude will act more like actually specifying exclude and not just another extra-excludes. If the previous behavior was desired, they should move their rules from exclude to extra-excludes.
* Enable ` --experimental-string-processing` on all primer projects
- We want to make this default so need to test it more
- Fixed splat/star bug in extending black args for each project
* Disable sqlalchemy due to crash
Previously the RELAXED_DECORATOR detection would be falsely True on that
example. The problem was that an argument-less parentheses pair didn't
pass the `is_simple_decorator_trailer` check even it should. OTOH a
parentheses pair containing an argument or more passed as expected.
- Allow runs with no code diff output
- This is handy for reducing output to see which file is erroring
Test:
- Edit config for 'channels' to expect no changes and run with `--no-diff` and see no diff output
- It no longer crashes black so we should test on it's code
- Update django reason to name the file causing error
- Seems it has a syntax error on purpose
Such shebangs are only ever used if the file is executed directly, i.e.:
$ /usr/lib/python3.9/site-packages/black_primer/cli.py
But that doesn't work:
$ /usr/lib/python3.9/site-packages/black_primer/cli.py
bash: /usr/lib/python3.9/site-packages/black_primer/cli.py: Permission denied
The lib file even has: "lib is a library, funnily enough"
We're only fixing them so fuzzers don't yell at us when we break "valid"
code. I mean "valid" because some of the examples aren't even accepted by
Python.
Black would previously strip the parenthesis away from statements like this these ones:
assert (spam := 12 + 1)
return (cheese := 1 - 12)
Which happens to be invalid code. Now before making the parenthesis invisible, Black
checks if the assignment expression's parent is an assert stamtment, aborting if True.
Raise, yield, and await are already handled fine.
I added a bunch of test cases from the PEP defining asssignment expressions (PEP 572).
Optional trailing commas put by Black become magic trailing commas on another
pass of the tool. Since they are influencing formatting around optional
parentheses, on rare occasions the tool changes its mind in terms of putting
parentheses or not.
Ideally this would never be the case but sadly the decision to put optional
parentheses or not (which looks at pre-existing "magic" trailing commas) is
happening around the same time as the decision to put an optional trailing
comma. Untangling the two proved to be impractically difficult.
This shameful workaround uses the fact that the formatting instability
introduced by magic trailing commas is deterministic: if the optional trailing
comma becoming a pre-existing "magic" trailing comma changes formatting, the
second pass becomes stable since there is no variable factor anymore on pass 3,
4, and so on.
For most files, this will introduce no performance penalty since `--safe` is
already re-formatting everything twice to ensure formatting stability. We're
using this result and if all's good, the behavior is equivalent. If there is
a difference, we treat the second result as the binding one, and check its
sanity again.
Fixes#1738. Fixes#1812.
Previously, Black removed leading and trailing spaces in multiline docstrings but failed to remove them from one-line docstrings.
* Added support for top-level user configuration
At the user level, a TOML config can be specified in the following locations:
* Windows: ~\.black
* Unix-like: $XDG_CONFIG_HOME/black (~/.config/black fallback)
Instead of changing env vars for the entire black-primer process, they
are now changed only for the black subprocess, using a tmpdir.
Fixes: #1662
Work-around for https://bugs.python.org/issue2142
The test has to slightly mess with its input data, because the utility
functions default to ensuring the test data has a final newline, which
defeats the point of the test.
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
* Add test case to illustrate the issue
* Accept carriage returns as valid separators while enumerating comments
Without this acceptance, escaped multi-line statments that use carriage returns will not be counted into the 'ignored_lines' variable since the emitted line values will end with a CR and not an escape character. That leads to comments associated with the line being incorrectly labeled with the STANDALONE_COMMENT type, affecting comment placement and line space management.
* Remove comment linking to ephemeral build log
To avoid hitting a mypy bug causes pre-commit to always fail on CPython
3.9. Even though it's still an outdated version, the bug effectively
blocks development on CPython 3.9 so that's why this commit exists
instead of waiting for cooperlees to finish his bump to 0.790 PR.
Also this fixes primer to ensure it always raises CalledProcessError
with an int error code. I stole the patch from cooperlees's mypy bump
PR.
It's funny how mypy 0.790 is already asked for in our
Pipfile.lock file, but oh well mypy is probably more commonly run
through pre-commit than standalone I guess.
Oh and if you're curious why the bug doesn't up on CPython 3.8 or lower:
there was some subscription AST changes in CPython 3.9.
* Provide a stdin-filename to allow stdin to respect exclude/force-exclude rules
This will allow automatic tools to enforce the project's
exclude/force-exclude rules even if they pass the file through stdin to
update its buffer.
This is a similar solution to --stdin-display-name in flake8.
* Update src/black/__init__.py
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
* --stdin-filename should only respect --exclude-filename
* Update README with the new --stdin-filename option
* Write some tests for the new stdin-filename functionality
* Apply suggestions from code review
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
* Force stdin output when we asked for stdin even if the file exists
* Add an entry in the changelog regarding --stdin-filename
* Reduce disk reads if possible
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
* Check for is_stdin and p.is_file before checking for p.is_dir()
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
The `fix_docstring` function expanded all tabs, which caused a
difference in the AST representation when those tabs were inline and not
leading. This changes the function to only expand leading tabs so inline
tabs are preserved.
Fixes#1601.
- If no aiohttp* deps exist nicely print a helpful message and exit
- There seems to be no nice way to optionally install the entry point, so lets make the entry point nicer
Test:
```
cooper-mbp1:black cooper$ /tmp/tb/bin/pip install .
cooper-mbp1:black cooper$ /tmp/tb/bin/blackd
aiohttp dependency is not installed: No module named 'aiohttp'. Please re-install black with the '[d]' extra install to obtain aiohttp_cors: `pip install black[d]`
cooper-mbp1:black cooper$ /tmp/tb/bin/pip install .[d]
...
Successfully installed aiohttp-3.6.3 aiohttp-cors-0.7.0 black
cooper-mbp1:black cooper$ /tmp/tb/bin/blackd
blackd version 20.8b2.dev31+gdd2f86a.d20201013 listening on localhost port 45484
```
Fixes#1688
* Repair colorama wrapping on non-Windows platforms
The wrap_stream_for_windows() function calls
colorama.initialise.wrap_stream() function to apply colorama's magic to
wrapper to the output stream. Except this wrapper is only applied on
Windows platforms that need it, otherwise the original stream is
returned as-is.
The colorama wrapped stream lacks a detach() method, so a no-op lambda
was being assigned to the wrapped stream.
The problem is that the no-op lambda was being assigned unconditionally
whether or not colorama actually returns a wrapped stream, thus
replacing the original TextIOWrapper's detach() method. Replacing the
detach() method with a no-op lambda is the root cause of the problem
observed in #1664.
The solution is to only assign the no-op detach() method if the stream
lacks its own detach() method.
Repairs #1664
Black used to erroneously remove all empty lines between non-function
code and decorators when formatting typing stubs. Now a single empty
line is enforced.
I chose for putting empty lines around decorated classes that have empty
bodies since removing empty lines around such classes would cause a
formatting issue that seems to be impossible to fix.
For example:
```
class A: ...
@some_decorator
class B: ...
class C: ...
class D: ...
@some_other_decorator
def foo(): -> None: ...
```
It is easy to enforce no empty lines between class A, B, and C.
Just return 0, 0 for a line that is a decorator and precedes an stub
class. Fortunately before this commit, empty lines after that class
would be removed already.
Now let's look at the empty line between class D and function foo. In
this case, there should be an empty line there since it's class code next
to function code. The problem is that when deciding to add X empty lines
before a decorator, you can't tell whether it's before a class or a
function. If the decorator is before a function, then an empty line
is needed, while no empty lines are needed when the decorator is
before a class.
So even though I personally prefer no empty lines around decorated
classes, I had to go the other way surrounding decorated classes with
empty lines.
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
- tox recently adopted Black
a7903508fa
- attrs already adopted Black but they updated to 20.08b1 + did a format pass and removed some trailing commas
f680c5b83e
* Fix incorrect space before colon in if/while stmts
Previously Black would format this code
```
if (foo := True):
print(foo)
```
as
```
if (foo := True) :
print(foo)
```
adding an incorrect space after the RPAR. Buggy code in the
normalize_invisible_parens function caused the colon to be wrapped in
invisible parentheses. The LPAR of that pair was then prefixed with a
single space at the request of the whitespace function.
This commit fixes the accidental skipping of a pre-condition check
which must return True before parenthesis normalization of a specific
child Leaf or Node can happen. The pre-condition check being skipped
was why the colon was wrapped in invisible parentheses.
* Add an entry in CHANGES.md
The quotes of multiline docstrings are now only normalized when string
normalization is off, instead of the string normalization setting being
ignored and the quotes being *always* normalized.
I had to make a new test case and data file since the current pair for
docstrings only worked when there is no formatting difference between the
formatting results with string normalization on and off. I needed to add
tests for when there *are* differences between the two. So I split
test_docstring's test code when string normalization is disabled into a
new test case along with a new data file.
On a second pass of Black on the same file, inserted trailing commas are now
pre-existing. Doesn't make sense to differentiate between the passes then.
Reformatted projects I have acceess to:
- aioexabgp
- bandersnatch
- flake8-bugbear
```
-- primer results 📊 --
13 / 16 succeeded (81.25%) ✅
0 / 16 FAILED (0.0%) 💩
- 3 projects disabled by config
- 0 projects skipped due to Python version
- 0 skipped due to long checkout
```
* Also re-enable pytest
```
-- primer results 📊 --
14 / 16 succeeded (87.5%) ✅
0 / 16 FAILED (0.0%) 💩
- 2 projects disabled by config
- 0 projects skipped due to Python version
- 0 skipped due to long checkout
real 2m26.207s
user 17m55.404s
sys 0m43.061s
```
This addresses a few crashers, namely:
* producing non-equivalent code due to mangling escaped newlines,
* invalid hugging quote characters in the docstring body to the docstring outer
triple quotes (causing a quadruple quote which is a syntax error),
* lack of handling for docstrings that start on the same line as the `def`, and
* invalid stripping of outer triple quotes when the docstring contained
a string prefix.
As a bonus, tests now also run when string normalization is disabled.
This required some hackery. Long story short, we need to reuse the ability to
omit rightmost bracket pairs (which glues them together and splits on something
else instead), for use with pre-existing trailing commas.
This form of user-controlled formatting is brittle so we have to be careful not
to cause a scenario where Black first formats code without trailing commas in
one way, and then looks at the same file with pre-existing trailing commas
(that it itself put on the previous run) and decides to format the code again.
One particular ugly edge case here is handling of optional parentheses. In
particular, the long-standing `line_length=1` hack got in the way of
pre-existing trailing commas and had to be removed. Instead, a more
intelligent but costly solution was put in place: a "second opinion" if the
formatting that omits optional parentheses ended up causing lines to be too
long. Again, for efficiency purposes, Black reuses Leaf objects from blib2to3
and modifies them in place, which was invalid for having two separate
formattings. Line cloning was used to mitigate this.
Fixes#1619
- when a trailing comma is specified in any bracket pair, that signals to Black
that this bracket pair needs to be always exploded, e.g. presented as "one
item per line";
- this causes some changes to previously formatted code that erroneously left
trailing commas embedded into single-line expressions;
- internally, Black needs to be able to identify trailing commas that it put
itself compared to pre-existing trailing commas. We do this by using/abusing
lib2to3's `was_checked` attribute. It's True for internally generated
trailing commas and False for pre-existing ones (in fact, for all
pre-existing leaves and nodes).
Fixes#1288
* Upgrade docs to Sphinx 3+
* Fix all the warnings...
- Fixed bad docstrings
- Fixed bad fenced code blocks in documentation
- Blocklisted some sections from being generated from the README
- Added missing documentation to index.rst
- Fixed an invalid autofunction directive in reference/reference_functions.rst
- Pin another documentation dependency
* Add documentation build test
* put experimental string stuff behind a flag
* update tests
* don't need an output section if it's the same as the input
* Primer: Expect no formatting changes in attrs, hypothesis and poetry with --experimental-string-processing off
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Ever since --force-exclude was added, --exclude started to touch files
that were given to Black through the CLI too. This is not documented
behaviour and neither expected as --exclude and --force-exclude now
behave the same!
Before this commit, get_sources() when encountering a file that was passed
explicitly through the CLI would pass a single Path object list to
gen_python_files(). This causes bad behaviour since that function
doesn't treat the exclude and force_exclude regexes differently. Which
is fine for recursively found files, but *not* for files given through
the CLI.
Now when get_sources() iterates through srcs and encounters
a file, it checks if the force_exclude regex matches, if not, then the
file will be added to the computed sources set.
A new function had to be created since before you can do regex matching,
the path must be normalized. The full process of normalizing the path is
somewhat long as there is special error handling. I didn't want to
duplicate this logic in get_sources() and gen_python_files() so that's
why there is a new helper function.
Partial fix for #1581
This assertion produces behavior quadratic in the number of leaves in a line, which is making Black extremely slow on files with very long expressions. On my benchmark file this change makes Black 10x faster.
* Bump toml from 0.10.0 to 0.10.1 to fix a bug
* Add tests for TOML parsing and reading
* Fix configuration bug affecting vim plugin
The vim plugin directly calls parse_pyproject and skips the Click processing
, but parse_pyproject assumed that it would only be used before Click processing
and therefore made the config values click friendly. This moves the "make the values
click friendly processing" into read_pyproject_toml which is only called by a Click
callback.
* Please mypy and flake8
* Convert config values to string
We need to convert all configuration values from the pyproject.toml
file because Click already does value processing and conversion and
it only expects the exact Python type or string. Click doesn't like
the integer 1 as a boolean for example. This also means other
unsupported types like datetime.time will be rejected by Click as
a unvalid value instead of throwing an exception.
We only skip converting objects that are an instance of
collections.abc.Iterable because it's almost impossible to get back
the original iterable of a stringified iterable.
* Move where the conversion happens
Instead of converting the values in the merged 'default_map', I should
convert the values that were read from the 'pyproject.toml' file.
* Change collections.abc.Iterable to (list, dict)
I also moved where the conversion happens... again. I am rather indecisive
if you haven't noticed. It should be better as it takes place in the
parse_pyproject_toml logic where configuration modification already takes
place.
Actually when this PR was first created I had the conversion happen in that
return statement, but the target_version check was complaining about it being
a string. So I moved the conversion after that check, but then Click didn't
like the stringifed list, which led me to check whether the value was an
instance of an Iterable before turning it into a string. And... I forgot that
type checking before conversion would allow it to work before the
target_version check anyway.
- Leave logic to still allow for formatting changes to be ignored
- Now just capture the output of any other error that has a > 1 returncode
- Raise on anything else
Test: Add unit test to exercise this new logic
* Add all listed by projects into primer.json + Enable on CI Runs
- Change workers default to 2 as black uses system CPU count
- Increase timeout to 5 mins for subprocess black runs
- Takes about 120s for 13 (3 disabled) projects on my 2018 Macbook Pro
- I was not removing directories tho ...
Will open an issue to investigate the failing projects and make this run cleaner.
- Once we get more stable we can expect more repos to be black formatted
Run it:
- `black-primer -k -w /tmp/primer_large_test --debug --rebase`
```
[2020-05-20 21:44:01,273] DEBUG: Starting /Users/cooper/venvs/b/bin/black-primer (cli.py:125)
[2020-05-20 21:44:01,273] DEBUG: Using selector: KqueueSelector (selector_events.py:53)
[2020-05-20 21:44:01,274] INFO: 16 projects to run Black over (lib.py:276)
[2020-05-20 21:44:01,274] DEBUG: Using 2 parallel workers to run Black (lib.py:281)
[2020-05-20 21:44:01,274] DEBUG: worker 0 workng on aioexabgp (lib.py:215)
[2020-05-20 21:44:01,276] DEBUG: worker 1 workng on attrs (lib.py:215)
[2020-05-20 21:44:02,443] INFO: Finished aioexabgp (lib.py:249)
[2020-05-20 21:44:02,443] DEBUG: worker 0 workng on bandersnatch (lib.py:215)
[2020-05-20 21:44:04,409] INFO: Finished bandersnatch (lib.py:249)
[2020-05-20 21:44:04,409] DEBUG: worker 0 workng on channels (lib.py:215)
[2020-05-20 21:44:04,702] INFO: Finished attrs (lib.py:249)
[2020-05-20 21:44:04,702] DEBUG: worker 1 workng on django (lib.py:215)
[2020-05-20 21:44:04,702] INFO: Skipping django as it's disabled via config (lib.py:222)
[2020-05-20 21:44:04,702] DEBUG: worker 1 workng on flake8-bugbear (lib.py:215)
[2020-05-20 21:44:05,813] INFO: Finished channels (lib.py:249)
[2020-05-20 21:44:05,813] DEBUG: worker 0 workng on hypothesis (lib.py:215)
[2020-05-20 21:44:06,071] INFO: Finished flake8-bugbear (lib.py:249)
[2020-05-20 21:44:06,071] DEBUG: worker 1 workng on pandas (lib.py:215)
[2020-05-20 21:44:06,071] INFO: Skipping pandas as it's disabled via config (lib.py:222)
[2020-05-20 21:44:06,071] DEBUG: worker 1 workng on poetry (lib.py:215)
[2020-05-20 21:44:16,207] INFO: Finished hypothesis (lib.py:249)
[2020-05-20 21:44:16,207] DEBUG: worker 0 workng on ptr (lib.py:215)
[2020-05-20 21:44:17,077] INFO: Finished poetry (lib.py:249)
[2020-05-20 21:44:17,077] DEBUG: worker 1 workng on pyramid (lib.py:215)
[2020-05-20 21:44:17,460] INFO: Finished ptr (lib.py:249)
[2020-05-20 21:44:17,460] DEBUG: worker 0 workng on pytest (lib.py:215)
[2020-05-20 21:44:17,460] INFO: Skipping pytest as it's disabled via config (lib.py:222)
[2020-05-20 21:44:17,460] DEBUG: worker 0 workng on sqlalchemy (lib.py:215)
[2020-05-20 21:44:33,319] INFO: Finished pyramid (lib.py:249)
[2020-05-20 21:44:33,319] DEBUG: worker 1 workng on tox (lib.py:215)
[2020-05-20 21:44:42,274] INFO: Finished tox (lib.py:249)
[2020-05-20 21:44:42,275] DEBUG: worker 1 workng on virtualenv (lib.py:215)
[2020-05-20 21:44:47,928] INFO: Finished virtualenv (lib.py:249)
[2020-05-20 21:44:47,928] DEBUG: worker 1 workng on warehouse (lib.py:215)
[2020-05-20 21:45:16,784] INFO: Finished warehouse (lib.py:249)
[2020-05-20 21:45:16,784] DEBUG: project_runner 1 exiting (lib.py:213)
[2020-05-20 21:45:45,700] INFO: Finished sqlalchemy (lib.py:249)
[2020-05-20 21:45:45,700] DEBUG: project_runner 0 exiting (lib.py:213)
[2020-05-20 21:45:45,701] INFO: Analyzing results (lib.py:292)
-- primer results 📊 --
13 / 16 succeeded (81.25%) ✅
0 / 16 FAILED (0.0%) 💩
- 3 projects disabled by config
- 0 projects skipped due to Python version
- 0 skipped due to long checkout
```
* Move to partial for rmtree + specify a onerror handler for PermissionError on Windows for git
* Set default coding to utf8 for very important emoji's on Windows
* Set Python encoding to utf-8 for Windows
* Appease the white space gods of Black!
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
* Add black-primer unittests
- Get this tool covered with some decent unittests for all unittests wins
- Have a CLI and lib test class
- Import it from `test_black.py` so we always run tests
- Revert typing asyncio.Queue as Queue[str] so we can work in 3.6
- **mypy**: Until black > 3.6 disallow_any_generics=False for primer code
Test:
- Run tests: `coverage run tests/test_primer.py` or `coverage run -m unittest`
```
(b) cooper-mbp1:black cooper$ coverage report
Name Stmts Miss Cover
---------------------------------------------
src/black_primer/cli.py 49 8 84%
src/black_primer/lib.py 148 28 81%
tests/test_primer.py 114 1 99%
---------------------------------------------
TOTAL 311 37 88%
```
* Use ProactorEventLoop for Windows + fix false path for Linux
* Set Windows to use ProactorEventLoop in to benefit all callers
* sys.platform seems to not having the loop applied - So type ignore and use platform.system() gate
* Have each test loop correctly set to ProactorEventLoop on Windows for < 3.8 too
