This fixes a bug where a trailing comma would be added to a
parenthesized return annotation changing its type to a tuple.
Here's one case where this bug shows up:
```
def spam() -> (
this_is_a_long_type_annotation_which_should_NOT_get_a_trailing_comma
):
pass
```
The root problem was that the type annotation was treated as if it was
a parameter & import list (is_body=True to linegen::bracket_split_build_line)
where a trailing comma is usually fine. Now there's another check in the
aforementioned function to make sure the body it's operating on isn't
a return annotation before truly adding a trailing comma.
* Add CPython repository into primer runs
- CPython tests is probably the best repo for black to test on as the stdlib's unittests should use all syntax
- Limit to running in recent versions of the python runtime - e.g. today >= 3.9
- This allows us to parse more syntax
- Exclude all failing files for now
- Definitely have bugs to explore there - Refer to #2407 for more details there
- Some test files on purpose have syntax errors, so we will never be able to parse them
- Add new black command arguments logging in debug mode; very handy for seeing how CLI arguments are formatted
CPython now succeeds ignoring 16 files:
```
Oh no! 💥💔💥
1859 files would be reformatted, 148 files would be left unchanged.
```
Testing
- Ran locally with and without string processing - Very little runtime difference BUT 3 more failed files
```
time /tmp/tb/bin/black --experimental-string-processing --check . 2>&1 | tee /tmp/black_cpython_esp
...
Oh no! 💥💔💥
1859 files would be reformatted, 148 files would be left unchanged, 16 files would fail to reformat.
real 4m8.563s
user 16m21.735s
sys 0m6.000s
```
- Add unittest for new covienence config file flattening that allows long arguments to be broke up into an array/list of strings
Addresses #2407
---
Commit history before merge:
* Add new `timeout_seconds` support into primer.json
- If present, will set forked process limit to that value in seconds
- Otherwise, stay with default 10 minutes (600 seconds)
* Add new "base_path" concept to black-primer
- Rather than start at the repo root start at a configured path within the repository
- e.g. for cpython only run black on `Lib`
* Disable by default - It's too much for GitHub Actions. But let's leave config for others to use
* Minor tweak to _flatten_cli_args
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
The fix for #1688 in #1761 breaks help("modules") introspection and also leads
to unhappy results when inadvertently importing blackd from Python. Basically
the sys.exit(-1) causes the whole Python REPL to exit -- not great to suffice.
Commit history before merge:
* Change sys.exit to Raise.
* Add #2440 to changelog.
* Fix lint error from prettier
* Remove exception chain for more helpful user message.
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
we don't accidentally add backslashes to them when normalizing quotes
because that's invalid syntax!
The problem this commit fixes is that matches would eat too much
blocking important matches to occur. For example, here's one f-string
body:
{a}{b}{c}
I know there's no risk of introducing backslashes here, but the regex
already goes sideways with this. Throwing this example at regex101
I get:
{a}{b}{c} # The As and Bs are the two matches, and the upper
---- ---- # case letters are the groups with those matches.
aAaa bbBb
... we've missed the middle expression (so if any backslashes in a
more complex example were introduced there we wouldn't bail out
even though we should -- hence the bug). As it stands the regex
needs somesort of extra character (or the start/end of the body)
around the expressions but that isn't always the case as shown
above.
The fix implemented here is to turn the "eat a surrounding non-curly
bracket character" groups ie. `(?:[^{]|^)` and `(?:[^}]|$)` into
negative lookaheads and lookbehinds. This still guarantees the
already specified rules but without problematically eating extra
characters ^^
Fixes#2359.
This commit now makes Black exit with an user-friendly error message if a
.gitignore file couldn't be parsed -- a massive improvement over an opaque
traceback!
To summarise, based on what was discussed in that issue:
due to not being able to parse automagics (e.g. pip install black)
without a running IPython kernel, cells with syntax which is parseable
by neither ast.parse nor IPython will be skipped cells with multiline
magics will be skipped trailing semicolons will be preserved, as they
are often put there intentionally in Jupyter Notebooks to suppress
unnecessary output
Commit history before merge (excluding merge commits):
* wip
* fixup tests
* skip tests if no IPython
* install test requirements in ipynb tests
* if --ipynb format all as ipynb
* wip
* add some whole-notebook tests
* docstrings
* skip multiline magics
* add test for nested cell magic
* remove ipynb_test.yml, put ipynb tests in tox.ini
* add changelog entry
* typo
* make token same length as magic it replaces
* only include .ipynb by default if jupyter dependencies are found
* remove logic from const
* fixup
* fixup
* re.compile
* noop
* clear up
* new_src -> dst
* early exit for non-python notebooks
* add non-python test notebook
* add repo with many notebooks to black-primer
* install extra dependencies for black-primer
* fix planetary computer examples url
* dont run on ipynb files by default
* add scikit-lego (Expected to change) to black-primer
* add ipynb-specific diff
* fixup
* run on all (including ipynb) by default
* remove --include .ipynb from scikit-lego black-primer
* use tokenize so as to mirror the exact logic in IPython.core.displayhooks quiet
* fixup
* 🎨
* clarify docstring
* add test for when comment is after trailing semicolon
* enumerate(reversed) instead of [::-1]
* clarify docstrings
* wip
* use jupyter and no_jupyter marks
* use THIS_DIR
* windows fixup
* perform safe check cell-by-cell for ipynb
* only perform safe check in ipynb if not fast
* remove redundant Optional
* 🎨
* use typeguard
* dont process cell containing transformed magic
* require typing extensions before 3.10 so as to have TypeGuard
* use dataclasses
* mention black[jupyter] in docs as well as in README
* add faq
* add message to assertion error
* add test for indented quieted cell
* use tokenize_rt else we cant roundtrip
* fmake fronzet set for tokens to ignore when looking for trailing semicolon
* remove planetary code examples as recent commits result in changes
* use dataclasses which inherit from ast.NodeVisitor
* bump typing-extensions so that TypeGuard is available
* bump typing-extensions in Pipfile
* add test with notebook with empty metadata
* pipenv lock
* deprivative validate_cell
* Update README.md
* Update docs/getting_started.md
* dont cache notebooks if jupyter dependencies arent found
* dont write to cache if jupyter deps are not installed
* add notebook which cant be parsed
* use clirunner
* remove other subprocess calls
* add docstring
* make verbose and quiet keyword only
* 🎨
* run second many test on directory, not on file
* test for warning message when running on directory
* early return from non-python cell magics
* move NothingChanged to report to avoid circular import
* remove circular import
* reinstate --ipynb flag
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
toml unfortunately has a lack of maintainership issue right now. It's
evident by the fact toml only supports TOML v0.5.0. TOML v1.0.0 has
been recently released and right now Black crashes hard on its usage.
tomli is a brand new parse only TOML library. It supports TOML
v1.0.0. Although TBH we're switching to this one mostly because
pip is doing the same.
*The upper bound was included at the library maintainer's request.
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Taneli Hukkinen <3275109+hukkin@users.noreply.github.com>
Commit history before merge:
* Accept empty stdin (close#2337)
* Update tests/test_black.py
* Add changelog
* Assert Black reformats an empty string to an empty string (#2337) (#2346)
* fix
* Add STDIN test to primer
- Check that out STDIN black support stays working
- Add asyncio.subprocess STDIN pip via communicate
- We just check we format python code from primer's `lib.py`
Fixes#2310
`black.strings.get_string_prefix` used to lowercase the extracted
prefix before returning it. This is wrong because 1) it ignores the
fact we should leave R prefixes alone because of MagicPython, and 2)
there is dedicated prefix casing handling code that fixes issue 1.
`.lower` is too naive.
This was originally fixed in 20.8b0, but was reintroduced since 21.4b0.
I also added proper prefix normalization for docstrings by using the
`black.strings.normalize_string_prefix` helper.
Some more test strings were added to make sure strings with capitalized
prefixes aren't treated differently (actually happened with my original
patch, Jelle had to point it out to me).
Not sure the fix is right. Here is what I found: issue is connected
with line
first.prefix = prefix[comment.consumed :]
in `comments.py`. `first.prefix` is a prefix of the line, that ends
with `# fmt: skip`, but `comment.consumed` is the length of the
`" # fmt: skip"` string. If prefix length is greater than 14,
`first.prefix` will grow every time we apply formatting.
Fixes#2254
See if we pass all our repos with experimental string processing enabled.
Django probably needed:
- Ignores >= 3.8 only
We could support PEP440 version specifiers, but that would introduce the packaging module as a dependency that I'd like to avoid ... Or I could implement a poor persons version or vendor
Commit history before merge:
* [primer] Enable everything
* Add exclude extend to django CLI args for primer
* Change default timeout to from 5 to 10 mins for a primer project
* Skip string normalization for Django
* Limit Django to >= 3.8 due to := operator
Closes#1246: This PR adds a new option (and automatically a toml entry, hooray for existing configuration management 🎉) to require a specific version of Black to be running.
For example: `black --required-version 20.8b -c "format = 'this'"`
Execution fails straight away if it doesn't match `__version__`.
PR #2286 did not fix the edge-cases (e.g. when the string is just long
enough to cause a line to be 89 characters long). This PR corrects that
mistake.
Behavior other than output shouldn't depend on the verbose/quiet option. As far as I can tell this currently has no visible effect, since code after this function is called handles an empty list gracefully.
* Add optional uvloop import
- If we find `uvloop` in the env for black, blackd or black-primer lets try and use it
- Add a uvloop extra install
Fixes#2257
Test:
- Add ci job to install black[uvloop] and run a primer run with uvloop
- Only with latest python (3.9)
- Will be handy to compare runtimes as a very unoffical benchmark
* Remove tox install
* Add to CHANGES/news
Resolves#2168 by disabling the insertion of a " " when the docstring is entirely empty.
Note that this PR is focussed only on the case of empty docstrings. In particular this does not make any changes to the behaviour that a " " is inserted if a non-empty docstring begins with the quoting character. That is, black still prefers:
""" "something" """
to:
""""something" """
and that:
""""Something""""
is not a legal docstring.
* Solved Problem with non-alphabetical .gitignore files
When .gitignore file in the user's project directory contained non-alphabetical
characters(Japanese, Korean, Chinese, etc), Nothing works and printed this
weird message in the console('cp949' is the encoding for Korean characters
in this case). It even blocks VSCode's formatting from working. This commit
solves the problem.
Traceback (most recent call last):
File "c:\users\username\anaconda3\envs\project-name\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "c:\users\username\anaconda3\envs\project-name\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\username\anaconda3\envs\project-name\Scripts\black.exe\__main__.py", line 7, in <module>
File "c:\users\username\anaconda3\envs\project-name\lib\site-packages\black\__init__.py", line 1056, in patched_main
main()
File "c:\users\username\anaconda3\envs\project-name\lib\site-packages\click\core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "c:\users\username\anaconda3\envs\project-name\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "c:\users\username\anaconda3\envs\project-name\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "c:\users\username\anaconda3\envs\project-name\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "c:\users\username\anaconda3\envs\project-name\lib\site-packages\click\decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "c:\users\username\anaconda3\envs\project-name\lib\site-packages\black\__init__.py", line 394, in main
stdin_filename=stdin_filename,
File "c:\users\username\anaconda3\envs\project-name\lib\site-packages\black\__init__.py", line 445, in get_sources
gitignore = get_gitignore(root)
File "c:\users\username\anaconda3\envs\project-name\lib\site-packages\black\files.py", line 122, in get_gitignore
lines = gf.readlines()
UnicodeDecodeError: 'cp949' codec can't decode byte 0xb0 in position 13: illegal multibyte sequence
* Made .gitignore File Reader Detect Its Encoding
* Revert "Made .gitignore File Reader Detect Its Encoding"
This reverts commit 6c3a7ea42b5b1e441cc0026c8205d1cee68c1bba.
* Revert "Solved Problem with non-alphabetical .gitignore files"
This reverts commit b0100b5d91c2f5db544a60f34aafab120f0aa458.
* Made .gitignore Reader Open the File with Auto Encoding Detecting
https://docs.python.org/3.8/library/tokenize.html#tokenize.open
* Revert "Made .gitignore Reader Open the File with Auto Encoding Detecting"
This reverts commit 50dd80422938649ccc8c7f43aac752f9f6481779.
* Made .gitignore Reader Use UTF-8
* Updated CHANGES.md for #2229
* Updated CHANGES.md for #2229
* Update CHANGES.md
* Update CHANGES.md
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
Commit history before merge:
Black now respects .gitignore files in all levels, not only root/.gitignore file
(apply .gitignore rules like git does).
* Fix: typo
* Fix: respect .gitignore files in all levels.
* Add: CHANGELOG note.
* Fix: TypeError: unsupported operand type(s) for +: 'NoneType' and 'PathSpec'
* Update docs.
* Fix: no parent .gitignore
* Add a comment since the if expression is a bit hard to understand
* Update tests - conver no parent .gitignore case.
* Use main's Pipfile.lock instead
The original changes in Pipfile.lock are whitespace only. The changes
turned the JSON's file indentation from 4 to 2. Effectively this
happened: `json.dumps(json.loads(old_pipfile_lock), indent=2) + "\n"`.
Just using main's Pipfile.lock instead of undoing the changes because
1) I don't know how to do that easily and quickly, and 2) there's a
merge conflict.
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
* Merge remote-tracking branch 'upstream/main' into i1730 …
conflicts for days ay?
It appears sqlalchemy has recently reformatted their project with
Black 21.5b1.
Most of our dependencies have a lower bound and creating a test
environment with the oldest acceptable dependencies runs the full
Black test suite just fine. The only exception to this is aiohttp-cors.
It's unbounded and the oldest version 0.1.0 until 0.4.0 breaks the
test suite in such an old environment.
Failure with 0.1.0:
```
tests/test_blackd.py:10: in <module>
import blackd
testenv/lib/python3.8/site-packages/blackd/__init__.py:12: in <module>
import aiohttp_cors
testenv/lib/python3.8/site-packages/aiohttp_cors/__init__.py:29: in <module>
from .urldispatcher_router_adapter import UrlDistatcherRouterAdapter
testenv/lib/python3.8/site-packages/aiohttp_cors/urldispatcher_router_adapter.py:27: in <module>
class UrlDistatcherRouterAdapter(RouterAdapter):
testenv/lib/python3.8/site-packages/aiohttp_cors/urldispatcher_router_adapter.py:32: in UrlDistatcherRouterAdapter
def route_methods(self, route: web.Route):
E AttributeError: module 'aiohttp.web' has no attribute 'Route'
```
For 0.2.0:
```
tests/test_blackd.py:10: in <module>
import blackd
testenv/lib/python3.8/site-packages/blackd/__init__.py:12: in <module>
import aiohttp_cors
testenv/lib/python3.8/site-packages/aiohttp_cors/__init__.py:27: in <module>
from .cors_config import CorsConfig
testenv/lib/python3.8/site-packages/aiohttp_cors/cors_config.py:24: in <module>
from .urldispatcher_router_adapter import UrlDistatcherRouterAdapter
testenv/lib/python3.8/site-packages/aiohttp_cors/urldispatcher_router_adapter.py:27: in <module>
class UrlDistatcherRouterAdapter(AbstractRouterAdapter):
testenv/lib/python3.8/site-packages/aiohttp_cors/urldispatcher_router_adapter.py:32: in UrlDistatcherRouterAdapter
def route_methods(self, route: web.Route):
E AttributeError: module 'aiohttp.web' has no attribute 'Route'
```
For 0.3.0:
```
ERROR: Cannot install aiohttp-cors==0.3.0 and aiohttp==3.6.0 because these package versions have conflicting dependencies.
The conflict is caused by:
The user requested aiohttp==3.6.0
aiohttp-cors 0.3.0 depends on aiohttp<=0.20.2 and >=0.18.0
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
```
* Move string-related utility to functions to strings.py, const.py
* Move Leaf/Node-related functionality to nodes.py
* Move comment-related functions to comments.py
* Move caching to cache.py and Mode/TargetVersion/Feature to mode.py
* Move some leftover functions to nodes.py, comments.py, strings.py
* Add missing files to source list for test runs
* Move line-related functionality into lines.py, brackets into brackets.py
* Move transformers to trans.py
* Move file handling, output, parsing, concurrency, debug, and report
* Move two more functions to nodes.py
* Add CHANGES
* Add numeric.py
* Add linegen.py
* More docstrings
* Include new files in tests
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Closes#2164.
Changes behavior of how .gitignore is handled. With this change, the rules in .gitignore are only used as a fallback if no exclusion rule is explicitly passed on the command line or in pyproject.toml. Previously they were used regardless if explicit exclusion rules were specified, preventing any overriding of .gitignore rules.
Those that depend only on .gitignore for their exclusion rules will not be affected. Those that use both .gitignore and exclude will find that exclude will act more like actually specifying exclude and not just another extra-excludes. If the previous behavior was desired, they should move their rules from exclude to extra-excludes.
* Enable ` --experimental-string-processing` on all primer projects
- We want to make this default so need to test it more
- Fixed splat/star bug in extending black args for each project
* Disable sqlalchemy due to crash
Previously the RELAXED_DECORATOR detection would be falsely True on that
example. The problem was that an argument-less parentheses pair didn't
pass the `is_simple_decorator_trailer` check even it should. OTOH a
parentheses pair containing an argument or more passed as expected.
- Allow runs with no code diff output
- This is handy for reducing output to see which file is erroring
Test:
- Edit config for 'channels' to expect no changes and run with `--no-diff` and see no diff output
- It no longer crashes black so we should test on it's code
- Update django reason to name the file causing error
- Seems it has a syntax error on purpose
Such shebangs are only ever used if the file is executed directly, i.e.:
$ /usr/lib/python3.9/site-packages/black_primer/cli.py
But that doesn't work:
$ /usr/lib/python3.9/site-packages/black_primer/cli.py
bash: /usr/lib/python3.9/site-packages/black_primer/cli.py: Permission denied
The lib file even has: "lib is a library, funnily enough"
