The `fix_docstring` function expanded all tabs, which caused a
difference in the AST representation when those tabs were inline and not
leading. This changes the function to only expand leading tabs so inline
tabs are preserved.
Fixes#1601.
- If no aiohttp* deps exist nicely print a helpful message and exit
- There seems to be no nice way to optionally install the entry point, so lets make the entry point nicer
Test:
```
cooper-mbp1:black cooper$ /tmp/tb/bin/pip install .
cooper-mbp1:black cooper$ /tmp/tb/bin/blackd
aiohttp dependency is not installed: No module named 'aiohttp'. Please re-install black with the '[d]' extra install to obtain aiohttp_cors: `pip install black[d]`
cooper-mbp1:black cooper$ /tmp/tb/bin/pip install .[d]
...
Successfully installed aiohttp-3.6.3 aiohttp-cors-0.7.0 black
cooper-mbp1:black cooper$ /tmp/tb/bin/blackd
blackd version 20.8b2.dev31+gdd2f86a.d20201013 listening on localhost port 45484
```
Fixes#1688
* Repair colorama wrapping on non-Windows platforms
The wrap_stream_for_windows() function calls
colorama.initialise.wrap_stream() function to apply colorama's magic to
wrapper to the output stream. Except this wrapper is only applied on
Windows platforms that need it, otherwise the original stream is
returned as-is.
The colorama wrapped stream lacks a detach() method, so a no-op lambda
was being assigned to the wrapped stream.
The problem is that the no-op lambda was being assigned unconditionally
whether or not colorama actually returns a wrapped stream, thus
replacing the original TextIOWrapper's detach() method. Replacing the
detach() method with a no-op lambda is the root cause of the problem
observed in #1664.
The solution is to only assign the no-op detach() method if the stream
lacks its own detach() method.
Repairs #1664
Black used to erroneously remove all empty lines between non-function
code and decorators when formatting typing stubs. Now a single empty
line is enforced.
I chose for putting empty lines around decorated classes that have empty
bodies since removing empty lines around such classes would cause a
formatting issue that seems to be impossible to fix.
For example:
```
class A: ...
@some_decorator
class B: ...
class C: ...
class D: ...
@some_other_decorator
def foo(): -> None: ...
```
It is easy to enforce no empty lines between class A, B, and C.
Just return 0, 0 for a line that is a decorator and precedes an stub
class. Fortunately before this commit, empty lines after that class
would be removed already.
Now let's look at the empty line between class D and function foo. In
this case, there should be an empty line there since it's class code next
to function code. The problem is that when deciding to add X empty lines
before a decorator, you can't tell whether it's before a class or a
function. If the decorator is before a function, then an empty line
is needed, while no empty lines are needed when the decorator is
before a class.
So even though I personally prefer no empty lines around decorated
classes, I had to go the other way surrounding decorated classes with
empty lines.
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
- tox recently adopted Black
a7903508fa
- attrs already adopted Black but they updated to 20.08b1 + did a format pass and removed some trailing commas
f680c5b83e
* Fix incorrect space before colon in if/while stmts
Previously Black would format this code
```
if (foo := True):
print(foo)
```
as
```
if (foo := True) :
print(foo)
```
adding an incorrect space after the RPAR. Buggy code in the
normalize_invisible_parens function caused the colon to be wrapped in
invisible parentheses. The LPAR of that pair was then prefixed with a
single space at the request of the whitespace function.
This commit fixes the accidental skipping of a pre-condition check
which must return True before parenthesis normalization of a specific
child Leaf or Node can happen. The pre-condition check being skipped
was why the colon was wrapped in invisible parentheses.
* Add an entry in CHANGES.md
The quotes of multiline docstrings are now only normalized when string
normalization is off, instead of the string normalization setting being
ignored and the quotes being *always* normalized.
I had to make a new test case and data file since the current pair for
docstrings only worked when there is no formatting difference between the
formatting results with string normalization on and off. I needed to add
tests for when there *are* differences between the two. So I split
test_docstring's test code when string normalization is disabled into a
new test case along with a new data file.
On a second pass of Black on the same file, inserted trailing commas are now
pre-existing. Doesn't make sense to differentiate between the passes then.
Reformatted projects I have acceess to:
- aioexabgp
- bandersnatch
- flake8-bugbear
```
-- primer results 📊 --
13 / 16 succeeded (81.25%) ✅
0 / 16 FAILED (0.0%) 💩
- 3 projects disabled by config
- 0 projects skipped due to Python version
- 0 skipped due to long checkout
```
* Also re-enable pytest
```
-- primer results 📊 --
14 / 16 succeeded (87.5%) ✅
0 / 16 FAILED (0.0%) 💩
- 2 projects disabled by config
- 0 projects skipped due to Python version
- 0 skipped due to long checkout
real 2m26.207s
user 17m55.404s
sys 0m43.061s
```
This addresses a few crashers, namely:
* producing non-equivalent code due to mangling escaped newlines,
* invalid hugging quote characters in the docstring body to the docstring outer
triple quotes (causing a quadruple quote which is a syntax error),
* lack of handling for docstrings that start on the same line as the `def`, and
* invalid stripping of outer triple quotes when the docstring contained
a string prefix.
As a bonus, tests now also run when string normalization is disabled.
This required some hackery. Long story short, we need to reuse the ability to
omit rightmost bracket pairs (which glues them together and splits on something
else instead), for use with pre-existing trailing commas.
This form of user-controlled formatting is brittle so we have to be careful not
to cause a scenario where Black first formats code without trailing commas in
one way, and then looks at the same file with pre-existing trailing commas
(that it itself put on the previous run) and decides to format the code again.
One particular ugly edge case here is handling of optional parentheses. In
particular, the long-standing `line_length=1` hack got in the way of
pre-existing trailing commas and had to be removed. Instead, a more
intelligent but costly solution was put in place: a "second opinion" if the
formatting that omits optional parentheses ended up causing lines to be too
long. Again, for efficiency purposes, Black reuses Leaf objects from blib2to3
and modifies them in place, which was invalid for having two separate
formattings. Line cloning was used to mitigate this.
Fixes#1619
- when a trailing comma is specified in any bracket pair, that signals to Black
that this bracket pair needs to be always exploded, e.g. presented as "one
item per line";
- this causes some changes to previously formatted code that erroneously left
trailing commas embedded into single-line expressions;
- internally, Black needs to be able to identify trailing commas that it put
itself compared to pre-existing trailing commas. We do this by using/abusing
lib2to3's `was_checked` attribute. It's True for internally generated
trailing commas and False for pre-existing ones (in fact, for all
pre-existing leaves and nodes).
Fixes#1288
* Upgrade docs to Sphinx 3+
* Fix all the warnings...
- Fixed bad docstrings
- Fixed bad fenced code blocks in documentation
- Blocklisted some sections from being generated from the README
- Added missing documentation to index.rst
- Fixed an invalid autofunction directive in reference/reference_functions.rst
- Pin another documentation dependency
* Add documentation build test
* put experimental string stuff behind a flag
* update tests
* don't need an output section if it's the same as the input
* Primer: Expect no formatting changes in attrs, hypothesis and poetry with --experimental-string-processing off
Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>
Ever since --force-exclude was added, --exclude started to touch files
that were given to Black through the CLI too. This is not documented
behaviour and neither expected as --exclude and --force-exclude now
behave the same!
Before this commit, get_sources() when encountering a file that was passed
explicitly through the CLI would pass a single Path object list to
gen_python_files(). This causes bad behaviour since that function
doesn't treat the exclude and force_exclude regexes differently. Which
is fine for recursively found files, but *not* for files given through
the CLI.
Now when get_sources() iterates through srcs and encounters
a file, it checks if the force_exclude regex matches, if not, then the
file will be added to the computed sources set.
A new function had to be created since before you can do regex matching,
the path must be normalized. The full process of normalizing the path is
somewhat long as there is special error handling. I didn't want to
duplicate this logic in get_sources() and gen_python_files() so that's
why there is a new helper function.
Partial fix for #1581
This assertion produces behavior quadratic in the number of leaves in a line, which is making Black extremely slow on files with very long expressions. On my benchmark file this change makes Black 10x faster.
* Bump toml from 0.10.0 to 0.10.1 to fix a bug
* Add tests for TOML parsing and reading
* Fix configuration bug affecting vim plugin
The vim plugin directly calls parse_pyproject and skips the Click processing
, but parse_pyproject assumed that it would only be used before Click processing
and therefore made the config values click friendly. This moves the "make the values
click friendly processing" into read_pyproject_toml which is only called by a Click
callback.
* Please mypy and flake8
* Convert config values to string
We need to convert all configuration values from the pyproject.toml
file because Click already does value processing and conversion and
it only expects the exact Python type or string. Click doesn't like
the integer 1 as a boolean for example. This also means other
unsupported types like datetime.time will be rejected by Click as
a unvalid value instead of throwing an exception.
We only skip converting objects that are an instance of
collections.abc.Iterable because it's almost impossible to get back
the original iterable of a stringified iterable.
* Move where the conversion happens
Instead of converting the values in the merged 'default_map', I should
convert the values that were read from the 'pyproject.toml' file.
* Change collections.abc.Iterable to (list, dict)
I also moved where the conversion happens... again. I am rather indecisive
if you haven't noticed. It should be better as it takes place in the
parse_pyproject_toml logic where configuration modification already takes
place.
Actually when this PR was first created I had the conversion happen in that
return statement, but the target_version check was complaining about it being
a string. So I moved the conversion after that check, but then Click didn't
like the stringifed list, which led me to check whether the value was an
instance of an Iterable before turning it into a string. And... I forgot that
type checking before conversion would allow it to work before the
target_version check anyway.
- Leave logic to still allow for formatting changes to be ignored
- Now just capture the output of any other error that has a > 1 returncode
- Raise on anything else
Test: Add unit test to exercise this new logic
* Add all listed by projects into primer.json + Enable on CI Runs
- Change workers default to 2 as black uses system CPU count
- Increase timeout to 5 mins for subprocess black runs
- Takes about 120s for 13 (3 disabled) projects on my 2018 Macbook Pro
- I was not removing directories tho ...
Will open an issue to investigate the failing projects and make this run cleaner.
- Once we get more stable we can expect more repos to be black formatted
Run it:
- `black-primer -k -w /tmp/primer_large_test --debug --rebase`
```
[2020-05-20 21:44:01,273] DEBUG: Starting /Users/cooper/venvs/b/bin/black-primer (cli.py:125)
[2020-05-20 21:44:01,273] DEBUG: Using selector: KqueueSelector (selector_events.py:53)
[2020-05-20 21:44:01,274] INFO: 16 projects to run Black over (lib.py:276)
[2020-05-20 21:44:01,274] DEBUG: Using 2 parallel workers to run Black (lib.py:281)
[2020-05-20 21:44:01,274] DEBUG: worker 0 workng on aioexabgp (lib.py:215)
[2020-05-20 21:44:01,276] DEBUG: worker 1 workng on attrs (lib.py:215)
[2020-05-20 21:44:02,443] INFO: Finished aioexabgp (lib.py:249)
[2020-05-20 21:44:02,443] DEBUG: worker 0 workng on bandersnatch (lib.py:215)
[2020-05-20 21:44:04,409] INFO: Finished bandersnatch (lib.py:249)
[2020-05-20 21:44:04,409] DEBUG: worker 0 workng on channels (lib.py:215)
[2020-05-20 21:44:04,702] INFO: Finished attrs (lib.py:249)
[2020-05-20 21:44:04,702] DEBUG: worker 1 workng on django (lib.py:215)
[2020-05-20 21:44:04,702] INFO: Skipping django as it's disabled via config (lib.py:222)
[2020-05-20 21:44:04,702] DEBUG: worker 1 workng on flake8-bugbear (lib.py:215)
[2020-05-20 21:44:05,813] INFO: Finished channels (lib.py:249)
[2020-05-20 21:44:05,813] DEBUG: worker 0 workng on hypothesis (lib.py:215)
[2020-05-20 21:44:06,071] INFO: Finished flake8-bugbear (lib.py:249)
[2020-05-20 21:44:06,071] DEBUG: worker 1 workng on pandas (lib.py:215)
[2020-05-20 21:44:06,071] INFO: Skipping pandas as it's disabled via config (lib.py:222)
[2020-05-20 21:44:06,071] DEBUG: worker 1 workng on poetry (lib.py:215)
[2020-05-20 21:44:16,207] INFO: Finished hypothesis (lib.py:249)
[2020-05-20 21:44:16,207] DEBUG: worker 0 workng on ptr (lib.py:215)
[2020-05-20 21:44:17,077] INFO: Finished poetry (lib.py:249)
[2020-05-20 21:44:17,077] DEBUG: worker 1 workng on pyramid (lib.py:215)
[2020-05-20 21:44:17,460] INFO: Finished ptr (lib.py:249)
[2020-05-20 21:44:17,460] DEBUG: worker 0 workng on pytest (lib.py:215)
[2020-05-20 21:44:17,460] INFO: Skipping pytest as it's disabled via config (lib.py:222)
[2020-05-20 21:44:17,460] DEBUG: worker 0 workng on sqlalchemy (lib.py:215)
[2020-05-20 21:44:33,319] INFO: Finished pyramid (lib.py:249)
[2020-05-20 21:44:33,319] DEBUG: worker 1 workng on tox (lib.py:215)
[2020-05-20 21:44:42,274] INFO: Finished tox (lib.py:249)
[2020-05-20 21:44:42,275] DEBUG: worker 1 workng on virtualenv (lib.py:215)
[2020-05-20 21:44:47,928] INFO: Finished virtualenv (lib.py:249)
[2020-05-20 21:44:47,928] DEBUG: worker 1 workng on warehouse (lib.py:215)
[2020-05-20 21:45:16,784] INFO: Finished warehouse (lib.py:249)
[2020-05-20 21:45:16,784] DEBUG: project_runner 1 exiting (lib.py:213)
[2020-05-20 21:45:45,700] INFO: Finished sqlalchemy (lib.py:249)
[2020-05-20 21:45:45,700] DEBUG: project_runner 0 exiting (lib.py:213)
[2020-05-20 21:45:45,701] INFO: Analyzing results (lib.py:292)
-- primer results 📊 --
13 / 16 succeeded (81.25%) ✅
0 / 16 FAILED (0.0%) 💩
- 3 projects disabled by config
- 0 projects skipped due to Python version
- 0 skipped due to long checkout
```
* Move to partial for rmtree + specify a onerror handler for PermissionError on Windows for git
* Set default coding to utf8 for very important emoji's on Windows
* Set Python encoding to utf-8 for Windows
* Appease the white space gods of Black!
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
* Add black-primer unittests
- Get this tool covered with some decent unittests for all unittests wins
- Have a CLI and lib test class
- Import it from `test_black.py` so we always run tests
- Revert typing asyncio.Queue as Queue[str] so we can work in 3.6
- **mypy**: Until black > 3.6 disallow_any_generics=False for primer code
Test:
- Run tests: `coverage run tests/test_primer.py` or `coverage run -m unittest`
```
(b) cooper-mbp1:black cooper$ coverage report
Name Stmts Miss Cover
---------------------------------------------
src/black_primer/cli.py 49 8 84%
src/black_primer/lib.py 148 28 81%
tests/test_primer.py 114 1 99%
---------------------------------------------
TOTAL 311 37 88%
```
* Use ProactorEventLoop for Windows + fix false path for Linux
* Set Windows to use ProactorEventLoop in to benefit all callers
* sys.platform seems to not having the loop applied - So type ignore and use platform.system() gate
* Have each test loop correctly set to ProactorEventLoop on Windows for < 3.8 too
* Add primer CI tool 💩
- Run in PATH `black` binary on configured projects
- Can set wether we expect changes or not per project
- Can set what python versions are supported for a project
- if `long_checkout` True project will not be ran on CI
Will add to CI after I finish unit tests to avoid silly bugs I'm sure I have 🤪
Tests:
- Manual Run - Will add unit tests if people think it will be useful
- Output:
```shell
(b) cooper-mbp1:black cooper$ time /tmp/b/bin/black-primer -k -w /tmp/cooper_primer_1
[2020-05-10 08:48:25,696] INFO: 4 projects to run black over (lib.py:212)
[2020-05-10 08:48:25,697] INFO: Skipping aioexabgp as it's disabled via config (lib.py:166)
[2020-05-10 08:48:25,699] INFO: Skipping bandersnatch as it's disabled via config (lib.py:166)
[2020-05-10 08:48:28,676] INFO: Analyzing results (lib.py:225)
-- primer results 📊 --
2 / 4 succeeded (50.0%) ✅
0 / 4 FAILED (0.0%) 💩
- 2 projects Disabled by config
- 0 projects skipped due to Python Version
- 0 skipped due to long checkout
real 0m3.304s
user 0m9.529s
sys 0m1.019s
```
- ls of /tmp/cooper_primer_1
```
(b) cooper-mbp1:black cooper$ ls -lh /tmp/cooper_primer_1
total 0
drwxr-xr-x 21 cooper wheel 672B May 10 08:48 attrs
drwxr-xr-x 14 cooper wheel 448B May 10 08:48 flake8-bugbear
```
* Address mypy 3.6 type errors
- Don't use asyncio.run() ... go back to the past :P
- Refactor results into a named tuple of two dicts to avoid typing nightmare
- Fix some variable names
- Fix bug with rebase logic in git_checkout_or_rebase
* Prettier the JSON config file for primer
* Delete projects when finished, move dir to be timestamped + shallow copy
* Re-enable disabled projects post @JelleZijlstra's docstring fix
* Workaround for future annotations until someone tells me the correct fix
Termux's Python environment doesn't provide sem_open, but fails with a
nested `ImportError` on import attempts:
ImportError: cannot import name 'SemLock' from '_multiprocessing'
This updates the existing handling for AWS Lambda to catch both
`OSError` and `ImportError`.
- Move black.py to src/black/__init__.py
- Have setuptools_scm make src/_black_version.py and exclude from git
- Move blackd.py to src/blackd/__init__.py
- Move blib2to3/ to src/
- Update `setup.py`
- Update unittests to pass
- Mostly path fixing + resolving
- Update CI
- pre-commit config
- appveyor + travis
Tested on my mac with python 3.7.5 via:
```
python3 -m venv /tmp/tb3
/tmp/tb3/bin/pip install --upgrade setuptools pip coverage pre-commit
/tmp/tb2/bin/pip install ~/repos/black/
cd ~/repos/black/
/tmp/tb2/bin/coverage run tests/test_black.py
/tmp/tb3/bin/pre-commit run -a
/tmp/tb3/bin/black --help
/tmp/tb3/bin/black ~/repos/ptr/ptr.py
```