This also introduces a script so we can reference the latest version in
the example pre-commit configuration in the docs without forgetting to
update it when doing a release!
Commit history before merge:
* document jupyter hook
* note minimum version
* add check for pre-commit version
* use git tag
* curl api during ci
* parse version from changes file
* fixup script
* rename variables
* Tweak the docs & magical script
* fix couple of typos
* pin additional dependencies in hook
* Add types-PyYAML to lockfile
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
Implementation stolen from PR davidhalter/parso#162. Thanks parso!
I could add support for these newer syntactical constructs in the
target version detection logic, but until I get diff-shades up
and running I don't feel very comfortable adding the code.
This fixes a bug where a trailing comma would be added to a
parenthesized return annotation changing its type to a tuple.
Here's one case where this bug shows up:
```
def spam() -> (
this_is_a_long_type_annotation_which_should_NOT_get_a_trailing_comma
):
pass
```
The root problem was that the type annotation was treated as if it was
a parameter & import list (is_body=True to linegen::bracket_split_build_line)
where a trailing comma is usually fine. Now there's another check in the
aforementioned function to make sure the body it's operating on isn't
a return annotation before truly adding a trailing comma.
* Add CPython repository into primer runs
- CPython tests is probably the best repo for black to test on as the stdlib's unittests should use all syntax
- Limit to running in recent versions of the python runtime - e.g. today >= 3.9
- This allows us to parse more syntax
- Exclude all failing files for now
- Definitely have bugs to explore there - Refer to #2407 for more details there
- Some test files on purpose have syntax errors, so we will never be able to parse them
- Add new black command arguments logging in debug mode; very handy for seeing how CLI arguments are formatted
CPython now succeeds ignoring 16 files:
```
Oh no! 💥💔💥
1859 files would be reformatted, 148 files would be left unchanged.
```
Testing
- Ran locally with and without string processing - Very little runtime difference BUT 3 more failed files
```
time /tmp/tb/bin/black --experimental-string-processing --check . 2>&1 | tee /tmp/black_cpython_esp
...
Oh no! 💥💔💥
1859 files would be reformatted, 148 files would be left unchanged, 16 files would fail to reformat.
real 4m8.563s
user 16m21.735s
sys 0m6.000s
```
- Add unittest for new covienence config file flattening that allows long arguments to be broke up into an array/list of strings
Addresses #2407
---
Commit history before merge:
* Add new `timeout_seconds` support into primer.json
- If present, will set forked process limit to that value in seconds
- Otherwise, stay with default 10 minutes (600 seconds)
* Add new "base_path" concept to black-primer
- Rather than start at the repo root start at a configured path within the repository
- e.g. for cpython only run black on `Lib`
* Disable by default - It's too much for GitHub Actions. But let's leave config for others to use
* Minor tweak to _flatten_cli_args
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
The fix for #1688 in #1761 breaks help("modules") introspection and also leads
to unhappy results when inadvertently importing blackd from Python. Basically
the sys.exit(-1) causes the whole Python REPL to exit -- not great to suffice.
Commit history before merge:
* Change sys.exit to Raise.
* Add #2440 to changelog.
* Fix lint error from prettier
* Remove exception chain for more helpful user message.
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
While this development environment / requirements situation is a mess,
let's at least make it consistent. We're effectively supporting two
modes of development in this project, 1) tox based dev commands
(e.g. `tox -e fuzz`) that are dead simple to use, and 2) manual dev
commands (e.g. `pytest -n auto`) that give more control and are usually
faster.
Right now the Pipfile.lock based development environment is incomplete
missing the test requirements specified in ./test_requirements.txt.
This is annoying since manual test commands (e.g. `pytest -k fmtonoff`)
fail. Let's fix this by making Pipfile.lock basically a
"everything you need" requirements file (fuzzing not included since
running it locally is not something common).
Oh and let's bump some documentation deps (and bring some requirements
across .pre-commit-config.yaml, Pipfile, and docs/requirement.txt in
alignment again). Don't worry, I tested these changes so they should
be fine (hopefully!).
we don't accidentally add backslashes to them when normalizing quotes
because that's invalid syntax!
The problem this commit fixes is that matches would eat too much
blocking important matches to occur. For example, here's one f-string
body:
{a}{b}{c}
I know there's no risk of introducing backslashes here, but the regex
already goes sideways with this. Throwing this example at regex101
I get:
{a}{b}{c} # The As and Bs are the two matches, and the upper
---- ---- # case letters are the groups with those matches.
aAaa bbBb
... we've missed the middle expression (so if any backslashes in a
more complex example were introduced there we wouldn't bail out
even though we should -- hence the bug). As it stands the regex
needs somesort of extra character (or the start/end of the body)
around the expressions but that isn't always the case as shown
above.
The fix implemented here is to turn the "eat a surrounding non-curly
bracket character" groups ie. `(?:[^{]|^)` and `(?:[^}]|$)` into
negative lookaheads and lookbehinds. This still guarantees the
already specified rules but without problematically eating extra
characters ^^
Fixes#2359.
This commit now makes Black exit with an user-friendly error message if a
.gitignore file couldn't be parsed -- a massive improvement over an opaque
traceback!
* Remove `language_version` for pre-commit
At my company, we set the Python version in `default_language_version`
in each repo's `.pre-commit-config.yaml`,
so that all hooks are running with the same Python version.
However, this currently doesn't work for black,
as the `language_version` specified here
in the upstream `.pre-commit-hooks.yaml` takes precedence.
Currently, this requires us to manually set `language_version`
specifically for black,
duplicating the value from `default_language_version`.
The failure mode otherwise is subtle -
black works most of the time,
but try to add a walrus operator and it suddenly breaks!
Given that black's `setup.py` already has `python_requires>=3.6.2`,
specifying that `python3` must be used here isn't needed
as folks inadvertently using Python 2 will get hook-install-time failures anyways.
Remove the `language_version` from these upstream hook configs
so that users of black are able to use `default_language_version`
and have it apply to all their hooks, black included.
Example `.pre-commit-config.yaml` before:
```
default_language_version:
python: python3.8
repos:
- repo: https://github.com/psf/black
rev: 21.7b0
hooks:
- id: black
language_version: python3.8
```
After:
```
default_language_version:
python: python3.8
repos:
- repo: https://github.com/psf/black
rev: 21.7b0
hooks:
- id: black
```
* Add changelog entry
To summarise, based on what was discussed in that issue:
due to not being able to parse automagics (e.g. pip install black)
without a running IPython kernel, cells with syntax which is parseable
by neither ast.parse nor IPython will be skipped cells with multiline
magics will be skipped trailing semicolons will be preserved, as they
are often put there intentionally in Jupyter Notebooks to suppress
unnecessary output
Commit history before merge (excluding merge commits):
* wip
* fixup tests
* skip tests if no IPython
* install test requirements in ipynb tests
* if --ipynb format all as ipynb
* wip
* add some whole-notebook tests
* docstrings
* skip multiline magics
* add test for nested cell magic
* remove ipynb_test.yml, put ipynb tests in tox.ini
* add changelog entry
* typo
* make token same length as magic it replaces
* only include .ipynb by default if jupyter dependencies are found
* remove logic from const
* fixup
* fixup
* re.compile
* noop
* clear up
* new_src -> dst
* early exit for non-python notebooks
* add non-python test notebook
* add repo with many notebooks to black-primer
* install extra dependencies for black-primer
* fix planetary computer examples url
* dont run on ipynb files by default
* add scikit-lego (Expected to change) to black-primer
* add ipynb-specific diff
* fixup
* run on all (including ipynb) by default
* remove --include .ipynb from scikit-lego black-primer
* use tokenize so as to mirror the exact logic in IPython.core.displayhooks quiet
* fixup
* 🎨
* clarify docstring
* add test for when comment is after trailing semicolon
* enumerate(reversed) instead of [::-1]
* clarify docstrings
* wip
* use jupyter and no_jupyter marks
* use THIS_DIR
* windows fixup
* perform safe check cell-by-cell for ipynb
* only perform safe check in ipynb if not fast
* remove redundant Optional
* 🎨
* use typeguard
* dont process cell containing transformed magic
* require typing extensions before 3.10 so as to have TypeGuard
* use dataclasses
* mention black[jupyter] in docs as well as in README
* add faq
* add message to assertion error
* add test for indented quieted cell
* use tokenize_rt else we cant roundtrip
* fmake fronzet set for tokens to ignore when looking for trailing semicolon
* remove planetary code examples as recent commits result in changes
* use dataclasses which inherit from ast.NodeVisitor
* bump typing-extensions so that TypeGuard is available
* bump typing-extensions in Pipfile
* add test with notebook with empty metadata
* pipenv lock
* deprivative validate_cell
* Update README.md
* Update docs/getting_started.md
* dont cache notebooks if jupyter dependencies arent found
* dont write to cache if jupyter deps are not installed
* add notebook which cant be parsed
* use clirunner
* remove other subprocess calls
* add docstring
* make verbose and quiet keyword only
* 🎨
* run second many test on directory, not on file
* test for warning message when running on directory
* early return from non-python cell magics
* move NothingChanged to report to avoid circular import
* remove circular import
* reinstate --ipynb flag
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
The template weren't applying the default labels ever since I renamed
the labels.
There has been enough issues about documentation opened recently so it's
probably worth a template for it.
* Update CHANGES.md for 21.7b0 release
* move some changes to the right section
* another one
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
They seem to be used as test cases for a specific region of formatting
that was slow. Now performance testing is probably something end users
won't be needing to do, so this is an easy way of reducing the sdist
size sigificantly.
toml unfortunately has a lack of maintainership issue right now. It's
evident by the fact toml only supports TOML v0.5.0. TOML v1.0.0 has
been recently released and right now Black crashes hard on its usage.
tomli is a brand new parse only TOML library. It supports TOML
v1.0.0. Although TBH we're switching to this one mostly because
pip is doing the same.
*The upper bound was included at the library maintainer's request.
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Taneli Hukkinen <3275109+hukkin@users.noreply.github.com>
Commit history before merge:
* Accept empty stdin (close#2337)
* Update tests/test_black.py
* Add changelog
* Assert Black reformats an empty string to an empty string (#2337) (#2346)
* fix
Click types have been moved to click repo itself. See pallets/click#1856
I've had some issues with typeshed types being outdated in another project
so might be good to avoid that here.
Commit history before merge:
* Get `click` types from main repo
* Fix mypy errors
* Require click v8 for type annotations
* Update Pipfile
This commit fixes parsing of the skip-string-normalization option in vim
plugin. Originally, the plugin read the string-normalization option,
which does not exist in help (--help) and it's not respected by black
on command line.
Commit history before merge:
* fix string normalization option in vim plugin
* fix string normalization option in vim plugin
* Finish and fix patch (thanks Matt Wozniski!)
FYI: this is totally the work and the comments below of Matt (AKA godlygeek)
This fixes two entirely different problems related to how pyproject.toml
files are handled by the vim plugin.
=== Problem #1 ===
The plugin fails to properly read boolean values from pyproject.toml.
For instance, if you create this pyproject.toml:
```
[tool.black]
quiet = true
```
the Black CLI is happy with it and runs without any messages, but the
:Black command provided by this plugin fails with:
```
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "<string>", line 102, in Black
File "<string>", line 150, in get_configs
File "<string>", line 150, in <dictcomp>
File "/usr/lib/python3.6/distutils/util.py", line 311, in strtobool
val = val.lower()
AttributeError: 'bool' object has no attribute 'lower'
```
That's because the value returned by the toml.load() is already a
bool, but the vim plugin incorrectly tries to convert it from a str to a bool.
The value returned by toml_config.get() was always being passed to
flag.cast(), which is a function that either converts a string to an
int or a string to a bool, depending on the flag. vim.eval()
returns integers and strings all as str, which is why we need the cast,
but that's the wrong thing to do for values that came from toml.load().
We should be applying the cast only to the return from vim.eval()
(since we know it always gives us a string), rather than casting the
value that toml.load() found - which is already the right type.
=== Problem #2 ===
The vim plugin fails to take the value for skip_string_normalization
from pyproject.toml. That's because it looks for a string_normalization
key instead of a skip_string_normalization key, thanks to this line
saying the name of the flag is string_normalization:
black/autoload/black.vim (line 25 in 05b54b8)
```
Flag(name="string_normalization", cast=strtobool),
```
and this dictcomp looking up each flag's name in the config dict:
black/autoload/black.vim (lines 148 to 151 in 05b54b8)
```
return {
flag.var_name: flag.cast(toml_config.get(flag.name, vim.eval(flag.vim_rc_name)))
for flag in FLAGS
}
```
For the second issue, I think I'd do a slightly different patch. I'd
keep the change to invert this flag's meaning and change its name that
this PR proposes, but I'd also change the handling of the
g:black_skip_string_normalization and g:black_string_normalization
variables to make it clear that g:black_skip_string_normalization is
the expected name, and g:black_string_normalization is only checked
when the expected name is unset, for backwards compatibility.
My proposed behavior is to check if g:black_skip_string_normalization
is defined and to define it if not, using the inverse of
g:black_string_normalization if that is set, and otherwise to the
default of 0. The Python code in autoload/black.vim runs later, and
will use the value of g:black_skip_string_normalization (and ignore
g:black_string_normalization; it will only be used to set
g:black_skip_string_normalization if it wasn't already set).
---
Co-authored-by: Matt Wozniski <mwozniski@bloomberg.net>
* Fix plugin/black.vim (need to up my vim game)
Co-authored-by: Matt Wozniski <godlygeek@gmail.com>
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
Co-authored-by: Matt Wozniski <mwozniski@bloomberg.net>
Co-authored-by: Matt Wozniski <godlygeek@gmail.com>
Commit history before merge:
* Find pyproject from vim relative to current file
* Merge remote-tracking branch 'upstream/main' into find-pyproject-vim
* Finish and fix this patch (thanks Matt Wozniski!)
Both the existing code and the proposed code are broken.
The vim.eval() call (whether it's vim.eval("@%") or
vim.eval("fnamemodify(getcwd(), ':t')) returns a string, and it passes
that string to find_pyproject_toml, which expects a sequence of strings,
not a single string, and - since a string is a sequence of single
character strings - it gets turned into a list of ridiculous paths. I
tested with a file called foo.py, and added a print(path_srcs) into
find_project_root, which printed out:
[
PosixPath('/home/matt/f'),
PosixPath('/home/matt/o'),
PosixPath('/home/matt/o'),
PosixPath('/home/matt'),
PosixPath('/home/matt/p'),
PosixPath('/home/matt/y')
]
This does work for an unnamed buffer, too - we wind up calling
black.find_pyproject_toml(("",)), and that winds up prepending the
working directory to any relative paths, so "" just gets turned into
the current working directory.
Note that find_pyproject_toml needs to be passed a 1-tuple, not a
list, because it requires something hashable (thanks to
functools.lru_cache being used)
Co-authored-by: Matt Wozniski <mwozniski@bloomberg.net>
* I forgot the CHANGELOG entry ... again
* I'm really bad at dealing with merge conflicts sometimes
* Be more correct describing search behaviour
Co-authored-by: Austin Glaser <austin.glaser@spacex.com>
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
Co-authored-by: Matt Wozniski <mwozniski@bloomberg.net>
* Add STDIN test to primer
- Check that out STDIN black support stays working
- Add asyncio.subprocess STDIN pip via communicate
- We just check we format python code from primer's `lib.py`
Fixes#2310
`black.strings.get_string_prefix` used to lowercase the extracted
prefix before returning it. This is wrong because 1) it ignores the
fact we should leave R prefixes alone because of MagicPython, and 2)
there is dedicated prefix casing handling code that fixes issue 1.
`.lower` is too naive.
This was originally fixed in 20.8b0, but was reintroduced since 21.4b0.
I also added proper prefix normalization for docstrings by using the
`black.strings.normalize_string_prefix` helper.
Some more test strings were added to make sure strings with capitalized
prefixes aren't treated differently (actually happened with my original
patch, Jelle had to point it out to me).
