While this development environment / requirements situation is a mess,
let's at least make it consistent. We're effectively supporting two
modes of development in this project, 1) tox based dev commands
(e.g. `tox -e fuzz`) that are dead simple to use, and 2) manual dev
commands (e.g. `pytest -n auto`) that give more control and are usually
faster.
Right now the Pipfile.lock based development environment is incomplete
missing the test requirements specified in ./test_requirements.txt.
This is annoying since manual test commands (e.g. `pytest -k fmtonoff`)
fail. Let's fix this by making Pipfile.lock basically a
"everything you need" requirements file (fuzzing not included since
running it locally is not something common).
Oh and let's bump some documentation deps (and bring some requirements
across .pre-commit-config.yaml, Pipfile, and docs/requirement.txt in
alignment again). Don't worry, I tested these changes so they should
be fine (hopefully!).
To summarise, based on what was discussed in that issue:
due to not being able to parse automagics (e.g. pip install black)
without a running IPython kernel, cells with syntax which is parseable
by neither ast.parse nor IPython will be skipped cells with multiline
magics will be skipped trailing semicolons will be preserved, as they
are often put there intentionally in Jupyter Notebooks to suppress
unnecessary output
Commit history before merge (excluding merge commits):
* wip
* fixup tests
* skip tests if no IPython
* install test requirements in ipynb tests
* if --ipynb format all as ipynb
* wip
* add some whole-notebook tests
* docstrings
* skip multiline magics
* add test for nested cell magic
* remove ipynb_test.yml, put ipynb tests in tox.ini
* add changelog entry
* typo
* make token same length as magic it replaces
* only include .ipynb by default if jupyter dependencies are found
* remove logic from const
* fixup
* fixup
* re.compile
* noop
* clear up
* new_src -> dst
* early exit for non-python notebooks
* add non-python test notebook
* add repo with many notebooks to black-primer
* install extra dependencies for black-primer
* fix planetary computer examples url
* dont run on ipynb files by default
* add scikit-lego (Expected to change) to black-primer
* add ipynb-specific diff
* fixup
* run on all (including ipynb) by default
* remove --include .ipynb from scikit-lego black-primer
* use tokenize so as to mirror the exact logic in IPython.core.displayhooks quiet
* fixup
* 🎨
* clarify docstring
* add test for when comment is after trailing semicolon
* enumerate(reversed) instead of [::-1]
* clarify docstrings
* wip
* use jupyter and no_jupyter marks
* use THIS_DIR
* windows fixup
* perform safe check cell-by-cell for ipynb
* only perform safe check in ipynb if not fast
* remove redundant Optional
* 🎨
* use typeguard
* dont process cell containing transformed magic
* require typing extensions before 3.10 so as to have TypeGuard
* use dataclasses
* mention black[jupyter] in docs as well as in README
* add faq
* add message to assertion error
* add test for indented quieted cell
* use tokenize_rt else we cant roundtrip
* fmake fronzet set for tokens to ignore when looking for trailing semicolon
* remove planetary code examples as recent commits result in changes
* use dataclasses which inherit from ast.NodeVisitor
* bump typing-extensions so that TypeGuard is available
* bump typing-extensions in Pipfile
* add test with notebook with empty metadata
* pipenv lock
* deprivative validate_cell
* Update README.md
* Update docs/getting_started.md
* dont cache notebooks if jupyter dependencies arent found
* dont write to cache if jupyter deps are not installed
* add notebook which cant be parsed
* use clirunner
* remove other subprocess calls
* add docstring
* make verbose and quiet keyword only
* 🎨
* run second many test on directory, not on file
* test for warning message when running on directory
* early return from non-python cell magics
* move NothingChanged to report to avoid circular import
* remove circular import
* reinstate --ipynb flag
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
toml unfortunately has a lack of maintainership issue right now. It's
evident by the fact toml only supports TOML v0.5.0. TOML v1.0.0 has
been recently released and right now Black crashes hard on its usage.
tomli is a brand new parse only TOML library. It supports TOML
v1.0.0. Although TBH we're switching to this one mostly because
pip is doing the same.
*The upper bound was included at the library maintainer's request.
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
Co-authored-by: Taneli Hukkinen <3275109+hukkin@users.noreply.github.com>
Click types have been moved to click repo itself. See pallets/click#1856
I've had some issues with typeshed types being outdated in another project
so might be good to avoid that here.
Commit history before merge:
* Get `click` types from main repo
* Fix mypy errors
* Require click v8 for type annotations
* Update Pipfile
So these won't go out of date. This does mean the environment has be
setup a bit more carefully so the right version of the tool is used,
but thankfully the build environment is rebuilt on change on RTD anyway.
Also since the HTML docs are known to build fine, let's provide
downloadable HTMLzips of our docs.
This change needs RTD and GH to install Black with the [d] extra so
blackd's help can generated. While editing RTD's config file, let's
migrate the file to a non-deprecated filename.
Also I missed adding AUTHORS.md to the files key in the doc GHA config.
I know I know, this is the second reorganization of the docs. I'm not
saying the first one was bad or anything... but.. actually wait nah,
*it was bad*.
Anyway, welcome to probably my biggest commit. The main thing with this
reorganization was to introduce nesting to the documentation! Having
all of the docs be part of the main TOC was becoming too much. There
wasn't much room to expand either. Finally, the old setup required
a documentation generation step which was just annoying.
The goals of this reorganization was to:
1. Significantly restructure the docs to be discoverable and
understandable
2. Add room for further docs (like guides or contributing docs)
3. Get rid of the doc generation step (it was slow and frustrating)
4. Unblock other improvements and also just make contributing to the
docs easier
Another important change with this is that we are no longer using GitHub
as a documentation host. While GitHub does support Markdown based docs
actually pretty well, the lack of any features outside of GitHub Flavoured
Markdown is quite limiting. ReadTheDocs is just much better suited for
documentation. You can use reST, MyST, CommonMark, and all of their
great features like toctrees and admonitions.
Related to this change, we're adopting MyST as our flavour of Markdown.
MyST introduces neat syntax extensions to Markdown that pretty much
gives us the best of both worlds. The ease of use and simplicity of MD
and the flexibility and expressiveness of reST. Also recommonmark is
deprecated now. This switch was possible now we don't use GH as a docs
host. MyST docs have to be built to really be usable / pretty, so the MD
docs are going to look pretty bad on GH, but that's fine now!
Another thing that should be noted is that the README has been stripped
of most content since it was confusing. Users would read the README and
then think some feature or bug was fixed already and is available in a
release when in reality, they weren't. They were reading effectively
the latest docs without knowing.
See also: https://github.com/psf/black/issues/1759
FYI: CommonMark is a rationalized version of Markdown syntax
--
Commit history before merge:
* Switch to MyST-Parser + doc config cleanup
recommonmark is being deprecated in favour of MyST-Parser. This change
is welcomed, especially since MyST-Parser has syntax extensions for the
Commonmark standard. Effectively we get to use a language that's powerful
and expressive like ReST, but get the simplicity of Markdown.
The rest of this effort will be using some MyST features.
This reorganization efforts aims to remove as much duplication as possible.
The regeneration step once needed is gone, significantly simplifing our
Sphinx documentation configuration.
* Tell pipenv we replaced recommonmark for MyST-Parser
Also update `docs/requirements.txt`
* Delete all auto generated content
* Switch prettier for mdformat (plus a few plugins)
**FYI: THIS WAS EFFECTIVELY REVERTED, SEE THIRD TO LAST COMMIT**
prettier doesn't support MyST's syntax extensions which are going to be
used in this reorganization effort so we have to switch formatter.
Unfortanately mdformat's style is different from prettier's so time to
reformat the whole repo too.
We're excluding .github/ISSUE_TEMPLATE because I have no idea whether
its changes are safe, so let's play it safe.
* Fix the heading levels in CHANGES.md + a link
MyST-Parser / sphinx's linkcheck complains otherwise.
* Move reference docs into a docs/contributing dir
They're for contributors of Black anyway. Also added a note in the
summary document warning about the lack of attention the reference has
been dealing with.
* Rewrite and setup the new landing page + main TOC
- add some more detail about Black's beta status
- add licensing info
- add external links in the main TOC for GitHub, PyPI, and IRC
- prepare main TOC for new structure
* Break out AUTHORS into its own file
Not only was the AUTHORS list quite long, this makes it easy to include
it in the Sphinx docs with just a simple symlink.
* Add license to docs via a simple include
Yes the document is orphaned but it is linked to in the landing page
(docs/index.rst).
* Add "The Black Code Style" section
This mostly was a restructuring commit, there has been a few updates but
not many. The main goal was to split "current style" and "planned
changes to the style that haven't happened yet" to avoid confusion.
* Add "Getting Started" page
This is basically a quick start + even more. This commit is certainly
one of most creatively involved in this effort.
* Add "Usage and Configuration" section
This commit was as much restructuring as new content. Instead of being
in one giant file, usage and configuration documentation can expand
without bloating a single file.
* Add "Integrations" section
Just a restructuring commit ...
* Add "Guides" section
This is a promising area of documentation that could easily grow in the
future, let's prepare for that!
* Add "Contributing" section
This is also another area that I expect to see significant growth in.
Contributors to Black could definitely do with some more specific docs
that clears up certain parts of our slightly confusing project (it's
only confusing because we're getting big and old!).
* Rewrite CONTRIBUTING.md to just point to RTD
* Rewrite README.md to delegate most info to RTD
* Address feedback + a lot of corrections and edits
I know I said I wanted to do these after landing this but given there's
going to be no time between this being merged and a release getting
pushed, I want these changes to make it in.
- drop the number flag for mdformat - to reduce diffs, see also:
https://mdformat.readthedocs.io/en/stable/users/style.html#ordered-lists
- the GH issue templates should be safe by mdformat, so get rid of the
exclude
- clarify our configuration position - i.e. stop claiming we don't have
many options, instead say we want as little formatting knobs as
possible
- lots and lots of punctuation, spelling, and grammar corrections (thanks
Jelle!)
- use RTD as the source for the CHANGELOG too
- visual style cleanups
- add docs about our .gitignore behaviour
- expand GHA Action docs
- clarify we want the PR number in the CHANGELOG entry
- claify Black's behaviour for with statements post Python 3.9
- italicize a bunch of "Black"s
Thank you goes to Jelle, Taneli (hukkinj1 on GH), Felix
(felix-hilden on GH), and Wouter (wbolster on GH) for the feedback!
* Merge remote-tracking branch 'upstream/master' into reorganize-docs-v2
merge conflicts suck, although these ones weren't too bad.
* Add changelog entry + fix merge conflict resolution error
I consider this important enough to be worthy of a changelog entry :)
* Merge branch 'master' into reorganize-docs-v2
Co-authored-by: Łukasz Langa <lukasz@langa.pl>
* Actually let's continue using prettier
Prettier works fine for all of the default MyST syntax so let's not
rock the boat as much. Dropping the mdformat commit was merge-conflict
filled so here's additional commit instead.
* Address Cooper's, Taneli's, and Jelle's feedback
Lots of wording improvements by Cooper. Taneli suggested to disable the
enabled by default MyST syntax not supported by Prettier and I agreed.
And Jelle found one more spelling error!
* More minor fixes
This is a follow-up of #2203 that uses a pytest marker instead of a bunch of
`skipUnless`. Similarly to the Python 2 tests, they are running by default and
will crash on an unsuspecting contributor with missing dependencies. This is
by design, we WANT contributors to test everything. Unless we actually don't
and then we can run:
pytest --run-optional=no_blackd
Relatedly, bump required aiohttp to 3.6.0 at least to get rid of expected
failures on Python 3.8 (see 6b5eb7d465).
flake8-mypy is long dead and shouldn't be used, see
https://github.com/ambv/flake8-mypy. We appear to use pre-commit to run
mypy now anyway.
I ran `pipenv uninstall flake8-mypy`, which seems to have made several
changes to Pipfile.lock. Let me know if there's a better way to do this.
Co-authored-by: hauntsaninja <>
* Bump toml from 0.10.0 to 0.10.1 to fix a bug
* Add tests for TOML parsing and reading
* Fix configuration bug affecting vim plugin
The vim plugin directly calls parse_pyproject and skips the Click processing
, but parse_pyproject assumed that it would only be used before Click processing
and therefore made the config values click friendly. This moves the "make the values
click friendly processing" into read_pyproject_toml which is only called by a Click
callback.
* Please mypy and flake8
* Make most of blib2to3 directly typed and mypyc-compatible
This used a combination of retype and pytype's merge-pyi to do the
initial merges of the stubs, which then required manual tweaking to
make actually typecheck and work with mypyc.
Co-authored-by: Sanjit Kalapatapu <sanjitkal@gmail.com>
Co-authored-by: Michael J. Sullivan <sully@msully.net>
* Make black able to compile and run with mypyc
The changes made fall into a couple categories:
* Fixing actual type mistakes that slip through the cracks
* Working around a couple mypy bugs (the most annoying of which being
that we need to add type annotations in a number of places where
variables are initialized to None)
Co-authored-by: Sanjit Kalapatapu <sanjitkal@gmail.com>
Co-authored-by: Michael J. Sullivan <sully@msully.net>
* add test for special unicode symbol which usual re can not process correctly
add regex lib which supports unicode 12.1.0 standard
replace re usage in project in favor to regex
* #455 fix dependency
Black will cache already formatted files using their file size and
modification timestamp. The cache is per-user and will always be used
unless Black is used with --diff or with code provided via standard
input.
