* Do not move docker `latest_release` tag for Pre-Releases
- When we do a pre-release lets not move the latest_release tag
- This tag should only move on official real releases
Fixes#3453
* Make it prettier - TIL we format our yaml
* Remove separate 3.11 CI now deps support 3.11
- We can run everything now like all other stable versions of Python
- test in a 3.11 vent: `/tmp/tb/bin/tox -e py311,ci-py311`
```
py311: OK (28.99=setup[7.90]+cmd[5.29,0.66,6.94,6.08,1.89,0.24] seconds)
ci-py311: OK (30.33=setup[3.20]+cmd[3.66,0.31,17.43,4.60,0.90,0.23] seconds)
congratulations :) (59.35 seconds)
```
* Add to CHANGES.md
* Add fuzz run in 3.11
This implements PEP 621, obviating the need for `setup.py`, `setup.cfg`,
and `MANIFEST.in`. The build backend Hatchling (of which I am a
maintainer in the PyPA) is now used as that is the default in the
official Python packaging tutorial. Hatchling is available on all the
major distribution channels such as Debian, Fedora, and many more.
## Python support
The earliest supported Python 3 version of Hatchling is 3.7, therefore
I've also set that as the minimum here. Python 3.6 is EOL and other
build backends like flit-core and setuptools also dropped support.
Python 3.6 accounted for 3-4% of downloads in the last month.
## Plugins
Configuration is now completely static with the help of 3 plugins:
### Readme
hynek's hatch-fancy-pypi-readme allows for the dynamic construction of
the readme which was previously coded up in `setup.py`. Now it's simply:
```toml
[tool.hatch.metadata.hooks.fancy-pypi-readme]
content-type = "text/markdown"
fragments = [
{ path = "README.md" },
{ path = "CHANGES.md" },
]
```
### Versioning
hatch-vcs is currently just a wrapper around setuptools-scm (which
despite the legacy naming is actually now decoupled from setuptools):
```toml
[tool.hatch.version]
source = "vcs"
[tool.hatch.build.hooks.vcs]
version-file = "src/_black_version.py"
template = '''
version = "{version}"
'''
```
### mypyc
hatch-mypyc offers many benefits over the existing approach:
- No need to manually select files for inclusion
- Avoids the need for the current CI workaround for https://github.com/mypyc/mypyc/issues/946
- Intermediate artifacts (like `build/`) from setuptools and mypyc
itself no longer clutter the project directory
- Runtime dependencies required at build time no longer need to be
manually redeclared as this is a built-in option of Hatchling
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Bumps cibuildwheel from 2.8.1 to 2.10.0 which has 3.11 building enabled
by default. Unfortunately mypyc errors out on 3.11:
src/black/files.py:29:9: error: Name "tomllib" already defined (by an import) [no-redef]
... so we have to also hide the fallback import of tomli on older 3.11
alphas from mypy[c].
* Move 311 tests to install aiohttp without C extensions
- Configure tox to install aiohttp without extensions
- i.e. use `AIOHTTP_NO_EXTENSIONS=1` for pip install
- This allows us to reenable blackd tests that use aiohttp testing helpers etc.
- Had to ignore `cgi` module deprecation warning
- Filed issue for aiohttp to fix: https://github.com/aio-libs/aiohttp/issues/6905
Test:
- `/tmp/tb/bin/tox -e 311`
* Fix formatting + linting
* Add latest aiohttp for loop fix + Try to exempt deprecation warning but failed - will ask for help
* Remove unnecessary warning ignore
Co-authored-by: Cooper Ry Lees <me@wcooperlees.com>
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
We've decided to a) convert stable back into a branch and b) to update
it immediately as part of the release process. We may as well automate
it. And about going back to a branch ...
Git tags are not the right tool, at all[^1]. They come with the
expectation that they will never change. Things will not work as
expected if they do change, doubly so if they change regularly. Once
you pull stable from the remote and it's copied in your local
repository, no matter how many times you run git pull you'll never see
it get updated automatically. Your only recourse is to delete the tag
via `git tag -d stable` before pulling.
This gets annoying really quickly since stable is supposed to be the
solution for folks "who want to move along as Black developers deem
the newest version reliable."[^2] See this comment for how this impacts
users using our Vim plugin[^3]. It also affects us developers[^4]. If
you have stable locally, once we cut a new release and update the stable
tag, a simple `git pull` / `git fetch` will not pull down the updated
stable tag. Unless you remember to delete stable before pulling, stable
will become stale and useless.
You can argue this is a good thing ("people should explicitly opt into
updating stable"), but IMO it does not match user expectations nor
developer expectations[^5]. Especially since not all our integrations
that use stable are bound by this security measure, for example our
GitHub Action (since it does a clean fetch of the repository every time
it's used). I believe consistency would be good here.
Finally, ever since we switched to a tag, we've been facing issues with
ReadTheDocs not picking up updates to stable unless we force a rebuild.
The initial rebuild on the stable update just pulls the commit the tag
previously pointed to. I'm not sure if switching back to a branch will
fix this, but I'd wager it will.
[^1]: https://git-scm.com/docs/git-tag#_on_re_tagging
[^2]: https://black.readthedocs.io/en/stable/contributing/release_process.html#moving-the-stable-tag
[^3]: https://github.com/psf/black/issues/2503#issuecomment-1196357379
[^4]: In fairness, most folks working on Black probably don't use the
`stable` ref anyway, especially us maintainers who'd know what is
the latest version by heart, but it'd still be nice to make it
usable for local dev though.
[^5]: Also what benefit does a `stable` ref have over explicit version
tags like `22.6.0`? If you're going to opt into some odd pin
mechanism, might as well use explicit version tags for clarity
and consistency.
Fixes#2734: a standalone comment causes strings to be merged into one far too long (and requiring two passes to do so).
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
- Had to exempt blackd tests for now due to aiohttp
- Skip by using `sys.version_info` tuple
- aiohttp does not compile in 3.11 yet - refer to #3230
- Add a deadsnakes ubuntu workflow to run 3.11-dev to ensure we don't regress
- Have it also format ourselves
Test:
- `tox -e 311`
Co-authored-by: Cooper Ry Lees <me@wcooperlees.com>
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
Building executables without any testing is quite sketchy, let's at
least verify they won't crash on startup and format Black's own
codebase.
Also replaced "binaries" with "executables" since it's clearer and
won't be confused with mypyc.
Finally, I added colorama so all Windows users can get colour.
* Add run_self environment in tox
* Add run_self task as part of the lint CI flow
* Remove hard coded sources list
* Remove black from pre-commit
Co-authored-by: Cooper Lees <me@cooperlees.com>
Now PRs will run two diff-shades jobs, "preview-changes" which formats
all projects with preview=True, and "assert-no-changes" which formats
all projects with preview=False. The latter also fails if any changes
were made.
Pushes to main will only run "preview-changes"
Also the workflow_dispatch feature was dropped since it was
complicating everything for little gain.
At the moment, it's just a source of spurious CI failures and busywork
updating the configuration file.
Unlike diff-shades, it is run across many different platforms and
Python versions, but that doesn't seem essential. We already run unit
tests across platforms and versions.
I chose to leave the code around for now in case somebody is using it,
but CI will no longer run it.
The recent 2021.4 release of pyinstaller-hooks-contrib now contains a
built-in hook for platformdirs. Manually specifying the hidden import
arg should no longer be needed.
Add new platformdirs dependencies as hidden imports when creating
PyInstaller-based binaries.
platformdirs imports the module for each platform dynamically, which
PyInstaller is unable to correctly detect for packing. By adding the
modules as hidden imports, we are telling PyInstaller to include the
modules in the packaged binary.
This issue seems to have been introduce when switching to platformdirs
in #2375. fixes#2464
Commit history before merge:
* Add hidden import to PyInstaller build
Add new platformdirs dependency as a hidden import when creating
PyInstaller based binaries.
* Only include the platformdirs for the relevant os
Draft releases don't trigger the workflows (that's good!) but since they only
Commit history before merge:
* fix: run pypi upload from published draft releases
* Fix broken task list markup in PR template
* change docker workflow to build on release publish
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
To summarise, based on what was discussed in that issue:
due to not being able to parse automagics (e.g. pip install black)
without a running IPython kernel, cells with syntax which is parseable
by neither ast.parse nor IPython will be skipped cells with multiline
magics will be skipped trailing semicolons will be preserved, as they
are often put there intentionally in Jupyter Notebooks to suppress
unnecessary output
Commit history before merge (excluding merge commits):
* wip
* fixup tests
* skip tests if no IPython
* install test requirements in ipynb tests
* if --ipynb format all as ipynb
* wip
* add some whole-notebook tests
* docstrings
* skip multiline magics
* add test for nested cell magic
* remove ipynb_test.yml, put ipynb tests in tox.ini
* add changelog entry
* typo
* make token same length as magic it replaces
* only include .ipynb by default if jupyter dependencies are found
* remove logic from const
* fixup
* fixup
* re.compile
* noop
* clear up
* new_src -> dst
* early exit for non-python notebooks
* add non-python test notebook
* add repo with many notebooks to black-primer
* install extra dependencies for black-primer
* fix planetary computer examples url
* dont run on ipynb files by default
* add scikit-lego (Expected to change) to black-primer
* add ipynb-specific diff
* fixup
* run on all (including ipynb) by default
* remove --include .ipynb from scikit-lego black-primer
* use tokenize so as to mirror the exact logic in IPython.core.displayhooks quiet
* fixup
* 🎨
* clarify docstring
* add test for when comment is after trailing semicolon
* enumerate(reversed) instead of [::-1]
* clarify docstrings
* wip
* use jupyter and no_jupyter marks
* use THIS_DIR
* windows fixup
* perform safe check cell-by-cell for ipynb
* only perform safe check in ipynb if not fast
* remove redundant Optional
* 🎨
* use typeguard
* dont process cell containing transformed magic
* require typing extensions before 3.10 so as to have TypeGuard
* use dataclasses
* mention black[jupyter] in docs as well as in README
* add faq
* add message to assertion error
* add test for indented quieted cell
* use tokenize_rt else we cant roundtrip
* fmake fronzet set for tokens to ignore when looking for trailing semicolon
* remove planetary code examples as recent commits result in changes
* use dataclasses which inherit from ast.NodeVisitor
* bump typing-extensions so that TypeGuard is available
* bump typing-extensions in Pipfile
* add test with notebook with empty metadata
* pipenv lock
* deprivative validate_cell
* Update README.md
* Update docs/getting_started.md
* dont cache notebooks if jupyter dependencies arent found
* dont write to cache if jupyter deps are not installed
* add notebook which cant be parsed
* use clirunner
* remove other subprocess calls
* add docstring
* make verbose and quiet keyword only
* 🎨
* run second many test on directory, not on file
* test for warning message when running on directory
* early return from non-python cell magics
* move NothingChanged to report to avoid circular import
* remove circular import
* reinstate --ipynb flag
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
The random asyncio bug is just too frequent and annoying to be
worth the speed improvements. Our test suite is already quite fast.
Random test failures hurt for 3 reasons, 1) they are discouraging for
new contributors who won't understand it's out of their control, 2)
it's annoying and time consuming to rerun the workflow, and 3) it
makes single job failures feel less important (even they should be
treated as important!).
There's some weird interaction between Click and
sphinxcontrib-programoutput on Windows that leads to an encoding error
during the printing of black-primer's help text.
Also symlinks aren't well supported on Windows so let's just use
includes which actually work because we now use MyST :D
* Add optional uvloop import
- If we find `uvloop` in the env for black, blackd or black-primer lets try and use it
- Add a uvloop extra install
Fixes#2257
Test:
- Add ci job to install black[uvloop] and run a primer run with uvloop
- Only with latest python (3.9)
- Will be handy to compare runtimes as a very unoffical benchmark
* Remove tox install
* Add to CHANGES/news
- Test and Primer don't run for documentation only changes since it's
unnecessary, eating unnecessary cycles and slowing down CI since these
workflows eat up the 20 max workers limit quite easily!
- Documentation Build runs all of the time now since quite a bit of the
content depends on Black's code so even a simple 1-file change in
src/black/__init__.py may break the docs build. It's not like this is
a costly workflow anyway.
Fuzz is still running on all changes because with fuzzing, the more the
better in general. 6 or 7 jobs on a documentation only commit is much
better than 27/28 jobs anyway :p
I also found an error in our bug report issue template :)
So these won't go out of date. This does mean the environment has be
setup a bit more carefully so the right version of the tool is used,
but thankfully the build environment is rebuilt on change on RTD anyway.
Also since the HTML docs are known to build fine, let's provide
downloadable HTMLzips of our docs.
This change needs RTD and GH to install Black with the [d] extra so
blackd's help can generated. While editing RTD's config file, let's
migrate the file to a non-deprecated filename.
Also I missed adding AUTHORS.md to the files key in the doc GHA config.
* Speed up test suite via distributed testing
Since we now run the test suite twice, one with Python 2 and another
without, full test runs are getting pretty slow. Let's try to
fix that with parallization.
Also use verbose mode on CI since more logs is usually better since
getting more is quite literally impossible.
The main issue we'll face with this is we'll hit
https://github.com/pytest-dev/pytest-xdist/issues/620 sometimes
(although pretty rarely). I suppose we can test this and see if how bad
this bug is for us, and revert if necessary down the line.
Also let's have some colours 🎉
Travis CI for Open Source is shutting down in a few weeks so the queue
for jobs is insane due to lower resources. I'm 99.99% sure we don't need
it as our Test, Lint, Docs, Upload / Package, Primer, and Fuzz workflows
are all on GitHub Actions. So even though we *can* migrate to the .com
version with its 1000 free Linux minutes(?), I don't think we need to.
more information here:
- https://blog.travis-ci.com/oss-announcement
- https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing
- https://docs.travis-ci.com/user/migrate/open-source-repository-migration
This commit does the following:
- delete the Travis CI configuration
- add to the GHA test workflows so coverage continues to be recorded
- tweaked coverage configuration so this wouldn't break
- remove any references to Travis CI in the docs (i.e. readme + sphinx
docs)
Regarding the Travis CI to GitHub Actions Coveralls transition, the
official action doesn't support the coverage files produced by coverage.py
unfornately. Also no, I don't really know what I am doing so don't @ me
if this breaks :p (well you can, but don't expect me to be THAT useful).
The Coveralls setup has two downfalls AFAIK:
- Only Linux runs are used because AndreMiras/coveralls-python-action
only supports Linux. Although this isn't a big issue since the Travis
Coveralls configuration only used Linux data too.
- Pull requests from an internal branch (i.e. one on psf/black) will be
marked as a push coverage build by Coveralls since our anti-duplicate-
workflows system runs under the push even for such cases.
* Add automatic version tagging to Docker Uploads
- If the git comment has a tag, set that on the docker images pushed
- If we don't have a tag, we just set `latest_non_release`
* Add trigger on release creation too
* Make prettier happy omn docker.yml
* Add Docker Github Action
- Build and upload arm64 + amd64 black images on push to master
This will need a `DOCKERHUB_USERNAME` and `DOCKERHUB_TOKEN` secrets set by someone with access.
* Change tag to push to pyfound/black repository. Thanks @ewdurbin
- Run grep to see commit has a line mentioning it to CHANGES.md
- Also add a label to disable this being required for PRs that don't need a change entry
Internal PRs (that come from branches of psf/black) match both the push
and pull_request events. This leads to a duplication of test, lint, etc.
checks on such PRs. Not only is it a waste of resources, it makes the
Status Checks UI more frustrating to go through.
Patch taken from:
https://github.community/t/duplicate-checks-on-push-and-pull-request-simultaneous-event/18012
This commit adds a new GitHub Actions workflow that builds self-contained
binaries / executables and uploads them as release assets to the triggering
release. Publishing a release, drafting one doesn't count, will trigger this
workflow.
I personally used GitHub Actions only because it's the CI/CD platform(?)
I am familiar with. Only Windows and Linux binaries are supported since
I don't have any systems running Mac OS.
For Linux, I had originally planned to use the manylinux2010 docker image
the PyPA provides for highly compatible wheel building, but unfortunately
it wasn't feasible due to GitHub Actions and PyInstaller incompatibilities.
As a stopgap the oldest versions of Linux and Windows are used although
Windows Server 2019 isn't that old nor is Ubuntu 16.04! I guess someone
(maybe me) could work out something else if compatibility is big problem.
A few things you should know about the workflow:
- You don't need to set the `GITHUB_TOKEN` secret as it is automatically
provided by GitHub.
- matrix.pathsep is used because PyInstaller configuration's format is OS
dependent for some reason ...
Also it's worth mentioning that Black once had Travis CI and AppVeyor
configuration that did the same thing as this commit. They were committed
in mid 2018 and worked (somewhat) well. Eventually we stopped using AppVeyor
and the refactor to packages broke the Travis CI config. This commit
replaces the still existing and broken Travis CI config wholesale.
Co-authored-by: Anders Fredrik Kiær <31612826+anders-kiaer@users.noreply.github.com>
- Anders told me that I could get the release asset upload URL directly
from the github.event.release payload. I originally planned to use
bruceadams/get-release to get such URL.
* Upgrade docs to Sphinx 3+
* Fix all the warnings...
- Fixed bad docstrings
- Fixed bad fenced code blocks in documentation
- Blocklisted some sections from being generated from the README
- Added missing documentation to index.rst
- Fixed an invalid autofunction directive in reference/reference_functions.rst
- Pin another documentation dependency
* Add documentation build test
* Add all listed by projects into primer.json + Enable on CI Runs
- Change workers default to 2 as black uses system CPU count
- Increase timeout to 5 mins for subprocess black runs
- Takes about 120s for 13 (3 disabled) projects on my 2018 Macbook Pro
- I was not removing directories tho ...
Will open an issue to investigate the failing projects and make this run cleaner.
- Once we get more stable we can expect more repos to be black formatted
Run it:
- `black-primer -k -w /tmp/primer_large_test --debug --rebase`
```
[2020-05-20 21:44:01,273] DEBUG: Starting /Users/cooper/venvs/b/bin/black-primer (cli.py:125)
[2020-05-20 21:44:01,273] DEBUG: Using selector: KqueueSelector (selector_events.py:53)
[2020-05-20 21:44:01,274] INFO: 16 projects to run Black over (lib.py:276)
[2020-05-20 21:44:01,274] DEBUG: Using 2 parallel workers to run Black (lib.py:281)
[2020-05-20 21:44:01,274] DEBUG: worker 0 workng on aioexabgp (lib.py:215)
[2020-05-20 21:44:01,276] DEBUG: worker 1 workng on attrs (lib.py:215)
[2020-05-20 21:44:02,443] INFO: Finished aioexabgp (lib.py:249)
[2020-05-20 21:44:02,443] DEBUG: worker 0 workng on bandersnatch (lib.py:215)
[2020-05-20 21:44:04,409] INFO: Finished bandersnatch (lib.py:249)
[2020-05-20 21:44:04,409] DEBUG: worker 0 workng on channels (lib.py:215)
[2020-05-20 21:44:04,702] INFO: Finished attrs (lib.py:249)
[2020-05-20 21:44:04,702] DEBUG: worker 1 workng on django (lib.py:215)
[2020-05-20 21:44:04,702] INFO: Skipping django as it's disabled via config (lib.py:222)
[2020-05-20 21:44:04,702] DEBUG: worker 1 workng on flake8-bugbear (lib.py:215)
[2020-05-20 21:44:05,813] INFO: Finished channels (lib.py:249)
[2020-05-20 21:44:05,813] DEBUG: worker 0 workng on hypothesis (lib.py:215)
[2020-05-20 21:44:06,071] INFO: Finished flake8-bugbear (lib.py:249)
[2020-05-20 21:44:06,071] DEBUG: worker 1 workng on pandas (lib.py:215)
[2020-05-20 21:44:06,071] INFO: Skipping pandas as it's disabled via config (lib.py:222)
[2020-05-20 21:44:06,071] DEBUG: worker 1 workng on poetry (lib.py:215)
[2020-05-20 21:44:16,207] INFO: Finished hypothesis (lib.py:249)
[2020-05-20 21:44:16,207] DEBUG: worker 0 workng on ptr (lib.py:215)
[2020-05-20 21:44:17,077] INFO: Finished poetry (lib.py:249)
[2020-05-20 21:44:17,077] DEBUG: worker 1 workng on pyramid (lib.py:215)
[2020-05-20 21:44:17,460] INFO: Finished ptr (lib.py:249)
[2020-05-20 21:44:17,460] DEBUG: worker 0 workng on pytest (lib.py:215)
[2020-05-20 21:44:17,460] INFO: Skipping pytest as it's disabled via config (lib.py:222)
[2020-05-20 21:44:17,460] DEBUG: worker 0 workng on sqlalchemy (lib.py:215)
[2020-05-20 21:44:33,319] INFO: Finished pyramid (lib.py:249)
[2020-05-20 21:44:33,319] DEBUG: worker 1 workng on tox (lib.py:215)
[2020-05-20 21:44:42,274] INFO: Finished tox (lib.py:249)
[2020-05-20 21:44:42,275] DEBUG: worker 1 workng on virtualenv (lib.py:215)
[2020-05-20 21:44:47,928] INFO: Finished virtualenv (lib.py:249)
[2020-05-20 21:44:47,928] DEBUG: worker 1 workng on warehouse (lib.py:215)
[2020-05-20 21:45:16,784] INFO: Finished warehouse (lib.py:249)
[2020-05-20 21:45:16,784] DEBUG: project_runner 1 exiting (lib.py:213)
[2020-05-20 21:45:45,700] INFO: Finished sqlalchemy (lib.py:249)
[2020-05-20 21:45:45,700] DEBUG: project_runner 0 exiting (lib.py:213)
[2020-05-20 21:45:45,701] INFO: Analyzing results (lib.py:292)
-- primer results 📊 --
13 / 16 succeeded (81.25%) ✅
0 / 16 FAILED (0.0%) 💩
- 3 projects disabled by config
- 0 projects skipped due to Python version
- 0 skipped due to long checkout
```
* Move to partial for rmtree + specify a onerror handler for PermissionError on Windows for git
* Set default coding to utf8 for very important emoji's on Windows
* Set Python encoding to utf-8 for Windows
* Appease the white space gods of Black!
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
