Add new platformdirs dependencies as hidden imports when creating
PyInstaller-based binaries.
platformdirs imports the module for each platform dynamically, which
PyInstaller is unable to correctly detect for packing. By adding the
modules as hidden imports, we are telling PyInstaller to include the
modules in the packaged binary.
This issue seems to have been introduce when switching to platformdirs
in #2375. fixes#2464
Commit history before merge:
* Add hidden import to PyInstaller build
Add new platformdirs dependency as a hidden import when creating
PyInstaller based binaries.
* Only include the platformdirs for the relevant os
Draft releases don't trigger the workflows (that's good!) but since they only
Commit history before merge:
* fix: run pypi upload from published draft releases
* Fix broken task list markup in PR template
* change docker workflow to build on release publish
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
To summarise, based on what was discussed in that issue:
due to not being able to parse automagics (e.g. pip install black)
without a running IPython kernel, cells with syntax which is parseable
by neither ast.parse nor IPython will be skipped cells with multiline
magics will be skipped trailing semicolons will be preserved, as they
are often put there intentionally in Jupyter Notebooks to suppress
unnecessary output
Commit history before merge (excluding merge commits):
* wip
* fixup tests
* skip tests if no IPython
* install test requirements in ipynb tests
* if --ipynb format all as ipynb
* wip
* add some whole-notebook tests
* docstrings
* skip multiline magics
* add test for nested cell magic
* remove ipynb_test.yml, put ipynb tests in tox.ini
* add changelog entry
* typo
* make token same length as magic it replaces
* only include .ipynb by default if jupyter dependencies are found
* remove logic from const
* fixup
* fixup
* re.compile
* noop
* clear up
* new_src -> dst
* early exit for non-python notebooks
* add non-python test notebook
* add repo with many notebooks to black-primer
* install extra dependencies for black-primer
* fix planetary computer examples url
* dont run on ipynb files by default
* add scikit-lego (Expected to change) to black-primer
* add ipynb-specific diff
* fixup
* run on all (including ipynb) by default
* remove --include .ipynb from scikit-lego black-primer
* use tokenize so as to mirror the exact logic in IPython.core.displayhooks quiet
* fixup
* 🎨
* clarify docstring
* add test for when comment is after trailing semicolon
* enumerate(reversed) instead of [::-1]
* clarify docstrings
* wip
* use jupyter and no_jupyter marks
* use THIS_DIR
* windows fixup
* perform safe check cell-by-cell for ipynb
* only perform safe check in ipynb if not fast
* remove redundant Optional
* 🎨
* use typeguard
* dont process cell containing transformed magic
* require typing extensions before 3.10 so as to have TypeGuard
* use dataclasses
* mention black[jupyter] in docs as well as in README
* add faq
* add message to assertion error
* add test for indented quieted cell
* use tokenize_rt else we cant roundtrip
* fmake fronzet set for tokens to ignore when looking for trailing semicolon
* remove planetary code examples as recent commits result in changes
* use dataclasses which inherit from ast.NodeVisitor
* bump typing-extensions so that TypeGuard is available
* bump typing-extensions in Pipfile
* add test with notebook with empty metadata
* pipenv lock
* deprivative validate_cell
* Update README.md
* Update docs/getting_started.md
* dont cache notebooks if jupyter dependencies arent found
* dont write to cache if jupyter deps are not installed
* add notebook which cant be parsed
* use clirunner
* remove other subprocess calls
* add docstring
* make verbose and quiet keyword only
* 🎨
* run second many test on directory, not on file
* test for warning message when running on directory
* early return from non-python cell magics
* move NothingChanged to report to avoid circular import
* remove circular import
* reinstate --ipynb flag
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
The random asyncio bug is just too frequent and annoying to be
worth the speed improvements. Our test suite is already quite fast.
Random test failures hurt for 3 reasons, 1) they are discouraging for
new contributors who won't understand it's out of their control, 2)
it's annoying and time consuming to rerun the workflow, and 3) it
makes single job failures feel less important (even they should be
treated as important!).
There's some weird interaction between Click and
sphinxcontrib-programoutput on Windows that leads to an encoding error
during the printing of black-primer's help text.
Also symlinks aren't well supported on Windows so let's just use
includes which actually work because we now use MyST :D
* Add optional uvloop import
- If we find `uvloop` in the env for black, blackd or black-primer lets try and use it
- Add a uvloop extra install
Fixes#2257
Test:
- Add ci job to install black[uvloop] and run a primer run with uvloop
- Only with latest python (3.9)
- Will be handy to compare runtimes as a very unoffical benchmark
* Remove tox install
* Add to CHANGES/news
- Test and Primer don't run for documentation only changes since it's
unnecessary, eating unnecessary cycles and slowing down CI since these
workflows eat up the 20 max workers limit quite easily!
- Documentation Build runs all of the time now since quite a bit of the
content depends on Black's code so even a simple 1-file change in
src/black/__init__.py may break the docs build. It's not like this is
a costly workflow anyway.
Fuzz is still running on all changes because with fuzzing, the more the
better in general. 6 or 7 jobs on a documentation only commit is much
better than 27/28 jobs anyway :p
I also found an error in our bug report issue template :)
So these won't go out of date. This does mean the environment has be
setup a bit more carefully so the right version of the tool is used,
but thankfully the build environment is rebuilt on change on RTD anyway.
Also since the HTML docs are known to build fine, let's provide
downloadable HTMLzips of our docs.
This change needs RTD and GH to install Black with the [d] extra so
blackd's help can generated. While editing RTD's config file, let's
migrate the file to a non-deprecated filename.
Also I missed adding AUTHORS.md to the files key in the doc GHA config.
* Speed up test suite via distributed testing
Since we now run the test suite twice, one with Python 2 and another
without, full test runs are getting pretty slow. Let's try to
fix that with parallization.
Also use verbose mode on CI since more logs is usually better since
getting more is quite literally impossible.
The main issue we'll face with this is we'll hit
https://github.com/pytest-dev/pytest-xdist/issues/620 sometimes
(although pretty rarely). I suppose we can test this and see if how bad
this bug is for us, and revert if necessary down the line.
Also let's have some colours 🎉
Travis CI for Open Source is shutting down in a few weeks so the queue
for jobs is insane due to lower resources. I'm 99.99% sure we don't need
it as our Test, Lint, Docs, Upload / Package, Primer, and Fuzz workflows
are all on GitHub Actions. So even though we *can* migrate to the .com
version with its 1000 free Linux minutes(?), I don't think we need to.
more information here:
- https://blog.travis-ci.com/oss-announcement
- https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing
- https://docs.travis-ci.com/user/migrate/open-source-repository-migration
This commit does the following:
- delete the Travis CI configuration
- add to the GHA test workflows so coverage continues to be recorded
- tweaked coverage configuration so this wouldn't break
- remove any references to Travis CI in the docs (i.e. readme + sphinx
docs)
Regarding the Travis CI to GitHub Actions Coveralls transition, the
official action doesn't support the coverage files produced by coverage.py
unfornately. Also no, I don't really know what I am doing so don't @ me
if this breaks :p (well you can, but don't expect me to be THAT useful).
The Coveralls setup has two downfalls AFAIK:
- Only Linux runs are used because AndreMiras/coveralls-python-action
only supports Linux. Although this isn't a big issue since the Travis
Coveralls configuration only used Linux data too.
- Pull requests from an internal branch (i.e. one on psf/black) will be
marked as a push coverage build by Coveralls since our anti-duplicate-
workflows system runs under the push even for such cases.
* Add automatic version tagging to Docker Uploads
- If the git comment has a tag, set that on the docker images pushed
- If we don't have a tag, we just set `latest_non_release`
* Add trigger on release creation too
* Make prettier happy omn docker.yml
* Add Docker Github Action
- Build and upload arm64 + amd64 black images on push to master
This will need a `DOCKERHUB_USERNAME` and `DOCKERHUB_TOKEN` secrets set by someone with access.
* Change tag to push to pyfound/black repository. Thanks @ewdurbin
- Run grep to see commit has a line mentioning it to CHANGES.md
- Also add a label to disable this being required for PRs that don't need a change entry
Internal PRs (that come from branches of psf/black) match both the push
and pull_request events. This leads to a duplication of test, lint, etc.
checks on such PRs. Not only is it a waste of resources, it makes the
Status Checks UI more frustrating to go through.
Patch taken from:
https://github.community/t/duplicate-checks-on-push-and-pull-request-simultaneous-event/18012
This commit adds a new GitHub Actions workflow that builds self-contained
binaries / executables and uploads them as release assets to the triggering
release. Publishing a release, drafting one doesn't count, will trigger this
workflow.
I personally used GitHub Actions only because it's the CI/CD platform(?)
I am familiar with. Only Windows and Linux binaries are supported since
I don't have any systems running Mac OS.
For Linux, I had originally planned to use the manylinux2010 docker image
the PyPA provides for highly compatible wheel building, but unfortunately
it wasn't feasible due to GitHub Actions and PyInstaller incompatibilities.
As a stopgap the oldest versions of Linux and Windows are used although
Windows Server 2019 isn't that old nor is Ubuntu 16.04! I guess someone
(maybe me) could work out something else if compatibility is big problem.
A few things you should know about the workflow:
- You don't need to set the `GITHUB_TOKEN` secret as it is automatically
provided by GitHub.
- matrix.pathsep is used because PyInstaller configuration's format is OS
dependent for some reason ...
Also it's worth mentioning that Black once had Travis CI and AppVeyor
configuration that did the same thing as this commit. They were committed
in mid 2018 and worked (somewhat) well. Eventually we stopped using AppVeyor
and the refactor to packages broke the Travis CI config. This commit
replaces the still existing and broken Travis CI config wholesale.
Co-authored-by: Anders Fredrik Kiær <31612826+anders-kiaer@users.noreply.github.com>
- Anders told me that I could get the release asset upload URL directly
from the github.event.release payload. I originally planned to use
bruceadams/get-release to get such URL.
* Upgrade docs to Sphinx 3+
* Fix all the warnings...
- Fixed bad docstrings
- Fixed bad fenced code blocks in documentation
- Blocklisted some sections from being generated from the README
- Added missing documentation to index.rst
- Fixed an invalid autofunction directive in reference/reference_functions.rst
- Pin another documentation dependency
* Add documentation build test
* Add all listed by projects into primer.json + Enable on CI Runs
- Change workers default to 2 as black uses system CPU count
- Increase timeout to 5 mins for subprocess black runs
- Takes about 120s for 13 (3 disabled) projects on my 2018 Macbook Pro
- I was not removing directories tho ...
Will open an issue to investigate the failing projects and make this run cleaner.
- Once we get more stable we can expect more repos to be black formatted
Run it:
- `black-primer -k -w /tmp/primer_large_test --debug --rebase`
```
[2020-05-20 21:44:01,273] DEBUG: Starting /Users/cooper/venvs/b/bin/black-primer (cli.py:125)
[2020-05-20 21:44:01,273] DEBUG: Using selector: KqueueSelector (selector_events.py:53)
[2020-05-20 21:44:01,274] INFO: 16 projects to run Black over (lib.py:276)
[2020-05-20 21:44:01,274] DEBUG: Using 2 parallel workers to run Black (lib.py:281)
[2020-05-20 21:44:01,274] DEBUG: worker 0 workng on aioexabgp (lib.py:215)
[2020-05-20 21:44:01,276] DEBUG: worker 1 workng on attrs (lib.py:215)
[2020-05-20 21:44:02,443] INFO: Finished aioexabgp (lib.py:249)
[2020-05-20 21:44:02,443] DEBUG: worker 0 workng on bandersnatch (lib.py:215)
[2020-05-20 21:44:04,409] INFO: Finished bandersnatch (lib.py:249)
[2020-05-20 21:44:04,409] DEBUG: worker 0 workng on channels (lib.py:215)
[2020-05-20 21:44:04,702] INFO: Finished attrs (lib.py:249)
[2020-05-20 21:44:04,702] DEBUG: worker 1 workng on django (lib.py:215)
[2020-05-20 21:44:04,702] INFO: Skipping django as it's disabled via config (lib.py:222)
[2020-05-20 21:44:04,702] DEBUG: worker 1 workng on flake8-bugbear (lib.py:215)
[2020-05-20 21:44:05,813] INFO: Finished channels (lib.py:249)
[2020-05-20 21:44:05,813] DEBUG: worker 0 workng on hypothesis (lib.py:215)
[2020-05-20 21:44:06,071] INFO: Finished flake8-bugbear (lib.py:249)
[2020-05-20 21:44:06,071] DEBUG: worker 1 workng on pandas (lib.py:215)
[2020-05-20 21:44:06,071] INFO: Skipping pandas as it's disabled via config (lib.py:222)
[2020-05-20 21:44:06,071] DEBUG: worker 1 workng on poetry (lib.py:215)
[2020-05-20 21:44:16,207] INFO: Finished hypothesis (lib.py:249)
[2020-05-20 21:44:16,207] DEBUG: worker 0 workng on ptr (lib.py:215)
[2020-05-20 21:44:17,077] INFO: Finished poetry (lib.py:249)
[2020-05-20 21:44:17,077] DEBUG: worker 1 workng on pyramid (lib.py:215)
[2020-05-20 21:44:17,460] INFO: Finished ptr (lib.py:249)
[2020-05-20 21:44:17,460] DEBUG: worker 0 workng on pytest (lib.py:215)
[2020-05-20 21:44:17,460] INFO: Skipping pytest as it's disabled via config (lib.py:222)
[2020-05-20 21:44:17,460] DEBUG: worker 0 workng on sqlalchemy (lib.py:215)
[2020-05-20 21:44:33,319] INFO: Finished pyramid (lib.py:249)
[2020-05-20 21:44:33,319] DEBUG: worker 1 workng on tox (lib.py:215)
[2020-05-20 21:44:42,274] INFO: Finished tox (lib.py:249)
[2020-05-20 21:44:42,275] DEBUG: worker 1 workng on virtualenv (lib.py:215)
[2020-05-20 21:44:47,928] INFO: Finished virtualenv (lib.py:249)
[2020-05-20 21:44:47,928] DEBUG: worker 1 workng on warehouse (lib.py:215)
[2020-05-20 21:45:16,784] INFO: Finished warehouse (lib.py:249)
[2020-05-20 21:45:16,784] DEBUG: project_runner 1 exiting (lib.py:213)
[2020-05-20 21:45:45,700] INFO: Finished sqlalchemy (lib.py:249)
[2020-05-20 21:45:45,700] DEBUG: project_runner 0 exiting (lib.py:213)
[2020-05-20 21:45:45,701] INFO: Analyzing results (lib.py:292)
-- primer results 📊 --
13 / 16 succeeded (81.25%) ✅
0 / 16 FAILED (0.0%) 💩
- 3 projects disabled by config
- 0 projects skipped due to Python version
- 0 skipped due to long checkout
```
* Move to partial for rmtree + specify a onerror handler for PermissionError on Windows for git
* Set default coding to utf8 for very important emoji's on Windows
* Set Python encoding to utf-8 for Windows
* Appease the white space gods of Black!
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
