actions/black
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Compare commits

base: actions:main
Branches Tags
actions:main
actions:25.1.0
actions:24.10.0
actions:24.8.0
actions:24.4.2
actions:24.4.1
actions:24.4.0
actions:24.3.0
actions:24.2.0
actions:24.1.1
actions:24.1.0
actions:23.12.1
actions:23.12.0
actions:24.1a1
actions:23.11.0
actions:23.10.1
actions:23.10.0
actions:23.9.1
actions:23.9.0
actions:23.7.0
actions:23.3.0
actions:23.1.0
actions:23.1a1
actions:22.12.0
actions:22.10.0
actions:22.8.0
actions:22.6.0
actions:22.3.0
actions:22.1.0
actions:21.12b0
actions:21.11b1
actions:21.11b0
actions:21.10b0
actions:21.9b0
actions:21.8b0
actions:21.7b0
actions:21.6b0
actions:21.5b2
actions:21.5b1
actions:21.5b0
actions:21.4b2
actions:21.4b1
actions:21.4b0
actions:20.8b1
actions:20.8b0
actions:19.10b0
actions:19.3b0
actions:18.9b0
actions:18.6b4
actions:18.6b3
actions:18.6b2
actions:18.6b1
actions:18.6b0
actions:18.5b1
actions:18.5b0
actions:18.4a4
actions:18.4a3
actions:18.4a2
actions:18.4a1
actions:18.4a0
actions:18.3a4
actions:18.3a3
actions:18.3a2
actions:18.3a1
actions:18.3a0
..
compare: actions:23.11.0
Branches Tags
actions:main
actions:25.1.0
actions:24.10.0
actions:24.8.0
actions:24.4.2
actions:24.4.1
actions:24.4.0
actions:24.3.0
actions:24.2.0
actions:24.1.1
actions:24.1.0
actions:23.12.1
actions:23.12.0
actions:24.1a1
actions:23.11.0
actions:23.10.1
actions:23.10.0
actions:23.9.1
actions:23.9.0
actions:23.7.0
actions:23.3.0
actions:23.1.0
actions:23.1a1
actions:22.12.0
actions:22.10.0
actions:22.8.0
actions:22.6.0
actions:22.3.0
actions:22.1.0
actions:21.12b0
actions:21.11b1
actions:21.11b0
actions:21.10b0
actions:21.9b0
actions:21.8b0
actions:21.7b0
actions:21.6b0
actions:21.5b2
actions:21.5b1
actions:21.5b0
actions:21.4b2
actions:21.4b1
actions:21.4b0
actions:20.8b1
actions:20.8b0
actions:19.10b0
actions:19.3b0
actions:18.9b0
actions:18.6b4
actions:18.6b3
actions:18.6b2
actions:18.6b1
actions:18.6b0
actions:18.5b1
actions:18.5b0
actions:18.4a4
actions:18.4a3
actions:18.4a2
actions:18.4a1
actions:18.4a0
actions:18.3a4
actions:18.3a3
actions:18.3a2
actions:18.3a1
actions:18.3a0

No commits in common. "main" and "23.11.0" have entirely different histories.
main ... 23.11.0

222 changed files with 3539 additions and 8136 deletions
Show Stats Expand all files Collapse all files

4
.flake8
View File

@ -1,8 +1,8 @@
[flake8]
# B905 should be enabled when we drop support for 3.9
ignore = E203, E266, E501, E701, E704, W503, B905, B907
ignore = E203, E266, E501, E704, W503, B905, B907
# line length is intentionally set to 80 here because black uses Bugbear
# See https://black.readthedocs.io/en/stable/guides/using_black_with_other_tools.html#bugbear for more details
# See https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length for more details
max-line-length = 80
max-complexity = 18
select = B,C,E,F,W,T4,B9

3
.git_archival.txt
View File

@ -1,3 +1,4 @@
node: $Format:%H$
node-date: $Format:%cI$
describe-name: $Format:%(describe:tags=true,match=[0-9]*)$
describe-name: $Format:%(describe:tags=true,match=*[0-9]*)$
ref-names: $Format:%D$

1
.gitattributes vendored
View File

@ -1,2 +1 @@
.git_archival.txt export-subst
*.py diff=python

4
.github/ISSUE_TEMPLATE/bug_report.md vendored
View File

@ -12,9 +12,7 @@ current development version. To confirm this, you have three options:
1. Update Black's version if a newer release exists: `pip install -U black`
2. Use the online formatter at <https://black.vercel.app/?version=main>, which will use
the latest main branch. Note that the online formatter currently runs on
an older version of Python and may not support newer syntax, such as the
extended f-string syntax added in Python 3.12.
the latest main branch.
3. Or run _Black_ on your machine:
- create a new virtualenv (make sure it's the same Python version);
- clone this repository;

1
.github/dependabot.yml vendored
View File

@ -14,3 +14,4 @@ updates:
schedule:
interval: "weekly"
labels: ["skip news", "C: dependencies", "T: documentation"]
reviewers: ["ichard26"]

32
.github/workflows/diff_shades.yml vendored
View File

@ -20,13 +20,13 @@ jobs:
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
- uses: actions/setup-python@v4
with:
python-version: "3.11"
- name: Install diff-shades and support dependencies
run: |
python -m pip install 'click>=8.1.7' packaging urllib3
python -m pip install 'click==8.1.3' packaging urllib3
python -m pip install https://github.com/ichard26/diff-shades/archive/stable.zip
- name: Calculate run configuration & metadata
@ -34,8 +34,7 @@ jobs:
env:
GITHUB_TOKEN: ${{ github.token }}
run: >
python scripts/diff_shades_gha_helper.py config ${{ github.event_name }}
${{ matrix.mode }}
python scripts/diff_shades_gha_helper.py config ${{ github.event_name }} ${{ matrix.mode }}
analysis:
name: analysis / ${{ matrix.mode }}
@ -45,11 +44,11 @@ jobs:
HATCH_BUILD_HOOKS_ENABLE: "1"
# Clang is less picky with the C code it's given than gcc (and may
# generate faster binaries too).
CC: clang-18
CC: clang-14
strategy:
fail-fast: false
matrix:
include: ${{ fromJson(needs.configure.outputs.matrix) }}
include: ${{ fromJson(needs.configure.outputs.matrix )}}
steps:
- name: Checkout this repository (full clone)
@ -58,14 +57,14 @@ jobs:
# The baseline revision could be rather old so a full clone is ideal.
fetch-depth: 0
- uses: actions/setup-python@v5
- uses: actions/setup-python@v4
with:
python-version: "3.11"
- name: Install diff-shades and support dependencies
run: |
python -m pip install https://github.com/ichard26/diff-shades/archive/stable.zip
python -m pip install 'click>=8.1.7' packaging urllib3
python -m pip install 'click==8.1.3' packaging urllib3
# After checking out old revisions, this might not exist so we'll use a copy.
cat scripts/diff_shades_gha_helper.py > helper.py
git config user.name "diff-shades-gha"
@ -73,7 +72,7 @@ jobs:
- name: Attempt to use cached baseline analysis
id: baseline-cache
uses: actions/cache@v4
uses: actions/cache@v3
with:
path: ${{ matrix.baseline-analysis }}
key: ${{ matrix.baseline-cache-key }}
@ -111,19 +110,19 @@ jobs:
${{ matrix.baseline-analysis }} ${{ matrix.target-analysis }}
- name: Upload diff report
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v3
with:
name: ${{ matrix.mode }}-diff.html
path: diff.html
- name: Upload baseline analysis
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v3
with:
name: ${{ matrix.baseline-analysis }}
path: ${{ matrix.baseline-analysis }}
- name: Upload target analysis
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v3
with:
name: ${{ matrix.target-analysis }}
path: ${{ matrix.target-analysis }}
@ -131,13 +130,14 @@ jobs:
- name: Generate summary file (PR only)
if: github.event_name == 'pull_request' && matrix.mode == 'preview-changes'
run: >
python helper.py comment-body ${{ matrix.baseline-analysis }}
${{ matrix.target-analysis }} ${{ matrix.baseline-sha }}
${{ matrix.target-sha }} ${{ github.event.pull_request.number }}
python helper.py comment-body
${{ matrix.baseline-analysis }} ${{ matrix.target-analysis }}
${{ matrix.baseline-sha }} ${{ matrix.target-sha }}
${{ github.event.pull_request.number }}
- name: Upload summary file (PR only)
if: github.event_name == 'pull_request' && matrix.mode == 'preview-changes'
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v3
with:
name: .pr-comment.json
path: .pr-comment.json

6
.github/workflows/diff_shades_comment.yml vendored
View File

@ -13,7 +13,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
- uses: actions/setup-python@v4
with:
python-version: "*"
@ -33,7 +33,7 @@ jobs:
- name: Try to find pre-existing PR comment
if: steps.metadata.outputs.needs-comment == 'true'
id: find-comment
uses: peter-evans/find-comment@3eae4d37986fb5a8592848f6a574fdf654e61f9e
uses: peter-evans/find-comment@a54c31d7fa095754bfef525c0c8e5e5674c4b4b1
with:
issue-number: ${{ steps.metadata.outputs.pr-number }}
comment-author: "github-actions[bot]"
@ -41,7 +41,7 @@ jobs:
- name: Create or update PR comment
if: steps.metadata.outputs.needs-comment == 'true'
uses: peter-evans/create-or-update-comment@71345be0265236311c031f5c7866368bd1eff043
uses: peter-evans/create-or-update-comment@23ff15729ef2fc348714a3bb66d2f655ca9066f2
with:
comment-id: ${{ steps.find-comment.outputs.comment-id }}
issue-number: ${{ steps.metadata.outputs.pr-number }}

12
.github/workflows/doc.yml vendored
View File

@ -24,17 +24,15 @@ jobs:
- uses: actions/checkout@v4
- name: Set up latest Python
uses: actions/setup-python@v5
uses: actions/setup-python@v4
with:
python-version: "3.13"
allow-prereleases: true
python-version: "3.11"
- name: Install dependencies
run: |
python -m pip install uv
python -m uv venv
python -m uv pip install -e ".[d]"
python -m uv pip install -r "docs/requirements.txt"
python -m pip install --upgrade pip setuptools wheel
python -m pip install -e ".[d]"
python -m pip install -r "docs/requirements.txt"
- name: Build documentation
run: sphinx-build -a -b html -W --keep-going docs/ docs/_build

6
.github/workflows/docker.yml vendored
View File

@ -36,7 +36,7 @@ jobs:
latest_non_release)" >> $GITHUB_ENV
- name: Build and push
uses: docker/build-push-action@v6
uses: docker/build-push-action@v5
with:
context: .
platforms: linux/amd64,linux/arm64
@ -47,7 +47,7 @@ jobs:
if:
${{ github.event_name == 'release' && github.event.action == 'published' &&
!github.event.release.prerelease }}
uses: docker/build-push-action@v6
uses: docker/build-push-action@v5
with:
context: .
platforms: linux/amd64,linux/arm64
@ -58,7 +58,7 @@ jobs:
if:
${{ github.event_name == 'release' && github.event.action == 'published' &&
github.event.release.prerelease }}
uses: docker/build-push-action@v6
uses: docker/build-push-action@v5
with:
context: .
platforms: linux/amd64,linux/arm64

5
.github/workflows/fuzz.yml vendored
View File

@ -22,16 +22,15 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12.4", "3.13"]
python-version: ["3.8", "3.9", "3.10", "3.11"]
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
allow-prereleases: true
- name: Install dependencies
run: |

16
.github/workflows/lint.yml vendored
View File

@ -1,4 +1,4 @@
name: Lint + format ourselves
name: Lint
on: [push, pull_request]
@ -24,25 +24,19 @@ jobs:
fi
- name: Set up latest Python
uses: actions/setup-python@v5
uses: actions/setup-python@v4
with:
python-version: "3.13"
allow-prereleases: true
python-version: "3.11"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install -e '.'
python -m pip install -e '.[d]'
python -m pip install tox
- name: Run pre-commit hooks
uses: pre-commit/action@v3.0.1
uses: pre-commit/action@v3.0.0
- name: Format ourselves
run: |
tox -e run_self
- name: Regenerate schema
run: |
tox -e generate_schema
git diff --exit-code

19
.github/workflows/pypi_upload.yml vendored
View File

@ -21,10 +21,9 @@ jobs:
- uses: actions/checkout@v4
- name: Set up latest Python
uses: actions/setup-python@v5
uses: actions/setup-python@v4
with:
python-version: "3.13"
allow-prereleases: true
python-version: "*"
- name: Install latest pip, build, twine
run: |
@ -47,11 +46,10 @@ jobs:
include: ${{ steps.set-matrix.outputs.include }}
steps:
- uses: actions/checkout@v4
# Keep cibuildwheel version in sync with below
- name: Install cibuildwheel and pypyp
run: |
pipx install cibuildwheel==2.22.0
pipx install pypyp==1.3.0
pipx install cibuildwheel==2.15.0
pipx install pypyp==1
- name: generate matrix
if: github.event_name != 'pull_request'
run: |
@ -75,7 +73,7 @@ jobs:
| pyp 'json.dumps({"only": x, "os": "ubuntu-latest"})'
} | pyp 'json.dumps(list(map(json.loads, lines)))' > /tmp/matrix
env:
CIBW_BUILD: "cp39-* cp313-*"
CIBW_BUILD: "cp38-* cp311-*"
CIBW_ARCHS_LINUX: x86_64
- id: set-matrix
run: echo "include=$(cat /tmp/matrix)" | tee -a $GITHUB_OUTPUT
@ -91,15 +89,14 @@ jobs:
steps:
- uses: actions/checkout@v4
# Keep cibuildwheel version in sync with above
- uses: pypa/cibuildwheel@v2.23.3
- uses: pypa/cibuildwheel@v2.16.2
with:
only: ${{ matrix.only }}
- name: Upload wheels as workflow artifacts
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v3
with:
name: ${{ matrix.only }}-mypyc-wheels
name: ${{ matrix.name }}-mypyc-wheels
path: ./wheelhouse/*.whl
- if: github.event_name == 'release'

4
.github/workflows/release_tests.yml vendored
View File

@ -25,7 +25,7 @@ jobs:
runs-on: ${{ matrix.os }}
strategy:
matrix:
python-version: ["3.13"]
python-version: ["3.12"]
os: [macOS-latest, ubuntu-latest, windows-latest]
steps:
@ -34,7 +34,7 @@ jobs:
# Give us all history, branches and tags
fetch-depth: 0
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
allow-prereleases: true

15
.github/workflows/test.yml vendored
View File

@ -31,17 +31,16 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12.4", "3.13", "pypy-3.9"]
python-version: ["3.8", "3.9", "3.10", "3.11", "pypy-3.8"]
os: [ubuntu-latest, macOS-latest, windows-latest]
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
allow-prereleases: true
- name: Install tox
run: |
@ -63,7 +62,7 @@ jobs:
if:
github.repository == 'psf/black' && matrix.os == 'ubuntu-latest' &&
!startsWith(matrix.python-version, 'pypy')
uses: AndreMiras/coveralls-python-action@ac868b9540fad490f7ca82b8ca00480fd751ed19
uses: AndreMiras/coveralls-python-action@8799c9f4443ac4201d2e2f2c725d577174683b99
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
parallel: true
@ -78,7 +77,7 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Send finished signal to Coveralls
uses: AndreMiras/coveralls-python-action@ac868b9540fad490f7ca82b8ca00480fd751ed19
uses: AndreMiras/coveralls-python-action@8799c9f4443ac4201d2e2f2c725d577174683b99
with:
parallel-finished: true
debug: true
@ -97,9 +96,9 @@ jobs:
- uses: actions/checkout@v4
- name: Set up latest Python
uses: actions/setup-python@v5
uses: actions/setup-python@v4
with:
python-version: "3.12.4"
python-version: "*"
- name: Install black with uvloop
run: |
@ -107,4 +106,4 @@ jobs:
python -m pip install -e ".[uvloop]"
- name: Format ourselves
run: python -m black --check src/ tests/
run: python -m black --check .

8
.github/workflows/upload_binary.yml vendored
View File

@ -13,13 +13,13 @@ jobs:
strategy:
fail-fast: false
matrix:
os: [windows-2019, ubuntu-22.04, macos-latest]
os: [windows-2019, ubuntu-20.04, macos-latest]
include:
- os: windows-2019
pathsep: ";"
asset_name: black_windows.exe
executable_mime: "application/vnd.microsoft.portable-executable"
- os: ubuntu-22.04
- os: ubuntu-20.04
pathsep: ":"
asset_name: black_linux
executable_mime: "application/x-executable"
@ -32,9 +32,9 @@ jobs:
- uses: actions/checkout@v4
- name: Set up latest Python
uses: actions/setup-python@v5
uses: actions/setup-python@v4
with:
python-version: "3.12.4"
python-version: "*"
- name: Install Black and PyInstaller
run: |

1
.gitignore vendored
View File

@ -25,4 +25,3 @@ src/_black_version.py
.hypothesis/
venv/
.ipynb_checkpoints/
node_modules/

32
.pre-commit-config.yaml
View File

@ -24,57 +24,47 @@ repos:
additional_dependencies: *version_check_dependencies
- repo: https://github.com/pycqa/isort
rev: 6.0.1
rev: 5.12.0
hooks:
- id: isort
- repo: https://github.com/pycqa/flake8
rev: 7.2.0
rev: 6.1.0
hooks:
- id: flake8
additional_dependencies:
- flake8-bugbear==24.2.6
- flake8-bugbear
- flake8-comprehensions
- flake8-simplify
exclude: ^src/blib2to3/
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.15.0
rev: v1.5.1
hooks:
- id: mypy
exclude: ^(docs/conf.py|scripts/generate_schema.py)$
args: []
additional_dependencies: &mypy_deps
exclude: ^docs/conf.py
args: ["--config-file", "pyproject.toml"]
additional_dependencies:
- types-PyYAML
- types-atheris
- tomli >= 0.2.6, < 2.0.0
- click >= 8.2.0
# Click is intentionally out-of-sync with pyproject.toml
# v8.2 has breaking changes. We work around them at runtime, but we need the newer stubs.
- click >= 8.1.0, != 8.1.4, != 8.1.5
- packaging >= 22.0
- platformdirs >= 2.1.0
- pytokens >= 0.1.10
- pytest
- hypothesis
- aiohttp >= 3.7.4
- types-commonmark
- urllib3
- hypothesmith
- id: mypy
name: mypy (Python 3.10)
files: scripts/generate_schema.py
args: ["--python-version=3.10"]
additional_dependencies: *mypy_deps
- repo: https://github.com/rbubley/mirrors-prettier
rev: v3.5.3
- repo: https://github.com/pre-commit/mirrors-prettier
rev: v3.0.3
hooks:
- id: prettier
types_or: [markdown, yaml, json]
exclude: \.github/workflows/diff_shades\.yml
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
rev: v4.4.0
hooks:
- id: end-of-file-fixer
- id: trailing-whitespace

2
.pre-commit-hooks.yaml
View File

@ -4,6 +4,7 @@
name: black
description: "Black: The uncompromising Python code formatter"
entry: black
stages: [pre-commit, pre-merge-commit, pre-push, manual]
language: python
minimum_pre_commit_version: 2.9.2
require_serial: true
@ -13,6 +14,7 @@
description:
"Black: The uncompromising Python code formatter (with Jupyter Notebook support)"
entry: black
stages: [pre-commit, pre-merge-commit, pre-push, manual]
language: python
minimum_pre_commit_version: 2.9.2
require_serial: true

3
.readthedocs.yaml
View File

@ -16,6 +16,3 @@ python:
path: .
extra_requirements:
- d
sphinx:
configuration: docs/conf.py

1
AUTHORS.md
View File

@ -181,7 +181,6 @@ Multiple contributions by:
- [Tony Narlock](mailto:tony@git-pull.com)
- [Tsuyoshi Hombashi](mailto:tsuyoshi.hombashi@gmail.com)
- [Tushar Chandra](mailto:tusharchandra2018@u.northwestern.edu)
- [Tushar Sadhwani](mailto:tushar.sadhwani000@gmail.com)
- [Tzu-ping Chung](mailto:uranusjr@gmail.com)
- [Utsav Shah](mailto:ukshah2@illinois.edu)
- utsav-dbx

453
CHANGES.md
View File

@ -1,458 +1,5 @@
# Change Log
## Unreleased
### Highlights
<!-- Include any especially major or disruptive changes here -->
### Stable style
<!-- Changes that affect Black's stable style -->
- Fix crash while formatting a long `del` statement containing tuples (#4628)
- Fix crash while formatting expressions using the walrus operator in complex `with`
statements (#4630)
- Handle `# fmt: skip` followed by a comment at the end of file (#4635)
- Fix crash when a tuple appears in the `as` clause of a `with` statement (#4634)
- Fix crash when tuple is used as a context manager inside a `with` statement (#4646)
- Fix crash on a `\\r\n` (#4673)
- Fix crash on `await ...` (where `...` is a literal `Ellipsis`) (#4676)
- Remove support for pre-python 3.7 `await/async` as soft keywords/variable names
(#4676)
### Preview style
<!-- Changes that affect Black's preview style -->
- Fix a bug where one-liner functions/conditionals marked with `# fmt: skip` would still
be formatted (#4552)
### Configuration
<!-- Changes to how Black can be configured -->
### Packaging
<!-- Changes to how Black is packaged, such as dependency requirements -->
### Parser
<!-- Changes to the parser or to version autodetection -->
- Rewrite tokenizer to improve performance and compliance (#4536)
- Fix bug where certain unusual expressions (e.g., lambdas) were not accepted in type
parameter bounds and defaults. (#4602)
### Performance
<!-- Changes that improve Black's performance. -->
### Output
<!-- Changes to Black's terminal output and error messages -->
### _Blackd_
<!-- Changes to blackd -->
### Integrations
<!-- For example, Docker, GitHub Actions, pre-commit, editors -->
- Fix the version check in the vim file to reject Python 3.8 (#4567)
- Enhance GitHub Action `psf/black` to read Black version from an additional section in
pyproject.toml: `[project.dependency-groups]` (#4606)
### Documentation
<!-- Major changes to documentation and policies. Small docs changes
don't need a changelog entry. -->
## 25.1.0
### Highlights
This release introduces the new 2025 stable style (#4558), stabilizing the following
changes:
- Normalize casing of Unicode escape characters in strings to lowercase (#2916)
- Fix inconsistencies in whether certain strings are detected as docstrings (#4095)
- Consistently add trailing commas to typed function parameters (#4164)
- Remove redundant parentheses in if guards for case blocks (#4214)
- Add parentheses to if clauses in case blocks when the line is too long (#4269)
- Whitespace before `# fmt: skip` comments is no longer normalized (#4146)
- Fix line length computation for certain expressions that involve the power operator
(#4154)
- Check if there is a newline before the terminating quotes of a docstring (#4185)
- Fix type annotation spacing between `*` and more complex type variable tuple (#4440)
The following changes were not in any previous release:
- Remove parentheses around sole list items (#4312)
- Generic function definitions are now formatted more elegantly: parameters are split
over multiple lines first instead of type parameter definitions (#4553)
### Stable style
- Fix formatting cells in IPython notebooks with magic methods and starting or trailing
empty lines (#4484)
- Fix crash when formatting `with` statements containing tuple generators/unpacking
(#4538)
### Preview style
- Fix/remove string merging changing f-string quotes on f-strings with internal quotes
(#4498)
- Collapse multiple empty lines after an import into one (#4489)
- Prevent `string_processing` and `wrap_long_dict_values_in_parens` from removing
parentheses around long dictionary values (#4377)
- Move `wrap_long_dict_values_in_parens` from the unstable to preview style (#4561)
### Packaging
- Store license identifier inside the `License-Expression` metadata field, see
[PEP 639](https://peps.python.org/pep-0639/). (#4479)
### Performance
- Speed up the `is_fstring_start` function in Black's tokenizer (#4541)
### Integrations
- If using stdin with `--stdin-filename` set to a force excluded path, stdin won't be
formatted. (#4539)
## 24.10.0
### Highlights
- Black is now officially tested with Python 3.13 and provides Python 3.13
mypyc-compiled wheels. (#4436) (#4449)
- Black will issue an error when used with Python 3.12.5, due to an upstream memory
safety issue in Python 3.12.5 that can cause Black's AST safety checks to fail. Please
use Python 3.12.6 or Python 3.12.4 instead. (#4447)
- Black no longer supports running with Python 3.8 (#4452)
### Stable style
- Fix crashes involving comments in parenthesised return types or `X | Y` style unions.
(#4453)
- Fix skipping Jupyter cells with unknown `%%` magic (#4462)
### Preview style
- Fix type annotation spacing between * and more complex type variable tuple (i.e. `def
fn(*args: *tuple[*Ts, T]) -> None: pass`) (#4440)
### Caching
- Fix bug where the cache was shared between runs with and without `--unstable` (#4466)
### Packaging
- Upgrade version of mypyc used to 1.12 beta (#4450) (#4449)
- `blackd` now requires a newer version of aiohttp. (#4451)
### Output
- Added Python target version information on parse error (#4378)
- Add information about Black version to internal error messages (#4457)
## 24.8.0
### Stable style
- Fix crash when `# fmt: off` is used before a closing parenthesis or bracket. (#4363)
### Packaging
- Packaging metadata updated: docs are explictly linked, the issue tracker is now also
linked. This improves the PyPI listing for Black. (#4345)
### Parser
- Fix regression where Black failed to parse a multiline f-string containing another
multiline string (#4339)
- Fix regression where Black failed to parse an escaped single quote inside an f-string
(#4401)
- Fix bug with Black incorrectly parsing empty lines with a backslash (#4343)
- Fix bugs with Black's tokenizer not handling `\{` inside f-strings very well (#4422)
- Fix incorrect line numbers in the tokenizer for certain tokens within f-strings
(#4423)
### Performance
- Improve performance when a large directory is listed in `.gitignore` (#4415)
### _Blackd_
- Fix blackd (and all extras installs) for docker container (#4357)
## 24.4.2
This is a bugfix release to fix two regressions in the new f-string parser introduced in
24.4.1.
### Parser
- Fix regression where certain complex f-strings failed to parse (#4332)
### Performance
- Fix bad performance on certain complex string literals (#4331)
## 24.4.1
### Highlights
- Add support for the new Python 3.12 f-string syntax introduced by PEP 701 (#3822)
### Stable style
- Fix crash involving indented dummy functions containing newlines (#4318)
### Parser
- Add support for type parameter defaults, a new syntactic feature added to Python 3.13
by PEP 696 (#4327)
### Integrations
- Github Action now works even when `git archive` is skipped (#4313)
## 24.4.0
### Stable style
- Fix unwanted crashes caused by AST equivalency check (#4290)
### Preview style
- `if` guards in `case` blocks are now wrapped in parentheses when the line is too long.
(#4269)
- Stop moving multiline strings to a new line unless inside brackets (#4289)
### Integrations
- Add a new option `use_pyproject` to the GitHub Action `psf/black`. This will read the
Black version from `pyproject.toml`. (#4294)
## 24.3.0
### Highlights
This release is a milestone: it fixes Black's first CVE security vulnerability. If you
run Black on untrusted input, or if you habitually put thousands of leading tab
characters in your docstrings, you are strongly encouraged to upgrade immediately to fix
[CVE-2024-21503](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-21503).
This release also fixes a bug in Black's AST safety check that allowed Black to make
incorrect changes to certain f-strings that are valid in Python 3.12 and higher.
### Stable style
- Don't move comments along with delimiters, which could cause crashes (#4248)
- Strengthen AST safety check to catch more unsafe changes to strings. Previous versions
of Black would incorrectly format the contents of certain unusual f-strings containing
nested strings with the same quote type. Now, Black will crash on such strings until
support for the new f-string syntax is implemented. (#4270)
- Fix a bug where line-ranges exceeding the last code line would not work as expected
(#4273)
### Performance
- Fix catastrophic performance on docstrings that contain large numbers of leading tab
characters. This fixes
[CVE-2024-21503](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-21503).
(#4278)
### Documentation
- Note what happens when `--check` is used with `--quiet` (#4236)
## 24.2.0
### Stable style
- Fixed a bug where comments where mistakenly removed along with redundant parentheses
(#4218)
### Preview style
- Move the `hug_parens_with_braces_and_square_brackets` feature to the unstable style
due to an outstanding crash and proposed formatting tweaks (#4198)
- Fixed a bug where base expressions caused inconsistent formatting of \*\* in tenary
expression (#4154)
- Checking for newline before adding one on docstring that is almost at the line limit
(#4185)
- Remove redundant parentheses in `case` statement `if` guards (#4214).
### Configuration
- Fix issue where _Black_ would ignore input files in the presence of symlinks (#4222)
- _Black_ now ignores `pyproject.toml` that is missing a `tool.black` section when
discovering project root and configuration. Since _Black_ continues to use version
control as an indicator of project root, this is expected to primarily change behavior
for users in a monorepo setup (desirably). If you wish to preserve previous behavior,
simply add an empty `[tool.black]` to the previously discovered `pyproject.toml`
(#4204)
### Output
- Black will swallow any `SyntaxWarning`s or `DeprecationWarning`s produced by the `ast`
module when performing equivalence checks (#4189)
### Integrations
- Add a JSONSchema and provide a validate-pyproject entry-point (#4181)
## 24.1.1
Bugfix release to fix a bug that made Black unusable on certain file systems with strict
limits on path length.
### Preview style
- Consistently add trailing comma on typed parameters (#4164)
### Configuration
- Shorten the length of the name of the cache file to fix crashes on file systems that
do not support long paths (#4176)
## 24.1.0
### Highlights
This release introduces the new 2024 stable style (#4106), stabilizing the following
changes:
- Add parentheses around `if`-`else` expressions (#2278)
- Dummy class and function implementations consisting only of `...` are formatted more
compactly (#3796)
- If an assignment statement is too long, we now prefer splitting on the right-hand side
(#3368)
- Hex codes in Unicode escape sequences are now standardized to lowercase (#2916)
- Allow empty first lines at the beginning of most blocks (#3967, #4061)
- Add parentheses around long type annotations (#3899)
- Enforce newline after module docstrings (#3932, #4028)
- Fix incorrect magic trailing comma handling in return types (#3916)
- Remove blank lines before class docstrings (#3692)
- Wrap multiple context managers in parentheses if combined in a single `with` statement
(#3489)
- Fix bug in line length calculations for power operations (#3942)
- Add trailing commas to collection literals even if there's a comment after the last
entry (#3393)
- When using `--skip-magic-trailing-comma` or `-C`, trailing commas are stripped from
subscript expressions with more than 1 element (#3209)
- Add extra blank lines in stubs in a few cases (#3564, #3862)
- Accept raw strings as docstrings (#3947)
- Split long lines in case blocks (#4024)
- Stop removing spaces from walrus operators within subscripts (#3823)
- Fix incorrect formatting of certain async statements (#3609)
- Allow combining `# fmt: skip` with other comments (#3959)
There are already a few improvements in the `--preview` style, which are slated for the
2025 stable style. Try them out and
[share your feedback](https://github.com/psf/black/issues). In the past, the preview
style has included some features that we were not able to stabilize. This year, we're
adding a separate `--unstable` style for features with known problems. Now, the
`--preview` style only includes features that we actually expect to make it into next
year's stable style.
### Stable style
Several bug fixes were made in features that are moved to the stable style in this
release:
- Fix comment handling when parenthesising conditional expressions (#4134)
- Fix bug where spaces were not added around parenthesized walruses in subscripts,
unlike other binary operators (#4109)
- Remove empty lines before docstrings in async functions (#4132)
- Address a missing case in the change to allow empty lines at the beginning of all
blocks, except immediately before a docstring (#4130)
- For stubs, fix logic to enforce empty line after nested classes with bodies (#4141)
### Preview style
- Add `--unstable` style, covering preview features that have known problems that would
block them from going into the stable style. Also add the `--enable-unstable-feature`
flag; for example, use
`--enable-unstable-feature hug_parens_with_braces_and_square_brackets` to apply this
preview feature throughout 2024, even if a later Black release downgrades the feature
to unstable (#4096)
- Format module docstrings the same as class and function docstrings (#4095)
- Fix crash when using a walrus in a dictionary (#4155)
- Fix unnecessary parentheses when wrapping long dicts (#4135)
- Stop normalizing spaces before `# fmt: skip` comments (#4146)
### Configuration
- Print warning when configuration in `pyproject.toml` contains an invalid key (#4165)
- Fix symlink handling, properly ignoring symlinks that point outside of root (#4161)
- Fix cache mtime logic that resulted in false positive cache hits (#4128)
- Remove the long-deprecated `--experimental-string-processing` flag. This feature can
currently be enabled with `--preview --enable-unstable-feature string_processing`.
(#4096)
### Integrations
- Revert the change to run Black's pre-commit integration only on specific git hooks
(#3940) for better compatibility with older versions of pre-commit (#4137)
## 23.12.1
### Packaging
- Fixed a bug that included dependencies from the `d` extra by default (#4108)
## 23.12.0
### Highlights
It's almost 2024, which means it's time for a new edition of _Black_'s stable style!
Together with this release, we'll put out an alpha release 24.1a1 showcasing the draft
2024 stable style, which we'll finalize in the January release. Please try it out and
[share your feedback](https://github.com/psf/black/issues/4042).
This release (23.12.0) will still produce the 2023 style. Most but not all of the
changes in `--preview` mode will be in the 2024 stable style.
### Stable style
- Fix bug where `# fmt: off` automatically dedents when used with the `--line-ranges`
option, even when it is not within the specified line range. (#4084)
- Fix feature detection for parenthesized context managers (#4104)
### Preview style
- Prefer more equal signs before a break when splitting chained assignments (#4010)
- Standalone form feed characters at the module level are no longer removed (#4021)
- Additional cases of immediately nested tuples, lists, and dictionaries are now
indented less (#4012)
- Allow empty lines at the beginning of all blocks, except immediately before a
docstring (#4060)
- Fix crash in preview mode when using a short `--line-length` (#4086)
- Keep suites consisting of only an ellipsis on their own lines if they are not
functions or class definitions (#4066) (#4103)
### Configuration
- `--line-ranges` now skips _Black_'s internal stability check in `--safe` mode. This
avoids a crash on rare inputs that have many unformatted same-content lines. (#4034)
### Packaging
- Upgrade to mypy 1.7.1 (#4049) (#4069)
- Faster compiled wheels are now available for CPython 3.12 (#4070)
### Integrations
- Enable 3.12 CI (#4035)
- Build docker images in parallel (#4054)
- Build docker images with 3.12 (#4055)
## 23.11.0
### Highlights

15
CONTRIBUTING.md
View File

@ -1,13 +1,10 @@
# Contributing to _Black_
Welcome future contributor! We're happy to see you willing to make the project better.
Welcome! Happy to see you willing to make the project better. Have you read the entire
[user documentation](https://black.readthedocs.io/en/latest/) yet?
If you aren't familiar with _Black_, or are looking for documentation on something
specific, the [user documentation](https://black.readthedocs.io/en/latest/) is the best
place to look.
Our [contributing documentation](https://black.readthedocs.org/en/latest/contributing/)
contains details on all you need to know about contributing to _Black_, the basics to
the internals of _Black_.
For getting started on contributing, please read the
[contributing documentation](https://black.readthedocs.org/en/latest/contributing/) for
all you need to know.
Thank you, and we look forward to your contributions!
We look forward to your contributions!

7
Dockerfile
View File

@ -1,4 +1,4 @@
FROM python:3.12-slim AS builder
FROM python:3.11-slim AS builder
RUN mkdir /src
COPY . /src/
@ -10,10 +10,9 @@ RUN python -m venv $VIRTUAL_ENV
RUN python -m pip install --no-cache-dir hatch hatch-fancy-pypi-readme hatch-vcs
RUN . /opt/venv/bin/activate && pip install --no-cache-dir --upgrade pip setuptools \
&& cd /src && hatch build -t wheel \
&& pip install --no-cache-dir dist/*-cp* \
&& pip install black[colorama,d,uvloop]
&& pip install --no-cache-dir dist/*-cp*[colorama,d,uvloop]
FROM python:3.12-slim
FROM python:3.11-slim
# copy only Python packages to limit the image size
COPY --from=builder /opt/venv /opt/venv

6
README.md
View File

@ -38,7 +38,7 @@ Try it out now using the [Black Playground](https://black.vercel.app). Watch the
### Installation
_Black_ can be installed by running `pip install black`. It requires Python 3.9+ to run.
_Black_ can be installed by running `pip install black`. It requires Python 3.8+ to run.
If you want to format Jupyter Notebooks, install with `pip install "black[jupyter]"`.
If you can't wait for the latest _hotness_ and want to install from GitHub, use:
@ -137,8 +137,8 @@ SQLAlchemy, Poetry, PyPA applications (Warehouse, Bandersnatch, Pipenv, virtuale
pandas, Pillow, Twisted, LocalStack, every Datadog Agent Integration, Home Assistant,
Zulip, Kedro, OpenOA, FLORIS, ORBIT, WOMBAT, and many more.
The following organizations use _Black_: Dropbox, KeepTruckin, Lyft, Mozilla, Quora,
Duolingo, QuantumBlack, Tesla, Archer Aviation.
The following organizations use _Black_: Facebook, Dropbox, KeepTruckin, Lyft, Mozilla,
Quora, Duolingo, QuantumBlack, Tesla, Archer Aviation.
Are we missing anyone? Let us know.

5
action.yml
View File

@ -27,10 +27,6 @@ inputs:
description: 'Python Version specifier (PEP440) - e.g. "21.5b1"'
required: false
default: ""
use_pyproject:
description: Read Black version specifier from pyproject.toml if `true`.
required: false
default: "false"
summary:
description: "Whether to add the output to the workflow summary"
required: false
@ -74,6 +70,5 @@ runs:
INPUT_JUPYTER: ${{ inputs.jupyter }}
INPUT_BLACK_ARGS: ${{ inputs.black_args }}
INPUT_VERSION: ${{ inputs.version }}
INPUT_USE_PYPROJECT: ${{ inputs.use_pyproject }}
pythonioencoding: utf-8
shell: bash

111
action/main.py
View File

@ -1,11 +1,9 @@
import os
import re
import shlex
import shutil
import sys
from pathlib import Path
from subprocess import PIPE, STDOUT, run
from typing import Union
ACTION_PATH = Path(os.environ["GITHUB_ACTION_PATH"])
ENV_PATH = ACTION_PATH / ".black-env"
@ -15,109 +13,12 @@
JUPYTER = os.getenv("INPUT_JUPYTER") == "true"
BLACK_ARGS = os.getenv("INPUT_BLACK_ARGS", default="")
VERSION = os.getenv("INPUT_VERSION", default="")
USE_PYPROJECT = os.getenv("INPUT_USE_PYPROJECT") == "true"
BLACK_VERSION_RE = re.compile(r"^black([^A-Z0-9._-]+.*)$", re.IGNORECASE)
EXTRAS_RE = re.compile(r"\[.*\]")
EXPORT_SUBST_FAIL_RE = re.compile(r"\$Format:.*\$")
def determine_version_specifier() -> str:
"""Determine the version of Black to install.
The version can be specified either via the `with.version` input or via the
pyproject.toml file if `with.use_pyproject` is set to `true`.
"""
if USE_PYPROJECT and VERSION:
print(
"::error::'with.version' and 'with.use_pyproject' inputs are "
"mutually exclusive.",
file=sys.stderr,
flush=True,
)
sys.exit(1)
if USE_PYPROJECT:
return read_version_specifier_from_pyproject()
elif VERSION and VERSION[0] in "0123456789":
return f"=={VERSION}"
else:
return VERSION
def read_version_specifier_from_pyproject() -> str:
if sys.version_info < (3, 11):
print(
"::error::'with.use_pyproject' input requires Python 3.11 or later.",
file=sys.stderr,
flush=True,
)
sys.exit(1)
import tomllib # type: ignore[import-not-found,unreachable]
try:
with Path("pyproject.toml").open("rb") as fp:
pyproject = tomllib.load(fp)
except FileNotFoundError:
print(
"::error::'with.use_pyproject' input requires a pyproject.toml file.",
file=sys.stderr,
flush=True,
)
sys.exit(1)
version = pyproject.get("tool", {}).get("black", {}).get("required-version")
if version is not None:
return f"=={version}"
arrays = [
*pyproject.get("dependency-groups", {}).values(),
pyproject.get("project", {}).get("dependencies"),
*pyproject.get("project", {}).get("optional-dependencies", {}).values(),
]
for array in arrays:
version = find_black_version_in_array(array)
if version is not None:
break
if version is None:
print(
"::error::'black' dependency missing from pyproject.toml.",
file=sys.stderr,
flush=True,
)
sys.exit(1)
return version
def find_black_version_in_array(array: object) -> Union[str, None]:
if not isinstance(array, list):
return None
try:
for item in array:
# Rudimentary PEP 508 parsing.
item = item.split(";")[0]
item = EXTRAS_RE.sub("", item).strip()
if item == "black":
print(
"::error::Version specifier missing for 'black' dependency in "
"pyproject.toml.",
file=sys.stderr,
flush=True,
)
sys.exit(1)
elif m := BLACK_VERSION_RE.match(item):
return m.group(1).strip()
except TypeError:
pass
return None
run([sys.executable, "-m", "venv", str(ENV_PATH)], check=True)
version_specifier = determine_version_specifier()
version_specifier = VERSION
if VERSION and VERSION[0] in "0123456789":
version_specifier = f"=={VERSION}"
if JUPYTER:
extra_deps = "[colorama,jupyter]"
else:
@ -137,11 +38,7 @@ def find_black_version_in_array(array: object) -> Union[str, None]:
# expected format is one of:
# - 23.1.0
# - 23.1.0-51-g448bba7
# - $Format:%(describe:tags=true,match=*[0-9]*)$ (if export-subst fails)
if (
describe_name.count("-") < 2
and EXPORT_SUBST_FAIL_RE.match(describe_name) is None
):
if describe_name.count("-") < 2:
# the action's commit matches a tag exactly, install exact version from PyPI
req = f"black{extra_deps}=={describe_name}"
else:

4
autoload/black.vim
View File

@ -75,8 +75,8 @@ def _initialize_black_env(upgrade=False):
return True
pyver = sys.version_info[:3]
if pyver < (3, 9):
print("Sorry, Black requires Python 3.9+ to run.")
if pyver < (3, 8):
print("Sorry, Black requires Python 3.8+ to run.")
return False
from pathlib import Path

2
docs/compatible_configs/flake8/.flake8
View File

@ -1,3 +1,3 @@
[flake8]
max-line-length = 88
extend-ignore = E203,E701
extend-ignore = E203

2
docs/compatible_configs/flake8/setup.cfg
View File

@ -1,3 +1,3 @@
[flake8]
max-line-length = 88
extend-ignore = E203,E701
extend-ignore = E203

2
docs/compatible_configs/flake8/tox.ini
View File

@ -1,3 +1,3 @@
[flake8]
max-line-length = 88
extend-ignore = E203,E701
extend-ignore = E203

3
docs/compatible_configs/pycodestyle/.flake8
View File

@ -1,3 +0,0 @@
[pycodestyle]
max-line-length = 88
ignore = E203,E701

3
docs/compatible_configs/pycodestyle/setup.cfg
View File

@ -1,3 +0,0 @@
[pycodestyle]
max-line-length = 88
ignore = E203,E701

3
docs/compatible_configs/pycodestyle/tox.ini
View File

@ -1,3 +0,0 @@
[pycodestyle]
max-line-length = 88
ignore = E203,E701

66
docs/conf.py
View File

@ -1,3 +1,4 @@
# -*- coding: utf-8 -*-
#
# Configuration file for the Sphinx documentation builder.
#
@ -13,12 +14,10 @@
#
import os
import re
import string
from importlib.metadata import version
from pathlib import Path
from sphinx.application import Sphinx
from pkg_resources import get_distribution
CURRENT_DIR = Path(__file__).parent
@ -26,33 +25,12 @@
def make_pypi_svg(version: str) -> None:
template: Path = CURRENT_DIR / "_static" / "pypi_template.svg"
target: Path = CURRENT_DIR / "_static" / "pypi.svg"
with open(str(template), encoding="utf8") as f:
with open(str(template), "r", encoding="utf8") as f:
svg: str = string.Template(f.read()).substitute(version=version)
with open(str(target), "w", encoding="utf8") as f:
f.write(svg)
def replace_pr_numbers_with_links(content: str) -> str:
"""Replaces all PR numbers with the corresponding GitHub link."""
return re.sub(r"#(\d+)", r"[#\1](https://github.com/psf/black/pull/\1)", content)
def handle_include_read(
app: Sphinx,
relative_path: Path,
parent_docname: str,
content: list[str],
) -> None:
"""Handler for the include-read sphinx event."""
if parent_docname == "change_log":
content[0] = replace_pr_numbers_with_links(content[0])
def setup(app: Sphinx) -> None:
"""Sets up a minimal sphinx extension."""
app.connect("include-read", handle_include_read)
# Necessary so Click doesn't hit an encode error when called by
# sphinxcontrib-programoutput on Windows.
os.putenv("pythonioencoding", "utf-8")
@ -65,7 +43,7 @@ def setup(app: Sphinx) -> None:
# Autopopulate version
# The version, including alpha/beta/rc tags, but not commit hash and datestamps
release = version("black").split("+")[0]
release = get_distribution("black").version.split("+")[0]
# The short X.Y version.
version = release
for sp in "abcfr":
@ -171,13 +149,15 @@ def setup(app: Sphinx) -> None:
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [(
master_doc,
"black.tex",
"Documentation for Black",
"Łukasz Langa and contributors to Black",
"manual",
)]
latex_documents = [
(
master_doc,
"black.tex",
"Documentation for Black",
"Łukasz Langa and contributors to Black",
"manual",
)
]
# -- Options for manual page output ------------------------------------------
@ -192,15 +172,17 @@ def setup(app: Sphinx) -> None:
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [(
master_doc,
"Black",
"Documentation for Black",
author,
"Black",
"The uncompromising Python code formatter",
"Miscellaneous",
)]
texinfo_documents = [
(
master_doc,
"Black",
"Documentation for Black",
author,
"Black",
"The uncompromising Python code formatter",
"Miscellaneous",
)
]
# -- Options for Epub output -------------------------------------------------

5
docs/contributing/index.md
View File

@ -9,6 +9,7 @@ the_basics
gauging_changes
issue_triage
release_process
reference/reference_summary
```
Welcome! Happy to see you willing to make the project better. Have you read the entire
@ -41,5 +42,9 @@ This section covers the following topics:
- {doc}`the_basics`
- {doc}`gauging_changes`
- {doc}`release_process`
- {doc}`reference/reference_summary`
For an overview on contributing to the _Black_, please checkout {doc}`the_basics`.
If you need a reference of the functions, classes, etc. available to you while
developing _Black_, there's the {doc}`reference/reference_summary` docs.

234
docs/contributing/reference/reference_classes.rst Normal file
View File

@ -0,0 +1,234 @@
*Black* classes
===============
*Contents are subject to change.*
Black Classes
~~~~~~~~~~~~~~
.. currentmodule:: black
:class:`BracketTracker`
-----------------------
.. autoclass:: black.brackets.BracketTracker
:members:
:class:`Line`
-------------
.. autoclass:: black.lines.Line
:members:
:special-members: __str__, __bool__
:class:`RHSResult`
-------------------------
.. autoclass:: black.lines.RHSResult
:members:
:class:`LinesBlock`
-------------------------
.. autoclass:: black.lines.LinesBlock
:members:
:class:`EmptyLineTracker`
-------------------------
.. autoclass:: black.lines.EmptyLineTracker
:members:
:class:`LineGenerator`
----------------------
.. autoclass:: black.linegen.LineGenerator
:show-inheritance:
:members:
:class:`ProtoComment`
---------------------
.. autoclass:: black.comments.ProtoComment
:members:
:class:`Mode`
---------------------
.. autoclass:: black.mode.Mode
:members:
:class:`Report`
---------------
.. autoclass:: black.report.Report
:members:
:special-members: __str__
:class:`Ok`
---------------
.. autoclass:: black.rusty.Ok
:show-inheritance:
:members:
:class:`Err`
---------------
.. autoclass:: black.rusty.Err
:show-inheritance:
:members:
:class:`Visitor`
----------------
.. autoclass:: black.nodes.Visitor
:show-inheritance:
:members:
:class:`StringTransformer`
----------------------------
.. autoclass:: black.trans.StringTransformer
:show-inheritance:
:members:
:class:`CustomSplit`
----------------------------
.. autoclass:: black.trans.CustomSplit
:members:
:class:`CustomSplitMapMixin`
-----------------------------
.. autoclass:: black.trans.CustomSplitMapMixin
:show-inheritance:
:members:
:class:`StringMerger`
----------------------
.. autoclass:: black.trans.StringMerger
:show-inheritance:
:members:
:class:`StringParenStripper`
-----------------------------
.. autoclass:: black.trans.StringParenStripper
:show-inheritance:
:members:
:class:`BaseStringSplitter`
-----------------------------
.. autoclass:: black.trans.BaseStringSplitter
:show-inheritance:
:members:
:class:`StringSplitter`
-----------------------------
.. autoclass:: black.trans.StringSplitter
:show-inheritance:
:members:
:class:`StringParenWrapper`
-----------------------------
.. autoclass:: black.trans.StringParenWrapper
:show-inheritance:
:members:
:class:`StringParser`
-----------------------------
.. autoclass:: black.trans.StringParser
:members:
:class:`DebugVisitor`
------------------------
.. autoclass:: black.debug.DebugVisitor
:show-inheritance:
:members:
:class:`Replacement`
------------------------
.. autoclass:: black.handle_ipynb_magics.Replacement
:members:
:class:`CellMagic`
------------------------
.. autoclass:: black.handle_ipynb_magics.CellMagic
:members:
:class:`CellMagicFinder`
------------------------
.. autoclass:: black.handle_ipynb_magics.CellMagicFinder
:show-inheritance:
:members:
:class:`OffsetAndMagic`
------------------------
.. autoclass:: black.handle_ipynb_magics.OffsetAndMagic
:members:
:class:`MagicFinder`
------------------------
.. autoclass:: black.handle_ipynb_magics.MagicFinder
:show-inheritance:
:members:
:class:`Cache`
------------------------
.. autoclass:: black.cache.Cache
:show-inheritance:
:members:
Enum Classes
~~~~~~~~~~~~~
Classes inherited from Python `Enum <https://docs.python.org/3/library/enum.html#enum.Enum>`_ class.
:class:`Changed`
----------------
.. autoclass:: black.report.Changed
:show-inheritance:
:members:
:class:`WriteBack`
------------------
.. autoclass:: black.WriteBack
:show-inheritance:
:members:
:class:`TargetVersion`
----------------------
.. autoclass:: black.mode.TargetVersion
:show-inheritance:
:members:
:class:`Feature`
------------------
.. autoclass:: black.mode.Feature
:show-inheritance:
:members:
:class:`Preview`
------------------
.. autoclass:: black.mode.Preview
:show-inheritance:
:members:

18
docs/contributing/reference/reference_exceptions.rst Normal file
View File

@ -0,0 +1,18 @@
*Black* exceptions
==================
*Contents are subject to change.*
.. currentmodule:: black
.. autoexception:: black.trans.CannotTransform
.. autoexception:: black.linegen.CannotSplit
.. autoexception:: black.brackets.BracketMatchError
.. autoexception:: black.report.NothingChanged
.. autoexception:: black.parsing.InvalidInput
.. autoexception:: black.mode.Deprecated

170
docs/contributing/reference/reference_functions.rst Normal file
View File

@ -0,0 +1,170 @@
*Black* functions
=================
*Contents are subject to change.*
.. currentmodule:: black
Assertions and checks
---------------------
.. autofunction:: black.assert_equivalent
.. autofunction:: black.assert_stable
.. autofunction:: black.lines.can_be_split
.. autofunction:: black.lines.can_omit_invisible_parens
.. autofunction:: black.nodes.is_empty_tuple
.. autofunction:: black.nodes.is_import
.. autofunction:: black.lines.is_line_short_enough
.. autofunction:: black.nodes.is_multiline_string
.. autofunction:: black.nodes.is_one_tuple
.. autofunction:: black.brackets.is_split_after_delimiter
.. autofunction:: black.brackets.is_split_before_delimiter
.. autofunction:: black.nodes.is_stub_body
.. autofunction:: black.nodes.is_stub_suite
.. autofunction:: black.nodes.is_vararg
.. autofunction:: black.nodes.is_yield
Formatting
----------
.. autofunction:: black.format_file_contents
.. autofunction:: black.format_file_in_place
.. autofunction:: black.format_stdin_to_stdout
.. autofunction:: black.format_str
.. autofunction:: black.reformat_one
.. autofunction:: black.concurrency.schedule_formatting
File operations
---------------
.. autofunction:: black.dump_to_file
.. autofunction:: black.find_project_root
.. autofunction:: black.gen_python_files
.. autofunction:: black.read_pyproject_toml
Parsing
-------
.. autofunction:: black.decode_bytes
.. autofunction:: black.parsing.lib2to3_parse
.. autofunction:: black.parsing.lib2to3_unparse
Split functions
---------------
.. autofunction:: black.linegen.bracket_split_build_line
.. autofunction:: black.linegen.bracket_split_succeeded_or_raise
.. autofunction:: black.linegen.delimiter_split
.. autofunction:: black.linegen.left_hand_split
.. autofunction:: black.linegen.right_hand_split
.. autofunction:: black.linegen.standalone_comment_split
.. autofunction:: black.linegen.transform_line
Caching
-------
.. autofunction:: black.cache.get_cache_dir
.. autofunction:: black.cache.get_cache_file
Utilities
---------
.. py:function:: black.debug.DebugVisitor.show(code: str) -> None
Pretty-print the lib2to3 AST of a given string of `code`.
.. autofunction:: black.concurrency.cancel
.. autofunction:: black.nodes.child_towards
.. autofunction:: black.nodes.container_of
.. autofunction:: black.comments.convert_one_fmt_off_pair
.. autofunction:: black.diff
.. autofunction:: black.linegen.dont_increase_indentation
.. autofunction:: black.numerics.format_float_or_int_string
.. autofunction:: black.nodes.ensure_visible
.. autofunction:: black.lines.enumerate_reversed
.. autofunction:: black.comments.generate_comments
.. autofunction:: black.comments.generate_ignored_nodes
.. autofunction:: black.comments.is_fmt_on
.. autofunction:: black.comments.children_contains_fmt_on
.. autofunction:: black.nodes.first_leaf_of
.. autofunction:: black.linegen.generate_trailers_to_omit
.. autofunction:: black.get_future_imports
.. autofunction:: black.comments.list_comments
.. autofunction:: black.comments.make_comment
.. autofunction:: black.linegen.maybe_make_parens_invisible_in_atom
.. autofunction:: black.brackets.max_delimiter_priority_in_atom
.. autofunction:: black.normalize_fmt_off
.. autofunction:: black.numerics.normalize_numeric_literal
.. autofunction:: black.linegen.normalize_prefix
.. autofunction:: black.strings.normalize_string_prefix
.. autofunction:: black.strings.normalize_string_quotes
.. autofunction:: black.linegen.normalize_invisible_parens
.. autofunction:: black.nodes.preceding_leaf
.. autofunction:: black.re_compile_maybe_verbose
.. autofunction:: black.linegen.should_split_line
.. autofunction:: black.concurrency.shutdown
.. autofunction:: black.strings.sub_twice
.. autofunction:: black.nodes.whitespace

19
docs/contributing/reference/reference_summary.rst Normal file
View File

@ -0,0 +1,19 @@
Developer reference
===================
.. note::
As of June 2023, the documentation of *Black classes* and *Black exceptions*
has been updated to the latest available version.
The documentation of *Black functions* is quite outdated and has been neglected. Many
functions worthy of inclusion aren't documented. Contributions are appreciated!
*Contents are subject to change.*
.. toctree::
:maxdepth: 2
reference_classes
reference_functions
reference_exceptions

44
docs/contributing/the_basics.md
View File

@ -7,14 +7,7 @@ An overview on contributing to the _Black_ project.
Development on the latest version of Python is preferred. You can use any operating
system.
First clone the _Black_ repository:
```console
$ git clone https://github.com/psf/black.git
$ cd black
```
Then install development dependencies inside a virtual environment of your choice, for
Install development dependencies inside a virtual environment of your choice, for
example:
```console
@ -23,7 +16,7 @@ $ source .venv/bin/activate # activation for linux and mac
$ .venv\Scripts\activate # activation for windows
(.venv)$ pip install -r test_requirements.txt
(.venv)$ pip install -e ".[d]"
(.venv)$ pip install -e .[d]
(.venv)$ pre-commit install
```
@ -55,16 +48,13 @@ Further examples of invoking the tests
# Run tests on a specific python version
(.venv)$ tox -e py39
# Run an individual test
(.venv)$ pytest -k <test name>
# Pass arguments to pytest
# pass arguments to pytest
(.venv)$ tox -e py -- --no-cov
# Print full tree diff, see documentation below
# print full tree diff, see documentation below
(.venv)$ tox -e py -- --print-full-tree
# Disable diff printing, see documentation below
# disable diff printing, see documentation below
(.venv)$ tox -e py -- --print-tree-diff=False
```
@ -109,22 +99,16 @@ default. To turn it off pass `--print-tree-diff=False`.
`Black` has CI that will check for an entry corresponding to your PR in `CHANGES.md`. If
you feel this PR does not require a changelog entry please state that in a comment and a
maintainer can add a `skip news` label to make the CI pass. Otherwise, please ensure you
have a line in the following format added below the appropriate header:
have a line in the following format:
```md
- `Black` is now more awesome (#X)
```
<!---
The Next PR Number link uses HTML because of a bug in MyST-Parser that double-escapes the ampersand, causing the query parameters to not be processed.
MyST-Parser issue: https://github.com/executablebooks/MyST-Parser/issues/760
MyST-Parser stalled fix PR: https://github.com/executablebooks/MyST-Parser/pull/929
-->
Note that X should be your PR number, not issue number! To workout X, please use
<a href="https://ichard26.github.io/next-pr-number/?owner=psf&name=black">Next PR
Number</a>. This is not perfect but saves a lot of release overhead as now the releaser
does not need to go back and workout what to add to the `CHANGES.md` for each release.
[Next PR Number](https://ichard26.github.io/next-pr-number/?owner=psf&name=black). This
is not perfect but saves a lot of release overhead as now the releaser does not need to
go back and workout what to add to the `CHANGES.md` for each release.
### Style Changes
@ -132,7 +116,7 @@ If a change would affect the advertised code style, please modify the documentat
_Black_ code style) to reflect that change. Patches that fix unintended bugs in
formatting don't need to be mentioned separately though. If the change is implemented
with the `--preview` flag, please include the change in the future style document
instead and write the changelog entry under the dedicated "Preview style" heading.
instead and write the changelog entry under a dedicated "Preview changes" heading.
### Docs Testing
@ -140,17 +124,17 @@ If you make changes to docs, you can test they still build locally too.
```console
(.venv)$ pip install -r docs/requirements.txt
(.venv)$ pip install -e ".[d]"
(.venv)$ pip install -e .[d]
(.venv)$ sphinx-build -a -b html -W docs/ docs/_build/
```
## Hygiene
If you're fixing a bug, add a test. Run it first to confirm it fails, then fix the bug,
and run the test again to confirm it's really fixed.
run it again to confirm it's really fixed.
If adding a new feature, add a test. In fact, always add a test. If adding a large
feature, please first open an issue to discuss it beforehand.
If adding a new feature, add a test. In fact, always add a test. But wait, before adding
any large feature, first open an issue for us to discuss the idea first.
## Finally

29
docs/faq.md
View File

@ -41,10 +41,9 @@ other tools, such as `# noqa`, may be moved by _Black_. See below for more detai
Stable. _Black_ aims to enforce one style and one style only, with some room for
pragmatism. See [The Black Code Style](the_black_code_style/index.md) for more details.
Starting in 2022, the formatting output is stable for the releases made in the same year
(other than unintentional bugs). At the beginning of every year, the first release will
make changes to the stable style. It is possible to opt in to the latest formatting
styles using the `--preview` flag.
Starting in 2022, the formatting output will be stable for the releases made in the same
year (other than unintentional bugs). It is possible to opt-in to the latest formatting
styles, using the `--preview` flag.
## Why is my file not formatted?
@ -77,26 +76,26 @@ following will not be formatted:
- invalid syntax, as it can't be safely distinguished from automagics in the absence of
a running `IPython` kernel.
## Why does Flake8 report warnings?
## Why are Flake8's E203 and W503 violated?
Some of Flake8's rules conflict with Black's style. We recommend disabling these rules.
See [Using _Black_ with other tools](labels/why-pycodestyle-warnings).
Because they go against PEP 8. E203 falsely triggers on list
[slices](the_black_code_style/current_style.md#slices), and adhering to W503 hinders
readability because operators are misaligned. Disable W503 and enable the
disabled-by-default counterpart W504. E203 should be disabled while changes are still
[discussed](https://github.com/PyCQA/pycodestyle/issues/373).
## Which Python versions does Black support?
_Black_ generally supports all Python versions supported by CPython (see
[the Python devguide](https://devguide.python.org/versions/) for current information).
We promise to support at least all Python versions that have not reached their end of
life. This is the case for both running _Black_ and formatting code.
Currently the runtime requires Python 3.8-3.11. Formatting is supported for files
containing syntax from Python 3.3 to 3.11. We promise to support at least all Python
versions that have not reached their end of life. This is the case for both running
_Black_ and formatting code.
Support for formatting Python 2 code was removed in version 22.0. While we've made no
plans to stop supporting older Python 3 minor versions immediately, their support might
also be removed some time in the future without a deprecation period.
`await`/`async` as soft keywords/indentifiers are no longer supported as of 25.2.0.
Runtime support for 3.6 was removed in version 22.10.0, for 3.7 in version 23.7.0, and
for 3.8 in version 24.10.0.
Runtime support for 3.7 was removed in version 23.7.0.
## Why does my linter or typechecker complain after I format my code?

4
docs/getting_started.md
View File

@ -16,11 +16,9 @@ Also, you can try out _Black_ online for minimal fuss on the
## Installation
_Black_ can be installed by running `pip install black`. It requires Python 3.9+ to run.
_Black_ can be installed by running `pip install black`. It requires Python 3.8+ to run.
If you want to format Jupyter Notebooks, install with `pip install "black[jupyter]"`.
If you use pipx, you can install Black with `pipx install black`.
If you can't wait for the latest _hotness_ and want to install from GitHub, use:
`pip install git+https://github.com/psf/black`

142
docs/guides/using_black_with_other_tools.md
View File

@ -134,10 +134,10 @@ profile = black
</details>
### pycodestyle
### Flake8
[pycodestyle](https://pycodestyle.pycqa.org/) is a code linter. It warns you of syntax
errors, possible bugs, stylistic errors, etc. For the most part, pycodestyle follows
[Flake8](https://pypi.org/p/flake8/) is a code linter. It warns you of syntax errors,
possible bugs, stylistic errors, etc. For the most part, Flake8 follows
[PEP 8](https://www.python.org/dev/peps/pep-0008/) when warning about stylistic errors.
There are a few deviations that cause incompatibilities with _Black_.
@ -145,115 +145,67 @@ There are a few deviations that cause incompatibilities with _Black_.
```
max-line-length = 88
ignore = E203,E701
extend-ignore = E203
```
(labels/why-pycodestyle-warnings)=
#### Why those options above?
##### `max-line-length`
As with isort, pycodestyle should be configured to allow lines up to the length limit of
`88`, _Black_'s default.
##### `E203`
In some cases, as determined by PEP 8, _Black_ will enforce an equal amount of
whitespace around slice operators. Due to this, pycodestyle will raise
`E203 whitespace before ':'` warnings. Since this warning is not PEP 8 compliant, it
should be disabled.
##### `E701` / `E704`
_Black_ will collapse implementations of classes and functions consisting solely of `..`
to a single line. This matches how such examples are formatted in PEP 8. It remains true
that in all other cases Black will prevent multiple statements on the same line, in
accordance with PEP 8 generally discouraging this.
However, `pycodestyle` does not mirror this logic and may raise
`E701 multiple statements on one line (colon)` in this situation. Its
disabled-by-default `E704 multiple statements on one line (def)` rule may also raise
warnings and should not be enabled.
##### `W503`
whitespace around slice operators. Due to this, Flake8 will raise
`E203 whitespace before ':'` warnings. Since this warning is not PEP 8 compliant, Flake8
should be configured to ignore it via `extend-ignore = E203`.
When breaking a line, _Black_ will break it before a binary operator. This is compliant
with PEP 8 as of
[April 2016](https://github.com/python/peps/commit/c59c4376ad233a62ca4b3a6060c81368bd21e85b#diff-64ec08cc46db7540f18f2af46037f599).
There's a disabled-by-default warning in Flake8 which goes against this PEP 8
recommendation called `W503 line break before binary operator`. It should not be enabled
in your configuration. You can use its counterpart
`W504 line break after binary operator` instead.
in your configuration.
Also, as like with isort, flake8 should be configured to allow lines up to the length
limit of `88`, _Black_'s default. This explains `max-line-length = 88`.
#### Formats
<details>
<summary>setup.cfg, .pycodestyle, tox.ini</summary>
<summary>.flake8</summary>
```ini
[pycodestyle]
[flake8]
max-line-length = 88
ignore = E203,E701
extend-ignore = E203, E704
```
</details>
### Flake8
[Flake8](https://pypi.org/p/flake8/) is a wrapper around multiple linters, including
pycodestyle. As such, it has many of the same issues.
#### Bugbear
It's recommended to use [the Bugbear plugin](https://github.com/PyCQA/flake8-bugbear)
and enable
[its B950 check](https://github.com/PyCQA/flake8-bugbear#opinionated-warnings#:~:text=you%20expect%20it.-,B950,-%3A%20Line%20too%20long)
instead of using Flake8's E501, because it aligns with
[Black's 10% rule](labels/line-length).
Install Bugbear and use the following config:
```
[flake8]
max-line-length = 80
extend-select = B950
extend-ignore = E203,E501,E701
```
#### Minimal Configuration
In cases where you can't or don't want to install Bugbear, you can use this minimally
compatible config:
```
[flake8]
max-line-length = 88
extend-ignore = E203,E701
```
#### Why those options above?
See [the pycodestyle section](labels/why-pycodestyle-warnings) above.
#### Formats
<details>
<summary>.flake8, setup.cfg, tox.ini</summary>
<summary>setup.cfg</summary>
```ini
[flake8]
max-line-length = 88
extend-ignore = E203,E701
extend-ignore = E203
```
</details>
<details>
<summary>tox.ini</summary>
```ini
[flake8]
max-line-length = 88
extend-ignore = E203
```
</details>
### Pylint
[Pylint](https://pypi.org/p/pylint/) is also a code linter like Flake8. It has many of
the same checks as Flake8 and more. It particularly has more formatting checks regarding
style conventions like variable naming.
[Pylint](https://pypi.org/p/pylint/) is also a code linter like Flake8. It has the same
checks as flake8 and more. In particular, it has more formatting checks regarding style
conventions like variable naming. With so many checks, Pylint is bound to have some
mixed feelings about _Black_'s formatting style.
#### Configuration
@ -300,3 +252,35 @@ max-line-length = "88"
```
</details>
### pycodestyle
[pycodestyle](https://pycodestyle.pycqa.org/) is also a code linter like Flake8.
#### Configuration
```
max-line-length = 88
ignore = E203
```
#### Why those options above?
pycodestyle should be configured to only complain about lines that surpass `88`
characters via `max_line_length = 88`.
See
[Why are Flake8’s E203 and W503 violated?](https://black.readthedocs.io/en/stable/faq.html#why-are-flake8-s-e203-and-w503-violated)
#### Formats
<details>
<summary>setup.cfg</summary>
```cfg
[pycodestyle]
ignore = E203
max_line_length = 88
```
</details>

4
docs/integrations/editors.md
View File

@ -236,7 +236,7 @@ Configuration:
#### Installation
This plugin **requires Vim 7.0+ built with Python 3.9+ support**. It needs Python 3.9 to
This plugin **requires Vim 7.0+ built with Python 3.8+ support**. It needs Python 3.8 to
be able to run _Black_ inside the Vim process which is much faster than calling an
external command.
@ -432,4 +432,4 @@ hook global WinSetOption filetype=python %{
## Thonny
Use [Thonny-black-formatter](https://pypi.org/project/thonny-black-formatter/).
Use [Thonny-black-code-format](https://github.com/Franccisco/thonny-black-code-format).

30
docs/integrations/github_actions.md
View File

@ -24,7 +24,7 @@ jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v3
- uses: psf/black@stable
```
@ -32,15 +32,12 @@ We recommend the use of the `@stable` tag, but per version tags also exist if yo
that. Note that the action's version you select is independent of the version of _Black_
the action will use.
The version of _Black_ the action will use can be configured via `version` or read from
the `pyproject.toml` file. `version` can be any
The version of _Black_ the action will use can be configured via `version`. This can be
any
[valid version specifier](https://packaging.python.org/en/latest/glossary/#term-Version-Specifier)
or just the version number if you want an exact version. To read the version from the
`pyproject.toml` file instead, set `use_pyproject` to `true`. This will first look into
the `tool.black.required-version` field, then the `dependency-groups` table, then the
`project.dependencies` array and finally the `project.optional-dependencies` table. The
action defaults to the latest release available on PyPI. Only versions available from
PyPI are supported, so no commit SHAs or branch names.
or just the version number if you want an exact version. The action defaults to the
latest release available on PyPI. Only versions available from PyPI are supported, so no
commit SHAs or branch names.
If you want to include Jupyter Notebooks, _Black_ must be installed with the `jupyter`
extra. Installing the extra and including Jupyter Notebook files can be configured via
@ -73,18 +70,3 @@ If you want to match versions covered by Black's
src: "./src"
version: "~= 22.0"
```
If you want to read the version from `pyproject.toml`, set `use_pyproject` to `true`.
Note that this requires Python >= 3.11, so using the setup-python action may be
required, for example:
```yaml
- uses: actions/setup-python@v5
with:
python-version: "3.13"
- uses: psf/black@stable
with:
options: "--check --verbose"
src: "./src"
use_pyproject: true
```

4
docs/integrations/source_version_control.md
View File

@ -8,7 +8,7 @@ Use [pre-commit](https://pre-commit.com/). Once you
repos:
# Using this mirror lets us use mypyc-compiled black, which is about 2x faster
- repo: https://github.com/psf/black-pre-commit-mirror
rev: 25.1.0
rev: 23.11.0
hooks:
- id: black
# It is recommended to specify the latest version of Python
@ -35,7 +35,7 @@ include Jupyter Notebooks. To use this hook, simply replace the hook's `id: blac
repos:
# Using this mirror lets us use mypyc-compiled black, which is about 2x faster
- repo: https://github.com/psf/black-pre-commit-mirror
rev: 25.1.0
rev: 23.11.0
hooks:
- id: black-jupyter
# It is recommended to specify the latest version of Python

10
docs/requirements.txt
View File

@ -1,9 +1,9 @@
# Used by ReadTheDocs; pinned requirements for stability.
myst-parser==4.0.1
Sphinx==8.2.3
myst-parser==2.0.0
Sphinx==7.2.6
# Older versions break Sphinx even though they're declared to be supported.
docutils==0.21.2
sphinxcontrib-programoutput==0.18
docutils==0.20.1
sphinxcontrib-programoutput==0.17
sphinx_copybutton==0.5.2
furo==2024.8.6
furo==2023.9.10

100
docs/the_black_code_style/current_style.md
View File

@ -8,9 +8,18 @@ deliberately limited and rarely added. Previous formatting is taken into account
little as possible, with rare exceptions like the magic trailing comma. The coding style
used by _Black_ can be viewed as a strict subset of PEP 8.
This document describes the current formatting style. If you're interested in trying out
where the style is heading, see [future style](./future_style.md) and try running
`black --preview`.
_Black_ reformats entire files in place. It doesn't reformat lines that contain
`# fmt: skip` or blocks that start with `# fmt: off` and end with `# fmt: on`.
`# fmt: skip` can be mixed with other pragmas/comments either with multiple comments
(e.g. `# fmt: skip # pylint # noqa`) or as a semicolon separated list (e.g.
`# fmt: skip; pylint; noqa`). `# fmt: on/off` must be on the same level of indentation
and in the same block, meaning no unindents beyond the initial indentation level between
them. It also recognizes [YAPF](https://github.com/google/yapf)'s block comments to the
same effect, as a courtesy for straddling code.
The rest of this document describes the current formatting style. If you're interested
in trying out where the style is heading, see [future style](./future_style.md) and try
running `black --preview`.
### How _Black_ wraps lines
@ -143,7 +152,7 @@ significantly shorter files than sticking with 80 (the most popular), or even 79
by the standard library). In general,
[90-ish seems like the wise choice](https://youtu.be/wf-BqAjZb8M?t=260).
If you're paid by the lines of code you write, you can pass `--line-length` with a lower
If you're paid by the line of code you write, you can pass `--line-length` with a lower
number. _Black_ will try to respect that. However, sometimes it won't be able to without
breaking other rules. In those rare cases, auto-formatted code will exceed your allotted
limit.
@ -153,10 +162,35 @@ harder to work with line lengths exceeding 100 characters. It also adversely aff
side-by-side diff review on typical screen resolutions. Long lines also make it harder
to present code neatly in documentation or talk slides.
#### Flake8 and other linters
#### Flake8
See [Using _Black_ with other tools](../guides/using_black_with_other_tools.md) about
linter compatibility.
If you use Flake8, you have a few options:
1. Recommended is using [Bugbear](https://github.com/PyCQA/flake8-bugbear) and enabling
its B950 check instead of using Flake8's E501, because it aligns with Black's 10%
rule. Install Bugbear and use the following config:
```ini
[flake8]
max-line-length = 80
...
select = C,E,F,W,B,B950
extend-ignore = E203, E501, E704
```
The rationale for B950 is explained in
[Bugbear's documentation](https://github.com/PyCQA/flake8-bugbear#opinionated-warnings).
2. For a minimally compatible config:
```ini
[flake8]
max-line-length = 88
extend-ignore = E203, E704
```
An explanation of why E203 is disabled can be found in the [Slices section](#slices) of
this page.
### Empty lines
@ -166,35 +200,44 @@ that in-function vertical whitespace should only be used sparingly.
_Black_ will allow single empty lines inside functions, and single and double empty
lines on module level left by the original editors, except when they're within
parenthesized expressions. Since such expressions are always reformatted to fit minimal
space, this whitespace is lost.
space, this whitespace is lost. The other exception is that it will remove any empty
lines immediately following a statement that introduces a new indentation level.
```python
# in:
def function(
some_argument: int,
def foo():
other_argument: int = 5,
) -> EmptyLineInParenWillBeDeleted:
print("All the newlines above me should be deleted!")
if condition:
print("One empty line above me will be kept!")
print("No newline above me!")
print("There is a newline above me, and that's OK!")
class Point:
x: int
y: int
def this_is_okay_too():
print("No empty line here")
# out:
def function(
some_argument: int,
other_argument: int = 5,
) -> EmptyLineInParenWillBeDeleted:
print("One empty line above me will be kept!")
def foo():
print("All the newlines above me should be deleted!")
def this_is_okay_too():
print("No empty line here")
if condition:
print("No newline above me!")
print("There is a newline above me, and that's OK!")
class Point:
x: int
y: int
```
It will also insert proper spacing before and after function definitions. It's one line
@ -250,11 +293,6 @@ exception of [capital "R" prefixes](#rstrings-and-rstrings), unicode literal mar
(`u`) are removed because they are meaningless in Python 3, and in the case of multiple
characters "r" is put first as in spoken language: "raw f-string".
Another area where Python allows multiple ways to format a string is escape sequences.
For example, `"\uabcd"` and `"\uABCD"` evaluate to the same string. _Black_ normalizes
such escape sequences to lowercase, but uses uppercase for `\N` named character escapes,
such as `"\N{MEETEI MAYEK LETTER HUK}"`.
The main reason to standardize on a single form of quotes is aesthetics. Having one kind
of quotes everywhere reduces reader distraction. It will also enable a future version of
_Black_ to merge consecutive string literals that ended up on the same line (see
@ -420,12 +458,6 @@ file that are not enforced yet but might be in a future version of the formatter
_Black_ will normalize line endings (`\n` or `\r\n`) based on the first line ending of
the file.
### Form feed characters
_Black_ will retain form feed characters on otherwise empty lines at the module level.
Only one form feed is retained for a group of consecutive empty lines. Where there are
two empty lines in a row, the form feed is placed on the second line.
## Pragmatism
Early versions of _Black_ used to be absolutist in some respects. They took after its

136
docs/the_black_code_style/future_style.md
View File

@ -1,49 +1,94 @@
# The (future of the) Black code style
## Preview style
```{warning}
Changes to this document often aren't tied and don't relate to releases of
_Black_. It's recommended that you read the latest version available.
```
## Using backslashes for with statements
[Backslashes are bad and should be never be used](labels/why-no-backslashes) however
there is one exception: `with` statements using multiple context managers. Before Python
3.9 Python's grammar does not allow organizing parentheses around the series of context
managers.
We don't want formatting like:
```py3
with make_context_manager1() as cm1, make_context_manager2() as cm2, make_context_manager3() as cm3, make_context_manager4() as cm4:
... # nothing to split on - line too long
```
So _Black_ will, when we implement this, format it like this:
```py3
with \
make_context_manager1() as cm1, \
make_context_manager2() as cm2, \
make_context_manager3() as cm3, \
make_context_manager4() as cm4 \
:
... # backslashes and an ugly stranded colon
```
Although when the target version is Python 3.9 or higher, _Black_ uses parentheses
instead in `--preview` mode (see below) since they're allowed in Python 3.9 and higher.
An alternative to consider if the backslashes in the above formatting are undesirable is
to use {external:py:obj}`contextlib.ExitStack` to combine context managers in the
following way:
```python
with contextlib.ExitStack() as exit_stack:
cm1 = exit_stack.enter_context(make_context_manager1())
cm2 = exit_stack.enter_context(make_context_manager2())
cm3 = exit_stack.enter_context(make_context_manager3())
cm4 = exit_stack.enter_context(make_context_manager4())
...
```
(labels/preview-style)=
## Preview style
Experimental, potentially disruptive style changes are gathered under the `--preview`
CLI flag. At the end of each year, these changes may be adopted into the default style,
as described in [The Black Code Style](index.md). Because the functionality is
experimental, feedback and issue reports are highly encouraged!
In the past, the preview style included some features with known bugs, so that we were
unable to move these features to the stable style. Therefore, such features are now
moved to the `--unstable` style. All features in the `--preview` style are expected to
make it to next year's stable style; features in the `--unstable` style will be
stabilized only if issues with them are fixed. If bugs are discovered in a `--preview`
feature, it is demoted to the `--unstable` style. To avoid thrash when a feature is
demoted from the `--preview` to the `--unstable` style, users can use the
`--enable-unstable-feature` flag to enable specific unstable features.
### Improved string processing
(labels/preview-features)=
_Black_ will split long string literals and merge short ones. Parentheses are used where
appropriate. When split, parts of f-strings that don't need formatting are converted to
plain strings. User-made splits are respected when they do not exceed the line length
limit. Line continuation backslashes are converted into parenthesized strings.
Unnecessary parentheses are stripped. The stability and status of this feature is
tracked in [this issue](https://github.com/psf/black/issues/2188).
Currently, the following features are included in the preview style:
### Improved line breaks
- `always_one_newline_after_import`: Always force one blank line after import
statements, except when the line after the import is a comment or an import statement
- `wrap_long_dict_values_in_parens`: Add parentheses around long values in dictionaries
([see below](labels/wrap-long-dict-values))
- `fix_fmt_skip_in_one_liners`: Fix `# fmt: skip` behaviour on one-liner declarations,
such as `def foo(): return "mock" # fmt: skip`, where previously the declaration
would have been incorrectly collapsed.
For assignment expressions, _Black_ now prefers to split and wrap the right side of the
assignment instead of left side. For example:
(labels/unstable-features)=
```python
some_dict[
"with_a_long_key"
] = some_looooooooong_module.some_looooooooooooooong_function_name(
first_argument, second_argument, third_argument
)
```
The unstable style additionally includes the following features:
will be changed to:
- `string_processing`: split long string literals and related changes
([see below](labels/string-processing))
- `multiline_string_handling`: more compact formatting of expressions involving
multiline strings ([see below](labels/multiline-string-handling))
- `hug_parens_with_braces_and_square_brackets`: more compact formatting of nested
brackets ([see below](labels/hug-parens))
```python
some_dict["with_a_long_key"] = (
some_looooooooong_module.some_looooooooooooooong_function_name(
first_argument, second_argument, third_argument
)
)
```
(labels/wrap-long-dict-values)=
### Improved parentheses management in dicts
### Improved parentheses management
For dict literals with long values, they are now wrapped in parentheses. Unnecessary
parentheses are now removed. For example:
@ -68,12 +113,11 @@ my_dict = {
}
```
(labels/hug-parens)=
### Improved multiline dictionary and list indentation for sole function parameter
For better readability and less verticality, _Black_ now pairs parentheses ("(", ")")
with braces ("{", "}") and square brackets ("[", "]") on the same line. For example:
with braces ("{", "}") and square brackets ("[", "]") on the same line for single
parameter function calls. For example:
```python
foo(
@ -83,14 +127,6 @@ foo(
3,
]
)
nested_array = [
[
1,
2,
3,
]
]
```
will be changed to:
@ -101,12 +137,6 @@ foo([
2,
3,
])
nested_array = [[
1,
2,
3,
]]
```
This also applies to list and dictionary unpacking:
@ -142,20 +172,6 @@ foo(
)
```
(labels/string-processing)=
### Improved string processing
_Black_ will split long string literals and merge short ones. Parentheses are used where
appropriate. When split, parts of f-strings that don't need formatting are converted to
plain strings. f-strings will not be merged if they contain internal quotes and it would
change their quotation mark style. User-made splits are respected when they do not
exceed the line length limit. Line continuation backslashes are converted into
parenthesized strings. Unnecessary parentheses are stripped. The stability and status of
this feature istracked in [this issue](https://github.com/psf/black/issues/2188).
(labels/multiline-string-handling)=
### Improved multiline string handling
_Black_ is smarter when formatting multiline strings, especially in function arguments,

8
docs/the_black_code_style/index.md
View File

@ -42,11 +42,9 @@ _Black_:
enabled by newer Python language syntax as well as due to improvements in the
formatting logic.
- The `--preview` and `--unstable` flags are exempt from this policy. There are no
guarantees around the stability of the output with these flags passed into _Black_.
They are intended for allowing experimentation with proposed changes to the _Black_
code style. The `--preview` style at the end of a year should closely match the stable
style for the next year, but we may always make changes.
- The `--preview` flag is exempt from this policy. There are no guarantees around the
stability of the output with that flag passed into _Black_. This flag is intended for
allowing experimentation with the proposed changes to the _Black_ code style.
Documentation for both the current and future styles can be found:

6
docs/usage_and_configuration/black_as_a_server.md
View File

@ -62,12 +62,6 @@ The headers controlling how source code is formatted are:
- `X-Preview`: corresponds to the `--preview` command line flag. If present and its
value is not an empty string, experimental and potentially disruptive style changes
will be used.
- `X-Unstable`: corresponds to the `--unstable` command line flag. If present and its
value is not an empty string, experimental style changes that are known to be buggy
will be used.
- `X-Enable-Unstable-Feature`: corresponds to the `--enable-unstable-feature` flag. The
contents of the flag must be a comma-separated list of unstable features to be
enabled. Example: `X-Enable-Unstable-Feature: feature1, feature2`.
- `X-Fast-Or-Safe`: if set to `fast`, `blackd` will act as _Black_ does when passed the
`--fast` command line flag.
- `X-Python-Variant`: if set to `pyi`, `blackd` will act as _Black_ does when passed the

14
docs/usage_and_configuration/black_docker_image.md
View File

@ -8,16 +8,16 @@ _Black_ images with the following tags are available:
- release numbers, e.g. `21.5b2`, `21.6b0`, `21.7b0` etc.\
ℹ Recommended for users who want to use a particular version of _Black_.
- `latest_release` - tag created when a new version of _Black_ is released.\
ℹ Recommended for users who want to use released versions of _Black_. It maps to
[the latest release](https://github.com/psf/black/releases/latest) of _Black_.
ℹ Recommended for users who want to use released versions of _Black_. It maps to [the latest release](https://github.com/psf/black/releases/latest)
of _Black_.
- `latest_prerelease` - tag created when a new alpha (prerelease) version of _Black_ is
released.\
ℹ Recommended for users who want to preview or test alpha versions of _Black_. Note
that the most recent release may be newer than any prerelease, because no prereleases
are created before most releases.
ℹ Recommended for users who want to preview or test alpha versions of _Black_. Note that
the most recent release may be newer than any prerelease, because no prereleases are created
before most releases.
- `latest` - tag used for the newest image of _Black_.\
ℹ Recommended for users who always want to use the latest version of _Black_, even
before it is released.
ℹ Recommended for users who always want to use the latest version of _Black_, even before
it is released.
There is one more tag used for _Black_ Docker images - `latest_non_release`. It is
created for all unreleased

160
docs/usage_and_configuration/the_basics.md
View File

@ -12,8 +12,7 @@ _Black_ is a well-behaved Unix-style command-line tool:
## Usage
_Black_ will reformat entire files in place. To get started right away with sensible
defaults:
To get started right away with sensible defaults:
```sh
black {source_file_or_directory}
@ -25,17 +24,6 @@ You can run _Black_ as a package if running it as a script doesn't work:
python -m black {source_file_or_directory}
```
### Ignoring sections
Black will not reformat lines that contain `# fmt: skip` or blocks that start with
`# fmt: off` and end with `# fmt: on`. `# fmt: skip` can be mixed with other
pragmas/comments either with multiple comments (e.g. `# fmt: skip # pylint # noqa`) or
as a semicolon separated list (e.g. `# fmt: skip; pylint; noqa`). `# fmt: on/off` must
be on the same level of indentation and in the same block, meaning no unindents beyond
the initial indentation level between them. Black also recognizes
[YAPF](https://github.com/google/yapf)'s block comments to the same effect, as a
courtesy for straddling code.
### Command line options
The CLI options of _Black_ can be displayed by running `black --help`. All options are
@ -47,10 +35,6 @@ are deliberately limited and rarely added.
Note that all command-line options listed above can also be configured using a
`pyproject.toml` file (more on that below).
#### `-h`, `--help`
Show available command-line options and exit.
#### `-c`, `--code`
Format the code passed in as a string.
@ -70,23 +54,19 @@ See also [the style documentation](labels/line-length).
Python versions that should be supported by Black's output. You can run `black --help`
and look for the `--target-version` option to see the full list of supported versions.
You should include all versions that your code supports. If you support Python 3.11
through 3.13, you should write:
You should include all versions that your code supports. If you support Python 3.8
through 3.11, you should write:
```console
$ black -t py311 -t py312 -t py313
$ black -t py38 -t py39 -t py310 -t py311
```
In a [configuration file](#configuration-via-a-file), you can write:
```toml
target-version = ["py311", "py312", "py313"]
target-version = ["py38", "py39", "py310", "py311"]
```
By default, Black will infer target versions from the project metadata in
`pyproject.toml`, specifically the `[project.requires-python]` field. If this does not
yield conclusive results, Black will use per-file auto-detection.
_Black_ uses this option to decide what grammar to use to parse your code. In addition,
it may use it to decide what style to use. For example, support for a trailing comma
after `*args` in a function call was added in Python 3.5, so _Black_ will add this comma
@ -125,10 +105,6 @@ useful when piping source on standard input.
When processing Jupyter Notebooks, add the given magic to the list of known python-
magics. Useful for formatting cells with custom python magics.
#### `-x, --skip-source-first-line`
Skip the first line of the source code.
#### `-S, --skip-string-normalization`
By default, _Black_ uses double quotes for all strings and normalizes string prefixes,
@ -144,48 +120,20 @@ magic trailing comma is ignored.
#### `--preview`
Enable potentially disruptive style changes that we expect to add to Black's main
functionality in the next major release. Use this if you want a taste of what next
year's style will look like.
Read more about [our preview style](labels/preview-style).
There is no guarantee on the code style produced by this flag across releases.
#### `--unstable`
Enable all style changes in `--preview`, plus additional changes that we would like to
make eventually, but that have known issues that need to be fixed before they can move
back to the `--preview` style. Use this if you want to experiment with these changes and
help fix issues with them.
There is no guarantee on the code style produced by this flag across releases.
#### `--enable-unstable-feature`
Enable specific features from the `--unstable` style. See
[the preview style documentation](labels/unstable-features) for the list of supported
features. This flag can only be used when `--preview` is enabled. Users are encouraged
to use this flag if they use `--preview` style and a feature that affects their code is
moved from the `--preview` to the `--unstable` style, but they want to avoid the thrash
from undoing this change.
There are no guarantees on the behavior of these features, or even their existence,
across releases.
Enable potentially disruptive style changes that may be added to Black's main
functionality in the next major release. Read more about
[our preview style](labels/preview-style).
(labels/exit-code)=
#### `--check`
Don't write the files back, just return the status. _Black_ will exit with:
Passing `--check` will make _Black_ exit with:
- code 0 if nothing would change;
- code 1 if some files would be reformatted; or
- code 123 if there was an internal error
If used in combination with `--quiet` then only the exit code will be returned, unless
there was an internal error.
```console
$ black test.py --check
All done! ✨ 🍰 ✨
@ -210,8 +158,8 @@ $ echo $?
#### `--diff`
Don't write the files back, just output a diff to indicate what changes _Black_ would've
made. They are printed to stdout so capturing them is simple.
Passing `--diff` will make _Black_ print out diffs that indicate what changes _Black_
would've made. They are printed to stdout so capturing them is simple.
If you'd like colored diffs, you can enable them with `--color`.
@ -227,11 +175,7 @@ All done! ✨ 🍰 ✨
1 file would be reformatted.
```
#### `--color` / `--no-color`
Show (or do not show) colored diff. Only applies when `--diff` is given.
#### `--line-ranges`
### `--line-ranges`
When specified, _Black_ will try its best to only format these lines.
@ -248,11 +192,9 @@ Example: `black --line-ranges=1-10 --line-ranges=21-30 test.py` will format line
This option is mainly for editor integrations, such as "Format Selection".
```{note}
Due to [#4052](https://github.com/psf/black/issues/4052), `--line-ranges` might format
extra lines outside of the ranges when ther are unformatted lines with the exact
content. It also disables _Black_'s formatting stability check in `--safe` mode.
```
#### `--color` / `--no-color`
Show (or do not show) colored diff. Only applies when `--diff` is given.
#### `--fast` / `--safe`
@ -269,8 +211,8 @@ configuration file for consistent results across environments.
```console
$ black --version
black, 25.1.0 (compiled: yes)
$ black --required-version 25.1.0 -c "format = 'this'"
black, 23.11.0 (compiled: yes)
$ black --required-version 23.11.0 -c "format = 'this'"
format = "this"
$ black --required-version 31.5b2 -c "still = 'beta?!'"
Oh no! 💥 💔 💥 The required version does not match the running version!
@ -289,27 +231,29 @@ Because of our [stability policy](../the_black_code_style/index.md), this will g
stable formatting, but still allow you to take advantage of improvements that do not
affect formatting.
#### `--include`
A regular expression that matches files and directories that should be included on
recursive searches. An empty value means all files are included regardless of the name.
Use forward slashes for directories on all platforms (Windows, too). Exclusions are
calculated first, inclusions later.
#### `--exclude`
A regular expression that matches files and directories that should be excluded on
recursive searches. An empty value means no paths are excluded. Use forward slashes for
directories on all platforms (Windows, too). By default, Black also ignores all paths
listed in `.gitignore`. Changing this value will override all default exclusions.
If the regular expression contains newlines, it is treated as a
[verbose regular expression](https://docs.python.org/3/library/re.html#re.VERBOSE). This
is typically useful when setting these options in a `pyproject.toml` configuration file;
see [Configuration format](#configuration-format) for more information.
directories on all platforms (Windows, too). Exclusions are calculated first, inclusions
later.
#### `--extend-exclude`
Like `--exclude`, but adds additional files and directories on top of the default values
instead of overriding them.
Like `--exclude`, but adds additional files and directories on top of the excluded ones.
Useful if you simply want to add to the default.
#### `--force-exclude`
Like `--exclude`, but files and directories matching this regex will be excluded even
when they are passed explicitly as arguments. This is useful when invoking Black
when they are passed explicitly as arguments. This is useful when invoking _Black_
programmatically on changed files, such as in a pre-commit hook or editor plugin.
#### `--stdin-filename`
@ -317,23 +261,16 @@ programmatically on changed files, such as in a pre-commit hook or editor plugin
The name of the file when passing it through stdin. Useful to make sure Black will
respect the `--force-exclude` option on some editors that rely on using stdin.
#### `--include`
A regular expression that matches files and directories that should be included on
recursive searches. An empty value means all files are included regardless of the name.
Use forward slashes for directories on all platforms (Windows, too). Overrides all
exclusions, including from `.gitignore` and command line options.
#### `-W`, `--workers`
When _Black_ formats multiple files, it may use a process pool to speed up formatting.
This option controls the number of parallel workers. This can also be specified via the
`BLACK_NUM_WORKERS` environment variable. Defaults to the number of CPUs in the system.
`BLACK_NUM_WORKERS` environment variable.
#### `-q`, `--quiet`
Stop emitting all non-critical output. Error messages will still be emitted (which can
silenced by `2>/dev/null`).
Passing `-q` / `--quiet` will cause _Black_ to stop emitting all non-critical output.
Error messages will still be emitted (which can silenced by `2>/dev/null`).
```console
$ black src/ -q
@ -342,9 +279,9 @@ error: cannot format src/black_primer/cli.py: Cannot parse: 5:6: mport asyncio
#### `-v`, `--verbose`
Emit messages about files that were not changed or were ignored due to exclusion
patterns. If _Black_ is using a configuration file, a message detailing which one it is
using will be emitted.
Passing `-v` / `--verbose` will cause _Black_ to also emit messages about files that
were not changed or were ignored due to exclusion patterns. If _Black_ is using a
configuration file, a blue message detailing which one it is using will be emitted.
```console
$ black src/ -v
@ -366,7 +303,7 @@ You can check the version of _Black_ you have installed using the `--version` fl
```console
$ black --version
black, 25.1.0
black, 23.11.0
```
#### `--config`
@ -374,6 +311,10 @@ black, 25.1.0
Read configuration options from a configuration file. See
[below](#configuration-via-a-file) for more details on the configuration file.
#### `-h`, `--help`
Show available command-line options and exit.
### Environment variable options
_Black_ supports the following configuration via environment variables.
@ -404,7 +345,7 @@ All done! ✨ 🍰 ✨
use `--stdin-filename`. Useful to make sure _Black_ will respect the `--force-exclude`
option on some editors that rely on using stdin.
You can also pass code as a string using the `--code` option.
You can also pass code as a string using the `-c` / `--code` option.
### Writeback and reporting
@ -459,11 +400,10 @@ of tools like [Poetry](https://python-poetry.org/),
### Where _Black_ looks for the file
By default _Black_ looks for `pyproject.toml` containing a `[tool.black]` section
starting from the common base directory of all files and directories passed on the
command line. If it's not there, it looks in parent directories. It stops looking when
it finds the file, or a `.git` directory, or a `.hg` directory, or the root of the file
system, whichever comes first.
By default _Black_ looks for `pyproject.toml` starting from the common base directory of
all files and directories passed on the command line. If it's not there, it looks in
parent directories. It stops looking when it finds the file, or a `.git` directory, or a
`.hg` directory, or the root of the file system, whichever comes first.
If you're formatting standard input, _Black_ will look for configuration starting from
the current working directory.
@ -478,15 +418,15 @@ operating system, this configuration file should be stored as:
`XDG_CONFIG_HOME` environment variable is not set)
Note that these are paths to the TOML file itself (meaning that they shouldn't be named
as `pyproject.toml`), not directories where you store the configuration (i.e.,
`black`/`.black` is the file to create and add your configuration options to, in the
`~/.config/` directory). Here, `~` refers to the path to your home directory. On
Windows, this will be something like `C:\\Users\UserName`.
as `pyproject.toml`), not directories where you store the configuration. Here, `~`
refers to the path to your home directory. On Windows, this will be something like
`C:\\Users\UserName`.
You can also explicitly specify the path to a particular file that you want with
`--config`. In this situation _Black_ will not look for any other file.
If you're running with `--verbose`, you will see a message if a file was found and used.
If you're running with `--verbose`, you will see a blue message if a file was found and
used.
Please note `blackd` will not use `pyproject.toml` configuration.

13
gallery/gallery.py
View File

@ -7,16 +7,15 @@
import venv
import zipfile
from argparse import ArgumentParser, Namespace
from collections.abc import Generator
from concurrent.futures import ThreadPoolExecutor
from functools import lru_cache, partial
from pathlib import Path
from typing import NamedTuple, Optional, Union, cast
from typing import Generator, List, NamedTuple, Optional, Tuple, Union, cast
from urllib.request import urlopen, urlretrieve
PYPI_INSTANCE = "https://pypi.org/pypi"
PYPI_TOP_PACKAGES = (
"https://hugovk.github.io/top-pypi-packages/top-pypi-packages.min.json"
"https://hugovk.github.io/top-pypi-packages/top-pypi-packages-30-days.min.json"
)
INTERNAL_BLACK_REPO = f"{tempfile.gettempdir()}/__black"
@ -55,7 +54,7 @@ def get_pypi_download_url(package: str, version: Optional[str]) -> str:
return cast(str, source["url"])
def get_top_packages() -> list[str]:
def get_top_packages() -> List[str]:
with urlopen(PYPI_TOP_PACKAGES) as page:
result = json.load(page)
@ -151,7 +150,7 @@ def git_switch_branch(
subprocess.run(args, cwd=repo)
def init_repos(options: Namespace) -> tuple[Path, ...]:
def init_repos(options: Namespace) -> Tuple[Path, ...]:
options.output.mkdir(exist_ok=True)
if options.top_packages:
@ -207,7 +206,7 @@ def format_repo_with_version(
git_switch_branch(black_version.version, repo=black_repo)
git_switch_branch(current_branch, repo=repo, new=True, from_branch=from_branch)
format_cmd: list[Union[Path, str]] = [
format_cmd: List[Union[Path, str]] = [
black_runner(black_version.version, black_repo),
(black_repo / "black.py").resolve(),
".",
@ -223,7 +222,7 @@ def format_repo_with_version(
return current_branch
def format_repos(repos: tuple[Path, ...], options: Namespace) -> None:
def format_repos(repos: Tuple[Path, ...], options: Namespace) -> None:
black_versions = tuple(
BlackVersion(*version.split(":")) for version in options.versions
)

9
plugin/black.vim
View File

@ -21,7 +21,7 @@ endif
if v:version < 700 || !has('python3')
func! __BLACK_MISSING()
echo "The black.vim plugin requires vim7.0+ with Python 3.9 support."
echo "The black.vim plugin requires vim7.0+ with Python 3.6 support."
endfunc
command! Black :call __BLACK_MISSING()
command! BlackUpgrade :call __BLACK_MISSING()
@ -72,11 +72,12 @@ endif
function BlackComplete(ArgLead, CmdLine, CursorPos)
return [
\ 'target_version=py27',
\ 'target_version=py36',
\ 'target_version=py37',
\ 'target_version=py38',
\ 'target_version=py39',
\ 'target_version=py310',
\ 'target_version=py311',
\ 'target_version=py312',
\ 'target_version=py313',
\ ]
endfunction

83
pyproject.toml
View File

@ -7,34 +7,32 @@
[tool.black]
line-length = 88
target-version = ['py39']
target-version = ['py37', 'py38']
include = '\.pyi?$'
extend-exclude = '''
/(
# The following are specific to Black, you probably don't want those.
tests/data/
| profiling/
| scripts/generate_schema.py # Uses match syntax
)
tests/data
| profiling
)/
'''
# We use the unstable style for formatting Black itself. If you
# want bug-free formatting, you should keep this off. If you want
# stable formatting across releases, you should also keep `preview = true`
# (which is implied by this flag) off.
unstable = true
# We use preview style for formatting Black itself. If you
# want stable formatting across releases, you should keep
# this off.
preview = true
# Build system information and other project-specific configuration below.
# NOTE: You don't need this in your own Black configuration.
[build-system]
requires = ["hatchling>=1.20.0", "hatch-vcs", "hatch-fancy-pypi-readme"]
requires = ["hatchling>=1.8.0", "hatch-vcs", "hatch-fancy-pypi-readme"]
build-backend = "hatchling.build"
[project]
name = "black"
description = "The uncompromising code formatter."
license = "MIT"
requires-python = ">=3.9"
license = { text = "MIT" }
requires-python = ">=3.8"
authors = [
{ name = "Łukasz Langa", email = "lukasz@langa.pl" },
]
@ -55,11 +53,11 @@ classifiers = [
"Operating System :: OS Independent",
"Programming Language :: Python",
"Programming Language :: Python :: 3 :: Only",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Topic :: Software Development :: Libraries :: Python Modules",
"Topic :: Software Development :: Quality Assurance",
]
@ -69,7 +67,6 @@ dependencies = [
"packaging>=22.0",
"pathspec>=0.9.0",
"platformdirs>=2",
"pytokens>=0.1.10",
"tomli>=1.1.0; python_version < '3.11'",
"typing_extensions>=4.0.1; python_version < '3.11'",
]
@ -78,7 +75,9 @@ dynamic = ["readme", "version"]
[project.optional-dependencies]
colorama = ["colorama>=0.4.3"]
uvloop = ["uvloop>=0.15.2"]
d = ["aiohttp>=3.10"]
d = [
"aiohttp>=3.7.4",
]
jupyter = [
"ipython>=7.8.0",
"tokenize-rt>=3.2.0",
@ -88,14 +87,9 @@ jupyter = [
black = "black:patched_main"
blackd = "blackd:patched_main [d]"
[project.entry-points."validate_pyproject.tool_schema"]
black = "black.schema:get_schema"
[project.urls]
Documentation = "https://black.readthedocs.io/"
Changelog = "https://github.com/psf/black/blob/main/CHANGES.md"
Repository = "https://github.com/psf/black"
Issues = "https://github.com/psf/black/issues"
Homepage = "https://github.com/psf/black"
[tool.hatch.metadata.hooks.fancy-pypi-readme]
content-type = "text/markdown"
@ -126,8 +120,8 @@ macos-max-compat = true
enable-by-default = false
dependencies = [
"hatch-mypyc>=0.16.0",
"mypy>=1.12",
"click>=8.1.7",
"mypy==1.5.1",
"click==8.1.3", # avoid https://github.com/pallets/click/issues/2558
]
require-runtime-dependencies = true
exclude = [
@ -150,14 +144,12 @@ options = { debug_level = "0" }
[tool.cibuildwheel]
build-verbosity = 1
# So these are the environments we target:
# - Python: CPython 3.9+ only
# - Python: CPython 3.8+ only
# - Architecture (64-bit only): amd64 / x86_64, universal2, and arm64
# - OS: Linux (no musl), Windows, and macOS
build = "cp3*"
skip = ["*-manylinux_i686", "*-musllinux_*", "*-win32", "pp*"]
skip = ["*-manylinux_i686", "*-musllinux_*", "*-win32", "pp*", "cp312-*"]
# This is the bare minimum needed to run the test suite. Pulling in the full
# test_requirements.txt would download a bunch of other packages not necessary
# here and would slow down the testing step a fair bit.
@ -172,9 +164,11 @@ test-skip = ["*-macosx_arm64", "*-macosx_universal2:arm64"]
HATCH_BUILD_HOOKS_ENABLE = "1"
MYPYC_OPT_LEVEL = "3"
MYPYC_DEBUG_LEVEL = "0"
# CPython 3.11 wheels aren't available for aiohttp and building a Cython extension
# from source also doesn't work.
AIOHTTP_NO_EXTENSIONS = "1"
[tool.cibuildwheel.linux]
manylinux-x86_64-image = "manylinux_2_28"
before-build = [
"yum install -y clang gcc",
]
@ -183,10 +177,19 @@ before-build = [
HATCH_BUILD_HOOKS_ENABLE = "1"
MYPYC_OPT_LEVEL = "3"
MYPYC_DEBUG_LEVEL = "0"
AIOHTTP_NO_EXTENSIONS = "1"
# Black needs Clang to compile successfully on Linux.
CC = "clang"
[tool.cibuildwheel.macos]
build-frontend = { name = "build", args = ["--no-isolation"] }
# Unfortunately, hatch doesn't respect MACOSX_DEPLOYMENT_TARGET
before-build = [
"python -m pip install 'hatchling==1.18.0' hatch-vcs hatch-fancy-pypi-readme 'hatch-mypyc>=0.16.0' 'mypy==1.5.1' 'click==8.1.3'",
"""sed -i '' -e "600,700s/'10_16'/os.environ['MACOSX_DEPLOYMENT_TARGET'].replace('.', '_')/" $(python -c 'import hatchling.builders.wheel as h; print(h.__file__)') """,
]
[tool.isort]
atomic = true
profile = "black"
@ -206,7 +209,23 @@ markers = [
"incompatible_with_mypyc: run when testing mypyc compiled black"
]
xfail_strict = true
filterwarnings = ["error"]
filterwarnings = [
"error",
# this is mitigated by a try/catch in https://github.com/psf/black/pull/2974/
# this ignore can be removed when support for aiohttp 3.7 is dropped.
'''ignore:Decorator `@unittest_run_loop` is no longer needed in aiohttp 3\.8\+:DeprecationWarning''',
# this is mitigated by a try/catch in https://github.com/psf/black/pull/3198/
# this ignore can be removed when support for aiohttp 3.x is dropped.
'''ignore:Middleware decorator is deprecated since 4\.0 and its behaviour is default, you can simply remove this decorator:DeprecationWarning''',
# aiohttp is using deprecated cgi modules - Safe to remove when fixed:
# https://github.com/aio-libs/aiohttp/issues/6905
'''ignore:'cgi' is deprecated and slated for removal in Python 3.13:DeprecationWarning''',
# Work around https://github.com/pytest-dev/pytest/issues/10977 for Python 3.12
'''ignore:(Attribute s|Attribute n|ast.Str|ast.Bytes|ast.NameConstant|ast.Num) is deprecated and will be removed in Python 3.14:DeprecationWarning''',
# Will be fixed with aiohttp 3.9.0
# https://github.com/aio-libs/aiohttp/pull/7302
"ignore:datetime.*utcfromtimestamp\\(\\) is deprecated and scheduled for removal:DeprecationWarning",
]
[tool.coverage.report]
omit = [
"src/blib2to3/*",
@ -222,11 +241,9 @@ branch = true
# Specify the target platform details in config, so your developers are
# free to run mypy on Windows, Linux, or macOS and get consistent
# results.
python_version = "3.9"
python_version = "3.8"
mypy_path = "src"
strict = true
strict_bytes = true
local_partial_types = true
# Unreachable blocks have been an issue when compiling mypyc, let's try to avoid 'em in the first place.
warn_unreachable = true
implicit_reexport = true

2
scripts/check_pre_commit_rev_in_example.py
View File

@ -14,7 +14,7 @@
import commonmark
import yaml
from bs4 import BeautifulSoup # type: ignore[import-untyped]
from bs4 import BeautifulSoup # type: ignore[import]
def main(changes: str, source_version_control: str) -> None:

2
scripts/check_version_in_basics_example.py
View File

@ -8,7 +8,7 @@
import sys
import commonmark
from bs4 import BeautifulSoup # type: ignore[import-untyped]
from bs4 import BeautifulSoup # type: ignore[import]
def main(changes: str, the_basics: str) -> None:

9
scripts/diff_shades_gha_helper.py
View File

@ -24,12 +24,17 @@
from base64 import b64encode
from io import BytesIO
from pathlib import Path
from typing import Any, Final, Literal
from typing import Any
import click
import urllib3
from packaging.version import Version
if sys.version_info >= (3, 8):
from typing import Final, Literal
else:
from typing_extensions import Final, Literal
COMMENT_FILE: Final = ".pr-comment.json"
DIFF_STEP_NAME: Final = "Generate HTML diff report"
DOCS_URL: Final = (
@ -114,7 +119,7 @@ def main() -> None:
@main.command("config", help="Acquire run configuration and metadata.")
@click.argument("event", type=click.Choice(["push", "pull_request"]))
def config(event: Literal["push", "pull_request"]) -> None:
import diff_shades # type: ignore[import-not-found]
import diff_shades # type: ignore[import]
if event == "push":
jobs = [{"mode": "preview-changes", "force-flag": "--force-preview-style"}]

23
scripts/fuzz.py
View File

@ -5,11 +5,14 @@
a coverage-guided fuzzer I'm working on.
"""
import re
import hypothesmith
from hypothesis import HealthCheck, given, settings
from hypothesis import strategies as st
import black
from blib2to3.pgen2.tokenize import TokenError
# This test uses the Hypothesis and Hypothesmith libraries to generate random
@ -42,7 +45,23 @@ def test_idempotent_any_syntatically_valid_python(
compile(src_contents, "<string>", "exec") # else the bug is in hypothesmith
# Then format the code...
dst_contents = black.format_str(src_contents, mode=mode)
try:
dst_contents = black.format_str(src_contents, mode=mode)
except black.InvalidInput:
# This is a bug - if it's valid Python code, as above, Black should be
# able to cope with it. See issues #970, #1012
# TODO: remove this try-except block when issues are resolved.
return
except TokenError as e:
if ( # Special-case logic for backslashes followed by newlines or end-of-input
e.args[0] == "EOF in multi-line statement"
and re.search(r"\\($|\r?\n)", src_contents) is not None
):
# This is a bug - if it's valid Python code, as above, Black should be
# able to cope with it. See issue #1012.
# TODO: remove this block when the issue is resolved.
return
raise
# And check that we got equivalent and stable output.
black.assert_equivalent(src_contents, dst_contents)
@ -61,7 +80,7 @@ def test_idempotent_any_syntatically_valid_python(
try:
import sys
import atheris
import atheris # type: ignore[import]
except ImportError:
pass
else:

74
scripts/generate_schema.py
View File

@ -1,74 +0,0 @@
import json
from typing import IO, Any
import click
import black
def generate_schema_from_click(
cmd: click.Command,
) -> dict[str, Any]:
result: dict[str, dict[str, Any]] = {}
for param in cmd.params:
if not isinstance(param, click.Option) or param.is_eager:
continue
assert param.name
name = param.name.replace("_", "-")
result[name] = {}
match param.type:
case click.types.IntParamType():
result[name]["type"] = "integer"
case click.types.StringParamType() | click.types.Path():
result[name]["type"] = "string"
case click.types.Choice(choices=choices):
result[name]["enum"] = choices
case click.types.BoolParamType():
result[name]["type"] = "boolean"
case _:
msg = f"{param.type!r} not a known type for {param}"
raise TypeError(msg)
if param.multiple:
result[name] = {"type": "array", "items": result[name]}
result[name]["description"] = param.help
if param.default is not None and not param.multiple:
result[name]["default"] = param.default
return result
@click.command(context_settings={"help_option_names": ["-h", "--help"]})
@click.option("--schemastore", is_flag=True, help="SchemaStore format")
@click.option("--outfile", type=click.File(mode="w"), help="Write to file")
def main(schemastore: bool, outfile: IO[str]) -> None:
properties = generate_schema_from_click(black.main)
del properties["line-ranges"]
schema: dict[str, Any] = {
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": (
"https://github.com/psf/black/blob/main/src/black/resources/black.schema.json"
),
"$comment": "tool.black table in pyproject.toml",
"type": "object",
"additionalProperties": False,
"properties": properties,
}
if schemastore:
schema["$id"] = "https://json.schemastore.org/partial-black.json"
# The precise list of unstable features may change frequently, so don't
# bother putting it in SchemaStore
schema["properties"]["enable-unstable-feature"]["items"] = {"type": "string"}
print(json.dumps(schema, indent=2), file=outfile)
if __name__ == "__main__":
main()

10
scripts/make_width_table.py
View File

@ -17,13 +17,13 @@
"""
import sys
from collections.abc import Iterable
from os.path import basename, dirname, join
from typing import Iterable, Tuple
import wcwidth # type: ignore[import-not-found]
import wcwidth # type: ignore[import]
def make_width_table() -> Iterable[tuple[int, int, int]]:
def make_width_table() -> Iterable[Tuple[int, int, int]]:
start_codepoint = -1
end_codepoint = -1
range_width = -2
@ -53,9 +53,9 @@ def main() -> None:
f.write(f"""# Generated by {basename(__file__)}
# wcwidth {wcwidth.__version__}
# Unicode {wcwidth.list_versions()[-1]}
from typing import Final
from typing import Final, List, Tuple
WIDTH_TABLE: Final[list[tuple[int, int, int]]] = [
WIDTH_TABLE: Final[List[Tuple[int, int, int]]] = [
""")
for triple in make_width_table():
f.write(f" {triple!r},\n")

14
scripts/migrate-black.py
View File

@ -9,7 +9,7 @@
def git(*args: str) -> str:
return check_output(["git", *args]).decode("utf8").strip()
return check_output(["git"] + list(args)).decode("utf8").strip()
def blackify(base_branch: str, black_command: str, logger: logging.Logger) -> int:
@ -26,19 +26,19 @@ def blackify(base_branch: str, black_command: str, logger: logging.Logger) -> in
merge_base = git("merge-base", "HEAD", base_branch)
if not merge_base:
logger.error(
f"Could not find a common commit for current head and {base_branch}"
"Could not find a common commit for current head and %s" % base_branch
)
return 1
commits = git(
"log", "--reverse", "--pretty=format:%H", f"{merge_base}~1..HEAD"
"log", "--reverse", "--pretty=format:%H", "%s~1..HEAD" % merge_base
).split()
for commit in commits:
git("checkout", commit, f"-b{commit}-black")
git("checkout", commit, "-b%s-black" % commit)
check_output(black_command, shell=True)
git("commit", "-aqm", "blackify")
git("checkout", base_branch, f"-b{current_branch}-black")
git("checkout", base_branch, "-b%s-black" % current_branch)
for last_commit, commit in zip(commits, commits[1:]):
allow_empty = (
@ -51,7 +51,7 @@ def blackify(base_branch: str, black_command: str, logger: logging.Logger) -> in
"diff",
"--binary",
"--find-copies",
f"{last_commit}-black..{commit}-black",
"%s-black..%s-black" % (last_commit, commit),
],
stdout=PIPE,
)
@ -77,7 +77,7 @@ def blackify(base_branch: str, black_command: str, logger: logging.Logger) -> in
git("commit", "--allow-empty", "-aqC", commit)
for commit in commits:
git("branch", "-qD", f"{commit}-black")
git("branch", "-qD", "%s-black" % commit)
return 0

11
scripts/release.py
View File

@ -11,7 +11,8 @@
import sys
from datetime import datetime
from pathlib import Path
from subprocess import run
from subprocess import PIPE, run
from typing import List
LOG = logging.getLogger(__name__)
NEW_VERSION_CHANGELOG_TEMPLATE = """\
@ -69,9 +70,9 @@ class NoGitTagsError(Exception): ... # noqa: E701,E761
# TODO: Do better with alpha + beta releases
# Maybe we vendor packaging library
def get_git_tags(versions_only: bool = True) -> list[str]:
def get_git_tags(versions_only: bool = True) -> List[str]:
"""Pull out all tags or calvers only"""
cp = run(["git", "tag"], capture_output=True, check=True, encoding="utf8")
cp = run(["git", "tag"], stdout=PIPE, stderr=PIPE, check=True, encoding="utf8")
if not cp.stdout:
LOG.error(f"Returned no git tags stdout: {cp.stderr}")
raise NoGitTagsError
@ -168,9 +169,7 @@ def get_next_version(self) -> str:
calver_parts = base_calver.split(".")
base_calver = f"{calver_parts[0]}.{int(calver_parts[1])}" # Remove leading 0
git_tags = get_git_tags()
same_month_releases = [
t for t in git_tags if t.startswith(base_calver) and "a" not in t
]
same_month_releases = [t for t in git_tags if t.startswith(base_calver)]
if len(same_month_releases) < 1:
return f"{base_calver}.0"
same_month_version = same_month_releases[-1].split(".", 2)[-1]

476
src/black/__init__.py
View File

@ -5,22 +5,28 @@
import sys
import tokenize
import traceback
from collections.abc import (
Collection,
Generator,
Iterator,
MutableMapping,
Sequence,
Sized,
)
from contextlib import contextmanager
from dataclasses import replace
from datetime import datetime, timezone
from enum import Enum
from json.decoder import JSONDecodeError
from pathlib import Path
from re import Pattern
from typing import Any, Optional, Union
from typing import (
Any,
Collection,
Dict,
Generator,
Iterator,
List,
MutableMapping,
Optional,
Pattern,
Sequence,
Set,
Sized,
Tuple,
Union,
)
import click
from click.core import ParameterSource
@ -38,47 +44,43 @@
STDIN_PLACEHOLDER,
)
from black.files import (
best_effort_relative_path,
find_project_root,
find_pyproject_toml,
find_user_pyproject_toml,
gen_python_files,
get_gitignore,
normalize_path_maybe_ignore,
parse_pyproject_toml,
path_is_excluded,
resolves_outside_root_or_cannot_stat,
wrap_stream_for_windows,
)
from black.handle_ipynb_magics import (
PYTHON_CELL_MAGICS,
TRANSFORMED_MAGICS,
jupyter_dependencies_are_installed,
mask_cell,
put_trailing_semicolon_back,
remove_trailing_semicolon,
unmask_cell,
validate_cell,
)
from black.linegen import LN, LineGenerator, transform_line
from black.lines import EmptyLineTracker, LinesBlock
from black.mode import FUTURE_FLAG_TO_FEATURE, VERSION_TO_FEATURES, Feature
from black.mode import Mode as Mode # re-exported
from black.mode import Preview, TargetVersion, supports_feature
from black.nodes import STARS, is_number_token, is_simple_decorator_expression, syms
from black.mode import TargetVersion, supports_feature
from black.nodes import (
STARS,
is_number_token,
is_simple_decorator_expression,
is_string_token,
syms,
)
from black.output import color_diff, diff, dump_to_file, err, ipynb_diff, out
from black.parsing import ( # noqa F401
ASTSafetyError,
InvalidInput,
lib2to3_parse,
parse_ast,
stringify_ast,
)
from black.ranges import (
adjusted_lines,
convert_unchanged_lines,
parse_line_ranges,
sanitized_lines,
)
from black.parsing import InvalidInput # noqa F401
from black.parsing import lib2to3_parse, parse_ast, stringify_ast
from black.ranges import adjusted_lines, convert_unchanged_lines, parse_line_ranges
from black.report import Changed, NothingChanged, Report
from black.trans import iter_fexpr_spans
from blib2to3.pgen2 import token
from blib2to3.pytree import Leaf, Node
@ -139,7 +141,6 @@ def read_pyproject_toml(
if not config:
return None
else:
spellcheck_pyproject_toml_keys(ctx, list(config), value)
# Sanitize the values to be Click friendly. For more information please see:
# https://github.com/psf/black/issues/1458
# https://github.com/pallets/click/issues/1567
@ -170,7 +171,7 @@ def read_pyproject_toml(
"line-ranges", "Cannot use line-ranges in the pyproject.toml file."
)
default_map: dict[str, Any] = {}
default_map: Dict[str, Any] = {}
if ctx.default_map:
default_map.update(ctx.default_map)
default_map.update(config)
@ -179,25 +180,9 @@ def read_pyproject_toml(
return value
def spellcheck_pyproject_toml_keys(
ctx: click.Context, config_keys: list[str], config_file_path: str
) -> None:
invalid_keys: list[str] = []
available_config_options = {param.name for param in ctx.command.params}
for key in config_keys:
if key not in available_config_options:
invalid_keys.append(key)
if invalid_keys:
keys_str = ", ".join(map(repr, invalid_keys))
out(
f"Invalid config keys detected: {keys_str} (in {config_file_path})",
fg="red",
)
def target_version_option_callback(
c: click.Context, p: Union[click.Option, click.Parameter], v: tuple[str, ...]
) -> list[TargetVersion]:
c: click.Context, p: Union[click.Option, click.Parameter], v: Tuple[str, ...]
) -> List[TargetVersion]:
"""Compute the target versions from a --target-version flag.
This is its own function because mypy couldn't infer the type correctly
@ -206,13 +191,6 @@ def target_version_option_callback(
return [TargetVersion[val.upper()] for val in v]
def enable_unstable_feature_callback(
c: click.Context, p: Union[click.Option, click.Parameter], v: tuple[str, ...]
) -> list[Preview]:
"""Compute the features from an --enable-unstable-feature flag."""
return [Preview[val] for val in v]
def re_compile_maybe_verbose(regex: str) -> Pattern[str]:
"""Compile a regular expression string in `regex`.
@ -257,26 +235,25 @@ def validate_regex(
callback=target_version_option_callback,
multiple=True,
help=(
"Python versions that should be supported by Black's output. You should"
" include all versions that your code supports. By default, Black will infer"
" target versions from the project metadata in pyproject.toml. If this does"
" not yield conclusive results, Black will use per-file auto-detection."
"Python versions that should be supported by Black's output. By default, Black"
" will try to infer this from the project metadata in pyproject.toml. If this"
" does not yield conclusive results, Black will use per-file auto-detection."
),
)
@click.option(
"--pyi",
is_flag=True,
help=(
"Format all input files like typing stubs regardless of file extension. This"
" is useful when piping source on standard input."
"Format all input files like typing stubs regardless of file extension (useful"
" when piping source on standard input)."
),
)
@click.option(
"--ipynb",
is_flag=True,
help=(
"Format all input files like Jupyter Notebooks regardless of file extension."
" This is useful when piping source on standard input."
"Format all input files like Jupyter Notebooks regardless of file extension "
"(useful when piping source on standard input)."
),
)
@click.option(
@ -307,6 +284,12 @@ def validate_regex(
is_flag=True,
help="Don't use trailing commas as a reason to split lines.",
)
@click.option(
"--experimental-string-processing",
is_flag=True,
hidden=True,
help="(DEPRECATED and now included in --preview) Normalize string literals.",
)
@click.option(
"--preview",
is_flag=True,
@ -315,26 +298,6 @@ def validate_regex(
" functionality in the next major release."
),
)
@click.option(
"--unstable",
is_flag=True,
help=(
"Enable potentially disruptive style changes that have known bugs or are not"
" currently expected to make it into the stable style Black's next major"
" release. Implies --preview."
),
)
@click.option(
"--enable-unstable-feature",
type=click.Choice([v.name for v in Preview]),
callback=enable_unstable_feature_callback,
multiple=True,
help=(
"Enable specific features included in the `--unstable` style. Requires"
" `--preview`. No compatibility guarantees are provided on the behavior"
" or existence of any unstable features."
),
)
@click.option(
"--check",
is_flag=True,
@ -347,22 +310,14 @@ def validate_regex(
@click.option(
"--diff",
is_flag=True,
help=(
"Don't write the files back, just output a diff to indicate what changes"
" Black would've made. They are printed to stdout so capturing them is simple."
),
)
@click.option(
"--color/--no-color",
is_flag=True,
help="Show (or do not show) colored diff. Only applies when --diff is given.",
help="Don't write the files back, just output a diff for each file on stdout.",
)
@click.option(
"--line-ranges",
multiple=True,
metavar="START-END",
help=(
"When specified, Black will try its best to only format these lines. This"
"When specified, _Black_ will try its best to only format these lines. This"
" option can be specified multiple times, and a union of the lines will be"
" formatted. Each range must be specified as two integers connected by a `-`:"
" `<START>-<END>`. The `<START>` and `<END>` integer indices are 1-based and"
@ -370,67 +325,23 @@ def validate_regex(
),
default=(),
)
@click.option(
"--color/--no-color",
is_flag=True,
help="Show colored diff. Only applies when `--diff` is given.",
)
@click.option(
"--fast/--safe",
is_flag=True,
help=(
"By default, Black performs an AST safety check after formatting your code."
" The --fast flag turns off this check and the --safe flag explicitly enables"
" it. [default: --safe]"
),
help="If --fast given, skip temporary sanity checks. [default: --safe]",
)
@click.option(
"--required-version",
type=str,
help=(
"Require a specific version of Black to be running. This is useful for"
" ensuring that all contributors to your project are using the same"
" version, because different versions of Black may format code a little"
" differently. This option can be set in a configuration file for consistent"
" results across environments."
),
)
@click.option(
"--exclude",
type=str,
callback=validate_regex,
help=(
"A regular expression that matches files and directories that should be"
" excluded on recursive searches. An empty value means no paths are excluded."
" Use forward slashes for directories on all platforms (Windows, too)."
" By default, Black also ignores all paths listed in .gitignore. Changing this"
f" value will override all default exclusions. [default: {DEFAULT_EXCLUDES}]"
),
show_default=False,
)
@click.option(
"--extend-exclude",
type=str,
callback=validate_regex,
help=(
"Like --exclude, but adds additional files and directories on top of the"
" default values instead of overriding them."
),
)
@click.option(
"--force-exclude",
type=str,
callback=validate_regex,
help=(
"Like --exclude, but files and directories matching this regex will be excluded"
" even when they are passed explicitly as arguments. This is useful when"
" invoking Black programmatically on changed files, such as in a pre-commit"
" hook or editor plugin."
),
)
@click.option(
"--stdin-filename",
type=str,
is_eager=True,
help=(
"The name of the file when passing it through stdin. Useful to make sure Black"
" will respect the --force-exclude option on some editors that rely on using"
" stdin."
"Require a specific version of Black to be running (useful for unifying results"
" across many environments e.g. with a pyproject.toml file). It can be"
" either a major version number or an exact version."
),
)
@click.option(
@ -442,21 +353,59 @@ def validate_regex(
"A regular expression that matches files and directories that should be"
" included on recursive searches. An empty value means all files are included"
" regardless of the name. Use forward slashes for directories on all platforms"
" (Windows, too). Overrides all exclusions, including from .gitignore and"
" command line options."
" (Windows, too). Exclusions are calculated first, inclusions later."
),
show_default=True,
)
@click.option(
"--exclude",
type=str,
callback=validate_regex,
help=(
"A regular expression that matches files and directories that should be"
" excluded on recursive searches. An empty value means no paths are excluded."
" Use forward slashes for directories on all platforms (Windows, too)."
" Exclusions are calculated first, inclusions later. [default:"
f" {DEFAULT_EXCLUDES}]"
),
show_default=False,
)
@click.option(
"--extend-exclude",
type=str,
callback=validate_regex,
help=(
"Like --exclude, but adds additional files and directories on top of the"
" excluded ones. (Useful if you simply want to add to the default)"
),
)
@click.option(
"--force-exclude",
type=str,
callback=validate_regex,
help=(
"Like --exclude, but files and directories matching this regex will be "
"excluded even when they are passed explicitly as arguments."
),
)
@click.option(
"--stdin-filename",
type=str,
is_eager=True,
help=(
"The name of the file when passing it through stdin. Useful to make "
"sure Black will respect --force-exclude option on some "
"editors that rely on using stdin."
),
)
@click.option(
"-W",
"--workers",
type=click.IntRange(min=1),
default=None,
help=(
"When Black formats multiple files, it may use a process pool to speed up"
" formatting. This option controls the number of parallel workers. This can"
" also be specified via the BLACK_NUM_WORKERS environment variable. Defaults"
" to the number of CPUs in the system."
"Number of parallel workers [default: BLACK_NUM_WORKERS environment variable "
"or number of CPUs in the system]"
),
)
@click.option(
@ -464,8 +413,8 @@ def validate_regex(
"--quiet",
is_flag=True,
help=(
"Stop emitting all non-critical output. Error messages will still be emitted"
" (which can silenced by 2>/dev/null)."
"Don't emit non-error messages to stderr. Errors are still emitted; silence"
" those with 2>/dev/null."
),
)
@click.option(
@ -473,9 +422,8 @@ def validate_regex(
"--verbose",
is_flag=True,
help=(
"Emit messages about files that were not changed or were ignored due to"
" exclusion patterns. If Black is using a configuration file, a message"
" detailing which one it is using will be emitted."
"Also emit messages to stderr about files that were not changed or were ignored"
" due to exclusion patterns."
),
)
@click.version_option(
@ -506,14 +454,14 @@ def validate_regex(
),
is_eager=True,
callback=read_pyproject_toml,
help="Read configuration options from a configuration file.",
help="Read configuration from FILE path.",
)
@click.pass_context
def main( # noqa: C901
ctx: click.Context,
code: Optional[str],
line_length: int,
target_version: list[TargetVersion],
target_version: List[TargetVersion],
check: bool,
diff: bool,
line_ranges: Sequence[str],
@ -525,9 +473,8 @@ def main( # noqa: C901
skip_source_first_line: bool,
skip_string_normalization: bool,
skip_magic_trailing_comma: bool,
experimental_string_processing: bool,
preview: bool,
unstable: bool,
enable_unstable_feature: list[Preview],
quiet: bool,
verbose: bool,
required_version: Optional[str],
@ -537,21 +484,12 @@ def main( # noqa: C901
force_exclude: Optional[Pattern[str]],
stdin_filename: Optional[str],
workers: Optional[int],
src: tuple[str, ...],
src: Tuple[str, ...],
config: Optional[str],
) -> None:
"""The uncompromising code formatter."""
ctx.ensure_object(dict)
assert sys.version_info >= (3, 9), "Black requires Python 3.9+"
if sys.version_info[:3] == (3, 12, 5):
out(
"Python 3.12.5 has a memory safety issue that can cause Black's "
"AST safety checks to fail. "
"Please upgrade to Python 3.12.6 or downgrade to Python 3.12.4"
)
ctx.exit(1)
if src and code is not None:
out(
main.get_usage(ctx)
@ -562,14 +500,6 @@ def main( # noqa: C901
out(main.get_usage(ctx) + "\n\nOne of 'SRC' or 'code' is required.")
ctx.exit(1)
# It doesn't do anything if --unstable is also passed, so just allow it.
if enable_unstable_feature and not (preview or unstable):
out(
main.get_usage(ctx)
+ "\n\n'--enable-unstable-feature' requires '--preview'."
)
ctx.exit(1)
root, method = (
find_project_root(src, stdin_filename) if code is None else (None, None)
)
@ -631,13 +561,12 @@ def main( # noqa: C901
skip_source_first_line=skip_source_first_line,
string_normalization=not skip_string_normalization,
magic_trailing_comma=not skip_magic_trailing_comma,
experimental_string_processing=experimental_string_processing,
preview=preview,
unstable=unstable,
python_cell_magics=set(python_cell_magics),
enabled_features=set(enable_unstable_feature),
)
lines: list[tuple[int, int]] = []
lines: List[Tuple[int, int]] = []
if line_ranges:
if ipynb:
err("Cannot use --line-ranges with ipynb files.")
@ -727,7 +656,7 @@ def main( # noqa: C901
def get_sources(
*,
root: Path,
src: tuple[str, ...],
src: Tuple[str, ...],
quiet: bool,
verbose: bool,
include: Pattern[str],
@ -736,25 +665,18 @@ def get_sources(
force_exclude: Optional[Pattern[str]],
report: "Report",
stdin_filename: Optional[str],
) -> set[Path]:
) -> Set[Path]:
"""Compute the set of files to be formatted."""
sources: set[Path] = set()
sources: Set[Path] = set()
assert root.is_absolute(), f"INTERNAL ERROR: `root` must be absolute but is {root}"
using_default_exclude = exclude is None
exclude = re_compile_maybe_verbose(DEFAULT_EXCLUDES) if exclude is None else exclude
gitignore: Optional[dict[Path, PathSpec]] = None
gitignore: Optional[Dict[Path, PathSpec]] = None
root_gitignore = get_gitignore(root)
for s in src:
if s == "-" and stdin_filename:
path = Path(stdin_filename)
if path_is_excluded(stdin_filename, force_exclude):
report.path_ignored(
path,
"--stdin-filename matches the --force-exclude regular expression",
)
continue
is_stdin = True
else:
path = Path(s)
@ -762,12 +684,8 @@ def get_sources(
# Compare the logic here to the logic in `gen_python_files`.
if is_stdin or path.is_file():
if resolves_outside_root_or_cannot_stat(path, root, report):
if verbose:
out(f'Skipping invalid source: "{path}"', fg="red")
continue
root_relative_path = path.absolute().relative_to(root).as_posix()
root_relative_path = best_effort_relative_path(path, root).as_posix()
root_relative_path = "/" + root_relative_path
# Hard-exclude any files that matches the `--force-exclude` regex.
@ -777,6 +695,14 @@ def get_sources(
)
continue
normalized_path: Optional[str] = normalize_path_maybe_ignore(
path, root, report
)
if normalized_path is None:
if verbose:
out(f'Skipping invalid source: "{normalized_path}"', fg="red")
continue
if is_stdin:
path = Path(f"{STDIN_PLACEHOLDER}{str(path)}")
@ -786,7 +712,7 @@ def get_sources(
continue
if verbose:
out(f'Found input source: "{path}"', fg="blue")
out(f'Found input source: "{normalized_path}"', fg="blue")
sources.add(path)
elif path.is_dir():
path = root / (path.resolve().relative_to(root))
@ -841,7 +767,7 @@ def reformat_code(
mode: Mode,
report: Report,
*,
lines: Collection[tuple[int, int]] = (),
lines: Collection[Tuple[int, int]] = (),
) -> None:
"""
Reformat and print out `content` without spawning child processes.
@ -874,7 +800,7 @@ def reformat_one(
mode: Mode,
report: "Report",
*,
lines: Collection[tuple[int, int]] = (),
lines: Collection[Tuple[int, int]] = (),
) -> None:
"""Reformat a single file under `src` without spawning child processes.
@ -930,7 +856,7 @@ def format_file_in_place(
write_back: WriteBack = WriteBack.NO,
lock: Any = None, # multiprocessing.Manager().Lock() is some crazy proxy
*,
lines: Collection[tuple[int, int]] = (),
lines: Collection[Tuple[int, int]] = (),
) -> bool:
"""Format file under `src` path. Return True if changed.
@ -997,7 +923,7 @@ def format_stdin_to_stdout(
content: Optional[str] = None,
write_back: WriteBack = WriteBack.NO,
mode: Mode,
lines: Collection[tuple[int, int]] = (),
lines: Collection[Tuple[int, int]] = (),
) -> bool:
"""Format file on stdin. Return True if changed.
@ -1048,7 +974,7 @@ def check_stability_and_equivalence(
dst_contents: str,
*,
mode: Mode,
lines: Collection[tuple[int, int]] = (),
lines: Collection[Tuple[int, int]] = (),
) -> None:
"""Perform stability and equivalence checks.
@ -1065,7 +991,7 @@ def format_file_contents(
*,
fast: bool,
mode: Mode,
lines: Collection[tuple[int, int]] = (),
lines: Collection[Tuple[int, int]] = (),
) -> FileContent:
"""Reformat contents of a file and return new contents.
@ -1088,6 +1014,32 @@ def format_file_contents(
return dst_contents
def validate_cell(src: str, mode: Mode) -> None:
"""Check that cell does not already contain TransformerManager transformations,
or non-Python cell magics, which might cause tokenizer_rt to break because of
indentations.
If a cell contains ``!ls``, then it'll be transformed to
``get_ipython().system('ls')``. However, if the cell originally contained
``get_ipython().system('ls')``, then it would get transformed in the same way:
>>> TransformerManager().transform_cell("get_ipython().system('ls')")
"get_ipython().system('ls')\n"
>>> TransformerManager().transform_cell("!ls")
"get_ipython().system('ls')\n"
Due to the impossibility of safely roundtripping in such situations, cells
containing transformed magics will be ignored.
"""
if any(transformed_magic in src for transformed_magic in TRANSFORMED_MAGICS):
raise NothingChanged
if (
src[:2] == "%%"
and src.split()[0][2:] not in PYTHON_CELL_MAGICS | mode.python_cell_magics
):
raise NothingChanged
def format_cell(src: str, *, fast: bool, mode: Mode) -> str:
"""Format code in given cell of Jupyter notebook.
@ -1170,7 +1122,7 @@ def format_ipynb_string(src_contents: str, *, fast: bool, mode: Mode) -> FileCon
def format_str(
src_contents: str, *, mode: Mode, lines: Collection[tuple[int, int]] = ()
src_contents: str, *, mode: Mode, lines: Collection[Tuple[int, int]] = ()
) -> str:
"""Reformat a string and return new contents.
@ -1201,10 +1153,6 @@ def f(
hey
"""
if lines:
lines = sanitized_lines(lines, src_contents)
if not lines:
return src_contents # Nothing to format
dst_contents = _format_str_once(src_contents, mode=mode, lines=lines)
# Forced second pass to work around optional trailing commas (becoming
# forced trailing commas on pass 2) interacting differently with optional
@ -1217,10 +1165,10 @@ def f(
def _format_str_once(
src_contents: str, *, mode: Mode, lines: Collection[tuple[int, int]] = ()
src_contents: str, *, mode: Mode, lines: Collection[Tuple[int, int]] = ()
) -> str:
src_node = lib2to3_parse(src_contents.lstrip(), mode.target_versions)
dst_blocks: list[LinesBlock] = []
dst_blocks: List[LinesBlock] = []
if mode.target_versions:
versions = mode.target_versions
else:
@ -1232,7 +1180,7 @@ def _format_str_once(
for feature in {Feature.PARENTHESIZED_CONTEXT_MANAGERS}
if supports_feature(versions, feature)
}
normalize_fmt_off(src_node, mode, lines)
normalize_fmt_off(src_node, mode)
if lines:
# This should be called after normalize_fmt_off.
convert_unchanged_lines(src_node, lines)
@ -1241,10 +1189,7 @@ def _format_str_once(
elt = EmptyLineTracker(mode=mode)
split_line_features = {
feature
for feature in {
Feature.TRAILING_COMMA_IN_CALL,
Feature.TRAILING_COMMA_IN_DEF,
}
for feature in {Feature.TRAILING_COMMA_IN_CALL, Feature.TRAILING_COMMA_IN_DEF}
if supports_feature(versions, feature)
}
block: Optional[LinesBlock] = None
@ -1270,7 +1215,7 @@ def _format_str_once(
return "".join(dst_contents)
def decode_bytes(src: bytes) -> tuple[FileContent, Encoding, NewLine]:
def decode_bytes(src: bytes) -> Tuple[FileContent, Encoding, NewLine]:
"""Return a tuple of (decoded_contents, encoding, newline).
`newline` is either CRLF or LF but `decoded_contents` is decoded with
@ -1288,8 +1233,8 @@ def decode_bytes(src: bytes) -> tuple[FileContent, Encoding, NewLine]:
def get_features_used( # noqa: C901
node: Node, *, future_imports: Optional[set[str]] = None
) -> set[Feature]:
node: Node, *, future_imports: Optional[Set[str]] = None
) -> Set[Feature]:
"""Return a set of (relatively) new Python features used in this file.
Currently looking for:
@ -1307,7 +1252,7 @@ def get_features_used( # noqa: C901
- except* clause;
- variadic generics;
"""
features: set[Feature] = set()
features: Set[Feature] = set()
if future_imports:
features |= {
FUTURE_FLAG_TO_FEATURE[future_import]
@ -1316,14 +1261,15 @@ def get_features_used( # noqa: C901
}
for n in node.pre_order():
if n.type == token.FSTRING_START:
features.add(Feature.F_STRINGS)
elif (
n.type == token.RBRACE
and n.parent is not None
and any(child.type == token.EQUAL for child in n.parent.children)
):
features.add(Feature.DEBUG_F_STRINGS)
if is_string_token(n):
value_head = n.value[:2]
if value_head in {'f"', 'F"', "f'", "F'", "rf", "fr", "RF", "FR"}:
features.add(Feature.F_STRINGS)
if Feature.DEBUG_F_STRINGS not in features:
for span_beg, span_end in iter_fexpr_spans(n.value):
if n.value[span_beg : span_end - 1].rstrip().endswith("="):
features.add(Feature.DEBUG_F_STRINGS)
break
elif is_number_token(n):
if "_" in n.value:
@ -1389,7 +1335,7 @@ def get_features_used( # noqa: C901
if (
len(atom_children) == 3
and atom_children[0].type == token.LPAR
and _contains_asexpr(atom_children[1])
and atom_children[1].type == syms.testlist_gexp
and atom_children[2].type == token.RPAR
):
features.add(Feature.PARENTHESIZED_CONTEXT_MANAGERS)
@ -1419,34 +1365,12 @@ def get_features_used( # noqa: C901
elif n.type in (syms.type_stmt, syms.typeparams):
features.add(Feature.TYPE_PARAMS)
elif (
n.type in (syms.typevartuple, syms.paramspec, syms.typevar)
and n.children[-2].type == token.EQUAL
):
features.add(Feature.TYPE_PARAM_DEFAULTS)
return features
def _contains_asexpr(node: Union[Node, Leaf]) -> bool:
"""Return True if `node` contains an as-pattern."""
if node.type == syms.asexpr_test:
return True
elif node.type == syms.atom:
if (
len(node.children) == 3
and node.children[0].type == token.LPAR
and node.children[2].type == token.RPAR
):
return _contains_asexpr(node.children[1])
elif node.type == syms.testlist_gexp:
return any(_contains_asexpr(child) for child in node.children)
return False
def detect_target_versions(
node: Node, *, future_imports: Optional[set[str]] = None
) -> set[TargetVersion]:
node: Node, *, future_imports: Optional[Set[str]] = None
) -> Set[TargetVersion]:
"""Detect the version to target based on the nodes used."""
features = get_features_used(node, future_imports=future_imports)
return {
@ -1454,11 +1378,11 @@ def detect_target_versions(
}
def get_future_imports(node: Node) -> set[str]:
def get_future_imports(node: Node) -> Set[str]:
"""Return a set of __future__ imports in the file."""
imports: set[str] = set()
imports: Set[str] = set()
def get_imports_from_children(children: list[LN]) -> Generator[str, None, None]:
def get_imports_from_children(children: List[LN]) -> Generator[str, None, None]:
for child in children:
if isinstance(child, Leaf):
if child.type == token.NAME:
@ -1504,19 +1428,12 @@ def get_imports_from_children(children: list[LN]) -> Generator[str, None, None]:
return imports
def _black_info() -> str:
return (
f"Black {__version__} on "
f"Python ({platform.python_implementation()}) {platform.python_version()}"
)
def assert_equivalent(src: str, dst: str) -> None:
"""Raise AssertionError if `src` and `dst` aren't equivalent."""
try:
src_ast = parse_ast(src)
except Exception as exc:
raise ASTSafetyError(
raise AssertionError(
"cannot use --safe with this file; failed to parse source file AST: "
f"{exc}\n"
"This could be caused by running Black with an older Python version "
@ -1527,8 +1444,8 @@ def assert_equivalent(src: str, dst: str) -> None:
dst_ast = parse_ast(dst)
except Exception as exc:
log = dump_to_file("".join(traceback.format_tb(exc.__traceback__)), dst)
raise ASTSafetyError(
f"INTERNAL ERROR: {_black_info()} produced invalid code: {exc}. "
raise AssertionError(
f"INTERNAL ERROR: Black produced invalid code: {exc}. "
"Please report a bug on https://github.com/psf/black/issues. "
f"This invalid output might be helpful: {log}"
) from None
@ -1537,27 +1454,22 @@ def assert_equivalent(src: str, dst: str) -> None:
dst_ast_str = "\n".join(stringify_ast(dst_ast))
if src_ast_str != dst_ast_str:
log = dump_to_file(diff(src_ast_str, dst_ast_str, "src", "dst"))
raise ASTSafetyError(
f"INTERNAL ERROR: {_black_info()} produced code that is not equivalent to"
" the source. Please report a bug on https://github.com/psf/black/issues."
f" This diff might be helpful: {log}"
raise AssertionError(
"INTERNAL ERROR: Black produced code that is not equivalent to the"
" source. Please report a bug on "
f"https://github.com/psf/black/issues. This diff might be helpful: {log}"
) from None
def assert_stable(
src: str, dst: str, mode: Mode, *, lines: Collection[tuple[int, int]] = ()
src: str, dst: str, mode: Mode, *, lines: Collection[Tuple[int, int]] = ()
) -> None:
"""Raise AssertionError if `dst` reformats differently the second time."""
if lines:
# Formatting specified lines requires `adjusted_lines` to map original lines
# to the formatted lines before re-formatting the previously formatted result.
# Due to less-ideal diff algorithm, some edge cases produce incorrect new line
# ranges. Hence for now, we skip the stable check.
# See https://github.com/psf/black/issues/4033 for context.
return
# We shouldn't call format_str() here, because that formats the string
# twice and may hide a bug where we bounce back and forth between two
# versions.
if lines:
lines = adjusted_lines(lines, src, dst)
newdst = _format_str_once(dst, mode=mode, lines=lines)
if dst != newdst:
log = dump_to_file(
@ -1566,9 +1478,9 @@ def assert_stable(
diff(dst, newdst, "first pass", "second pass"),
)
raise AssertionError(
f"INTERNAL ERROR: {_black_info()} produced different code on the second"
" pass of the formatter. Please report a bug on"
f" https://github.com/psf/black/issues. This diff might be helpful: {log}"
"INTERNAL ERROR: Black produced different code on the second pass of the"
" formatter. Please report a bug on https://github.com/psf/black/issues."
f" This diff might be helpful: {log}"
) from None

4
src/black/_width_table.py
View File

@ -1,9 +1,9 @@
# Generated by make_width_table.py
# wcwidth 0.2.6
# Unicode 15.0.0
from typing import Final
from typing import Final, List, Tuple
WIDTH_TABLE: Final[list[tuple[int, int, int]]] = [
WIDTH_TABLE: Final[List[Tuple[int, int, int]]] = [
(0, 0, 0),
(1, 31, -1),
(127, 159, -1),

19
src/black/brackets.py
View File

@ -1,8 +1,7 @@
"""Builds on top of nodes.py to track brackets."""
from collections.abc import Iterable, Sequence
from dataclasses import dataclass, field
from typing import Final, Optional, Union
from typing import Dict, Final, Iterable, List, Optional, Sequence, Set, Tuple, Union
from black.nodes import (
BRACKET,
@ -61,12 +60,12 @@ class BracketTracker:
"""Keeps track of brackets on a line."""
depth: int = 0
bracket_match: dict[tuple[Depth, NodeType], Leaf] = field(default_factory=dict)
delimiters: dict[LeafID, Priority] = field(default_factory=dict)
bracket_match: Dict[Tuple[Depth, NodeType], Leaf] = field(default_factory=dict)
delimiters: Dict[LeafID, Priority] = field(default_factory=dict)
previous: Optional[Leaf] = None
_for_loop_depths: list[int] = field(default_factory=list)
_lambda_argument_depths: list[int] = field(default_factory=list)
invisible: list[Leaf] = field(default_factory=list)
_for_loop_depths: List[int] = field(default_factory=list)
_lambda_argument_depths: List[int] = field(default_factory=list)
invisible: List[Leaf] = field(default_factory=list)
def mark(self, leaf: Leaf) -> None:
"""Mark `leaf` with bracket-related metadata. Keep track of delimiters.
@ -116,7 +115,7 @@ def mark(self, leaf: Leaf) -> None:
if delim and self.previous is not None:
self.delimiters[id(self.previous)] = delim
else:
delim = is_split_after_delimiter(leaf)
delim = is_split_after_delimiter(leaf, self.previous)
if delim:
self.delimiters[id(leaf)] = delim
if leaf.type in OPENING_BRACKETS:
@ -216,7 +215,7 @@ def get_open_lsqb(self) -> Optional[Leaf]:
return self.bracket_match.get((self.depth - 1, token.RSQB))
def is_split_after_delimiter(leaf: Leaf) -> Priority:
def is_split_after_delimiter(leaf: Leaf, previous: Optional[Leaf] = None) -> Priority:
"""Return the priority of the `leaf` delimiter, given a line break after it.
The delimiter priorities returned here are from those delimiters that would
@ -354,7 +353,7 @@ def max_delimiter_priority_in_atom(node: LN) -> Priority:
return 0
def get_leaves_inside_matching_brackets(leaves: Sequence[Leaf]) -> set[LeafID]:
def get_leaves_inside_matching_brackets(leaves: Sequence[Leaf]) -> Set[LeafID]:
"""Return leaves that are inside matching brackets.
The input `leaves` can have non-matching brackets at the head or tail parts.

39
src/black/cache.py
View File

@ -5,16 +5,14 @@
import pickle
import sys
import tempfile
from collections.abc import Iterable
from dataclasses import dataclass, field
from pathlib import Path
from typing import NamedTuple
from typing import Dict, Iterable, NamedTuple, Set, Tuple
from platformdirs import user_cache_dir
from _black_version import version as __version__
from black.mode import Mode
from black.output import err
if sys.version_info >= (3, 11):
from typing import Self
@ -56,28 +54,22 @@ def get_cache_file(mode: Mode) -> Path:
class Cache:
mode: Mode
cache_file: Path
file_data: dict[str, FileData] = field(default_factory=dict)
file_data: Dict[str, FileData] = field(default_factory=dict)
@classmethod
def read(cls, mode: Mode) -> Self:
"""Read the cache if it exists and is well-formed.
"""Read the cache if it exists and is well formed.
If it is not well-formed, the call to write later should
If it is not well formed, the call to write later should
resolve the issue.
"""
cache_file = get_cache_file(mode)
try:
exists = cache_file.exists()
except OSError as e:
# Likely file too long; see #4172 and #4174
err(f"Unable to read cache file {cache_file} due to {e}")
return cls(mode, cache_file)
if not exists:
if not cache_file.exists():
return cls(mode, cache_file)
with cache_file.open("rb") as fobj:
try:
data: dict[str, tuple[float, int, str]] = pickle.load(fobj)
data: Dict[str, Tuple[float, int, str]] = pickle.load(fobj)
file_data = {k: FileData(*v) for k, v in data.items()}
except (pickle.UnpicklingError, ValueError, IndexError):
return cls(mode, cache_file)
@ -109,20 +101,20 @@ def is_changed(self, source: Path) -> bool:
st = res_src.stat()
if st.st_size != old.st_size:
return True
if st.st_mtime != old.st_mtime:
if int(st.st_mtime) != int(old.st_mtime):
new_hash = Cache.hash_digest(res_src)
if new_hash != old.hash:
return True
return False
def filtered_cached(self, sources: Iterable[Path]) -> tuple[set[Path], set[Path]]:
def filtered_cached(self, sources: Iterable[Path]) -> Tuple[Set[Path], Set[Path]]:
"""Split an iterable of paths in `sources` into two sets.
The first contains paths of files that modified on disk or are not in the
cache. The other contains paths to non-modified files.
"""
changed: set[Path] = set()
done: set[Path] = set()
changed: Set[Path] = set()
done: Set[Path] = set()
for src in sources:
if self.is_changed(src):
changed.add(src)
@ -132,16 +124,17 @@ def filtered_cached(self, sources: Iterable[Path]) -> tuple[set[Path], set[Path]
def write(self, sources: Iterable[Path]) -> None:
"""Update the cache file data and write a new cache file."""
self.file_data.update(
**{str(src.resolve()): Cache.get_file_data(src) for src in sources}
)
self.file_data.update(**{
str(src.resolve()): Cache.get_file_data(src) for src in sources
})
try:
CACHE_DIR.mkdir(parents=True, exist_ok=True)
with tempfile.NamedTemporaryFile(
dir=str(self.cache_file.parent), delete=False
) as f:
# We store raw tuples in the cache because it's faster.
data: dict[str, tuple[float, int, str]] = {
# We store raw tuples in the cache because pickling NamedTuples
# doesn't work with mypyc on Python 3.8, and because it's faster.
data: Dict[str, Tuple[float, int, str]] = {
k: (*v,) for k, v in self.file_data.items()
}
pickle.dump(data, f, protocol=4)

189
src/black/comments.py
View File

@ -1,8 +1,7 @@
import re
from collections.abc import Collection, Iterator
from dataclasses import dataclass
from functools import lru_cache
from typing import Final, Optional, Union
from typing import Final, Iterator, List, Optional, Union
from black.mode import Mode, Preview
from black.nodes import (
@ -11,7 +10,6 @@
WHITESPACE,
container_of,
first_leaf_of,
make_simple_prefix,
preceding_leaf,
syms,
)
@ -46,8 +44,6 @@ class ProtoComment:
value: str # content of the comment
newlines: int # how many newlines before the comment
consumed: int # how many characters of the original leaf's prefix did we consume
form_feed: bool # is there a form feed before the comment
leading_whitespace: str # leading whitespace before the comment, if any
def generate_comments(leaf: LN) -> Iterator[Leaf]:
@ -69,34 +65,25 @@ def generate_comments(leaf: LN) -> Iterator[Leaf]:
Inline comments are emitted as regular token.COMMENT leaves. Standalone
are emitted with a fake STANDALONE_COMMENT token identifier.
"""
total_consumed = 0
for pc in list_comments(leaf.prefix, is_endmarker=leaf.type == token.ENDMARKER):
total_consumed = pc.consumed
prefix = make_simple_prefix(pc.newlines, pc.form_feed)
yield Leaf(pc.type, pc.value, prefix=prefix)
normalize_trailing_prefix(leaf, total_consumed)
yield Leaf(pc.type, pc.value, prefix="\n" * pc.newlines)
@lru_cache(maxsize=4096)
def list_comments(prefix: str, *, is_endmarker: bool) -> list[ProtoComment]:
def list_comments(prefix: str, *, is_endmarker: bool) -> List[ProtoComment]:
"""Return a list of :class:`ProtoComment` objects parsed from the given `prefix`."""
result: list[ProtoComment] = []
result: List[ProtoComment] = []
if not prefix or "#" not in prefix:
return result
consumed = 0
nlines = 0
ignored_lines = 0
form_feed = False
for index, full_line in enumerate(re.split("\r?\n", prefix)):
consumed += len(full_line) + 1 # adding the length of the split '\n'
match = re.match(r"^(\s*)(\S.*|)$", full_line)
assert match
whitespace, line = match.groups()
for index, line in enumerate(re.split("\r?\n", prefix)):
consumed += len(line) + 1 # adding the length of the split '\n'
line = line.lstrip()
if not line:
nlines += 1
if "\f" in full_line:
form_feed = True
if not line.startswith("#"):
# Escaped newlines outside of a comment are not really newlines at
# all. We treat a single-line comment following an escaped newline
@ -112,34 +99,13 @@ def list_comments(prefix: str, *, is_endmarker: bool) -> list[ProtoComment]:
comment = make_comment(line)
result.append(
ProtoComment(
type=comment_type,
value=comment,
newlines=nlines,
consumed=consumed,
form_feed=form_feed,
leading_whitespace=whitespace,
type=comment_type, value=comment, newlines=nlines, consumed=consumed
)
)
form_feed = False
nlines = 0
return result
def normalize_trailing_prefix(leaf: LN, total_consumed: int) -> None:
"""Normalize the prefix that's left over after generating comments.
Note: don't use backslashes for formatting or you'll lose your voting rights.
"""
remainder = leaf.prefix[total_consumed:]
if "\\" not in remainder:
nl_count = remainder.count("\n")
form_feed = "\f" in remainder and remainder.endswith("\n")
leaf.prefix = make_simple_prefix(nl_count, form_feed)
return
leaf.prefix = ""
def make_comment(content: str) -> str:
"""Return a consistently formatted comment from the given `content` string.
@ -166,18 +132,14 @@ def make_comment(content: str) -> str:
return "#" + content
def normalize_fmt_off(
node: Node, mode: Mode, lines: Collection[tuple[int, int]]
) -> None:
def normalize_fmt_off(node: Node, mode: Mode) -> None:
"""Convert content between `# fmt: off`/`# fmt: on` into standalone comments."""
try_again = True
while try_again:
try_again = convert_one_fmt_off_pair(node, mode, lines)
try_again = convert_one_fmt_off_pair(node, mode)
def convert_one_fmt_off_pair(
node: Node, mode: Mode, lines: Collection[tuple[int, int]]
) -> bool:
def convert_one_fmt_off_pair(node: Node, mode: Mode) -> bool:
"""Convert content of a single `# fmt: off`/`# fmt: on` into a standalone comment.
Returns True if a pair was converted.
@ -185,24 +147,24 @@ def convert_one_fmt_off_pair(
for leaf in node.leaves():
previous_consumed = 0
for comment in list_comments(leaf.prefix, is_endmarker=False):
is_fmt_off = comment.value in FMT_OFF
is_fmt_skip = _contains_fmt_skip_comment(comment.value, mode)
if (not is_fmt_off and not is_fmt_skip) or (
# Invalid use when `# fmt: off` is applied before a closing bracket.
is_fmt_off
and leaf.type in CLOSING_BRACKETS
):
should_pass_fmt = comment.value in FMT_OFF or _contains_fmt_skip_comment(
comment.value, mode
)
if not should_pass_fmt:
previous_consumed = comment.consumed
continue
# We only want standalone comments. If there's no previous leaf or
# the previous leaf is indentation, it's a standalone comment in
# disguise.
if comment.type != STANDALONE_COMMENT:
if should_pass_fmt and comment.type != STANDALONE_COMMENT:
prev = preceding_leaf(leaf)
if prev:
if is_fmt_off and prev.type not in WHITESPACE:
if comment.value in FMT_OFF and prev.type not in WHITESPACE:
continue
if is_fmt_skip and prev.type in WHITESPACE:
if (
_contains_fmt_skip_comment(comment.value, mode)
and prev.type in WHITESPACE
):
continue
ignored_nodes = list(generate_ignored_nodes(leaf, comment, mode))
@ -214,7 +176,7 @@ def convert_one_fmt_off_pair(
prefix = first.prefix
if comment.value in FMT_OFF:
first.prefix = prefix[comment.consumed :]
if is_fmt_skip:
if _contains_fmt_skip_comment(comment.value, mode):
first.prefix = ""
standalone_comment_prefix = prefix
else:
@ -222,20 +184,10 @@ def convert_one_fmt_off_pair(
prefix[:previous_consumed] + "\n" * comment.newlines
)
hidden_value = "".join(str(n) for n in ignored_nodes)
comment_lineno = leaf.lineno - comment.newlines
if comment.value in FMT_OFF:
fmt_off_prefix = ""
if len(lines) > 0 and not any(
line[0] <= comment_lineno <= line[1] for line in lines
):
# keeping indentation of comment by preserving original whitespaces.
fmt_off_prefix = prefix.split(comment.value)[0]
if "\n" in fmt_off_prefix:
fmt_off_prefix = fmt_off_prefix.split("\n")[-1]
standalone_comment_prefix += fmt_off_prefix
hidden_value = comment.value + "\n" + hidden_value
if is_fmt_skip:
hidden_value += comment.leading_whitespace + comment.value
if _contains_fmt_skip_comment(comment.value, mode):
hidden_value += " " + comment.value
if hidden_value.endswith("\n"):
# That happens when one of the `ignored_nodes` ended with a NEWLINE
# leaf (possibly followed by a DEDENT).
@ -270,7 +222,7 @@ def generate_ignored_nodes(
Stops at the end of the block.
"""
if _contains_fmt_skip_comment(comment.value, mode):
yield from _generate_ignored_nodes_from_fmt_skip(leaf, comment, mode)
yield from _generate_ignored_nodes_from_fmt_skip(leaf, comment)
return
container: Optional[LN] = container_of(leaf)
while container is not None and container.type != token.ENDMARKER:
@ -309,67 +261,23 @@ def generate_ignored_nodes(
def _generate_ignored_nodes_from_fmt_skip(
leaf: Leaf, comment: ProtoComment, mode: Mode
leaf: Leaf, comment: ProtoComment
) -> Iterator[LN]:
"""Generate all leaves that should be ignored by the `# fmt: skip` from `leaf`."""
prev_sibling = leaf.prev_sibling
parent = leaf.parent
ignored_nodes: list[LN] = []
# Need to properly format the leaf prefix to compare it to comment.value,
# which is also formatted
comments = list_comments(leaf.prefix, is_endmarker=False)
if not comments or comment.value != comments[0].value:
return
if prev_sibling is not None:
leaf.prefix = leaf.prefix[comment.consumed :]
if Preview.fix_fmt_skip_in_one_liners not in mode:
siblings = [prev_sibling]
while (
"\n" not in prev_sibling.prefix
and prev_sibling.prev_sibling is not None
):
prev_sibling = prev_sibling.prev_sibling
siblings.insert(0, prev_sibling)
yield from siblings
return
# Generates the nodes to be ignored by `fmt: skip`.
# Nodes to ignore are the ones on the same line as the
# `# fmt: skip` comment, excluding the `# fmt: skip`
# node itself.
# Traversal process (starting at the `# fmt: skip` node):
# 1. Move to the `prev_sibling` of the current node.
# 2. If `prev_sibling` has children, go to its rightmost leaf.
# 3. If there’s no `prev_sibling`, move up to the parent
# node and repeat.
# 4. Continue until:
# a. You encounter an `INDENT` or `NEWLINE` node (indicates
# start of the line).
# b. You reach the root node.
# Include all visited LEAVES in the ignored list, except INDENT
# or NEWLINE leaves.
current_node = prev_sibling
ignored_nodes = [current_node]
if current_node.prev_sibling is None and current_node.parent is not None:
current_node = current_node.parent
while "\n" not in current_node.prefix and current_node.prev_sibling is not None:
leaf_nodes = list(current_node.prev_sibling.leaves())
current_node = leaf_nodes[-1] if leaf_nodes else current_node
if current_node.type in (token.NEWLINE, token.INDENT):
current_node.prefix = ""
break
ignored_nodes.insert(0, current_node)
if current_node.prev_sibling is None and current_node.parent is not None:
current_node = current_node.parent
yield from ignored_nodes
leaf.prefix = ""
siblings = [prev_sibling]
while "\n" not in prev_sibling.prefix and prev_sibling.prev_sibling is not None:
prev_sibling = prev_sibling.prev_sibling
siblings.insert(0, prev_sibling)
yield from siblings
elif (
parent is not None and parent.type == syms.suite and leaf.type == token.NEWLINE
):
@ -377,6 +285,7 @@ def _generate_ignored_nodes_from_fmt_skip(
# statements. The ignored nodes should be previous siblings of the
# parent suite node.
leaf.prefix = ""
ignored_nodes: List[LN] = []
parent_sibling = parent.prev_sibling
while parent_sibling is not None and parent_sibling.type != syms.suite:
ignored_nodes.insert(0, parent_sibling)
@ -416,7 +325,7 @@ def children_contains_fmt_on(container: LN) -> bool:
return False
def contains_pragma_comment(comment_list: list[Leaf]) -> bool:
def contains_pragma_comment(comment_list: List[Leaf]) -> bool:
"""
Returns:
True iff one of the comments in @comment_list is a pragma used by one
@ -438,18 +347,22 @@ def _contains_fmt_skip_comment(comment_line: str, mode: Mode) -> bool:
# noqa:XXX # fmt:skip # a nice line <-- multiple comments (Preview)
# pylint:XXX; fmt:skip <-- list of comments (; separated, Preview)
"""
semantic_comment_blocks = [
comment_line,
*[
_COMMENT_PREFIX + comment.strip()
for comment in comment_line.split(_COMMENT_PREFIX)[1:]
],
*[
_COMMENT_PREFIX + comment.strip()
for comment in comment_line.strip(_COMMENT_PREFIX).split(
_COMMENT_LIST_SEPARATOR
)
],
]
semantic_comment_blocks = (
[
comment_line,
*[
_COMMENT_PREFIX + comment.strip()
for comment in comment_line.split(_COMMENT_PREFIX)[1:]
],
*[
_COMMENT_PREFIX + comment.strip()
for comment in comment_line.strip(_COMMENT_PREFIX).split(
_COMMENT_LIST_SEPARATOR
)
],
]
if Preview.single_line_format_skip_with_multiple_comments in mode
else [comment_line]
)
return any(comment in FMT_SKIP for comment in semantic_comment_blocks)

9
src/black/concurrency.py
View File

@ -10,11 +10,10 @@
import signal
import sys
import traceback
from collections.abc import Iterable
from concurrent.futures import Executor, ProcessPoolExecutor, ThreadPoolExecutor
from multiprocessing import Manager
from pathlib import Path
from typing import Any, Optional
from typing import Any, Iterable, Optional, Set
from mypy_extensions import mypyc_attr
@ -39,7 +38,7 @@ def maybe_install_uvloop() -> None:
pass
def cancel(tasks: Iterable["asyncio.Future[Any]"]) -> None:
def cancel(tasks: Iterable["asyncio.Task[Any]"]) -> None:
"""asyncio signal handler that cancels all `tasks` and reports to stderr."""
err("Aborted!")
for task in tasks:
@ -70,7 +69,7 @@ def shutdown(loop: asyncio.AbstractEventLoop) -> None:
# not ideal, but this shouldn't cause any issues ... hopefully. ~ichard26
@mypyc_attr(patchable=True)
def reformat_many(
sources: set[Path],
sources: Set[Path],
fast: bool,
write_back: WriteBack,
mode: Mode,
@ -120,7 +119,7 @@ def reformat_many(
async def schedule_formatting(
sources: set[Path],
sources: Set[Path],
fast: bool,
write_back: WriteBack,
mode: Mode,

5
src/black/debug.py
View File

@ -1,6 +1,5 @@
from collections.abc import Iterator
from dataclasses import dataclass, field
from typing import Any, TypeVar, Union
from typing import Any, Iterator, List, TypeVar, Union
from black.nodes import Visitor
from black.output import out
@ -15,7 +14,7 @@
@dataclass
class DebugVisitor(Visitor[T]):
tree_depth: int = 0
list_output: list[str] = field(default_factory=list)
list_output: List[str] = field(default_factory=list)
print_output: bool = True
def out(self, message: str, *args: Any, **kwargs: Any) -> None:

119
src/black/files.py
View File

@ -1,11 +1,21 @@
import io
import os
import sys
from collections.abc import Iterable, Iterator, Sequence
from functools import lru_cache
from pathlib import Path
from re import Pattern
from typing import TYPE_CHECKING, Any, Optional, Union
from typing import (
TYPE_CHECKING,
Any,
Dict,
Iterable,
Iterator,
List,
Optional,
Pattern,
Sequence,
Tuple,
Union,
)
from mypy_extensions import mypyc_attr
from packaging.specifiers import InvalidSpecifier, Specifier, SpecifierSet
@ -32,26 +42,12 @@
import colorama # noqa: F401
@lru_cache
def _load_toml(path: Union[Path, str]) -> dict[str, Any]:
with open(path, "rb") as f:
return tomllib.load(f)
@lru_cache
def _cached_resolve(path: Path) -> Path:
return path.resolve()
@lru_cache
def find_project_root(
srcs: Sequence[str], stdin_filename: Optional[str] = None
) -> tuple[Path, str]:
) -> Tuple[Path, str]:
"""Return a directory containing .git, .hg, or pyproject.toml.
pyproject.toml files are only considered if they contain a [tool.black]
section and are ignored otherwise.
That directory will be a common parent of all files and directories
passed in `srcs`.
@ -65,9 +61,9 @@ def find_project_root(
if stdin_filename is not None:
srcs = tuple(stdin_filename if s == "-" else s for s in srcs)
if not srcs:
srcs = [str(_cached_resolve(Path.cwd()))]
srcs = [str(Path.cwd().resolve())]
path_srcs = [_cached_resolve(Path(Path.cwd(), src)) for src in srcs]
path_srcs = [Path(Path.cwd(), src).resolve() for src in srcs]
# A list of lists of parents for each 'src'. 'src' is included as a
# "parent" of itself if it is a directory
@ -88,15 +84,13 @@ def find_project_root(
return directory, ".hg directory"
if (directory / "pyproject.toml").is_file():
pyproject_toml = _load_toml(directory / "pyproject.toml")
if "black" in pyproject_toml.get("tool", {}):
return directory, "pyproject.toml"
return directory, "pyproject.toml"
return directory, "file system root"
def find_pyproject_toml(
path_search_start: tuple[str, ...], stdin_filename: Optional[str] = None
path_search_start: Tuple[str, ...], stdin_filename: Optional[str] = None
) -> Optional[str]:
"""Find the absolute filepath to a pyproject.toml if it exists"""
path_project_root, _ = find_project_root(path_search_start, stdin_filename)
@ -118,13 +112,14 @@ def find_pyproject_toml(
@mypyc_attr(patchable=True)
def parse_pyproject_toml(path_config: str) -> dict[str, Any]:
def parse_pyproject_toml(path_config: str) -> Dict[str, Any]:
"""Parse a pyproject toml file, pulling out relevant parts for Black.
If parsing fails, will raise a tomllib.TOMLDecodeError.
"""
pyproject_toml = _load_toml(path_config)
config: dict[str, Any] = pyproject_toml.get("tool", {}).get("black", {})
with open(path_config, "rb") as f:
pyproject_toml = tomllib.load(f)
config: Dict[str, Any] = pyproject_toml.get("tool", {}).get("black", {})
config = {k.replace("--", "").replace("-", "_"): v for k, v in config.items()}
if "target_version" not in config:
@ -136,8 +131,8 @@ def parse_pyproject_toml(path_config: str) -> dict[str, Any]:
def infer_target_version(
pyproject_toml: dict[str, Any],
) -> Optional[list[TargetVersion]]:
pyproject_toml: Dict[str, Any]
) -> Optional[List[TargetVersion]]:
"""Infer Black's target version from the project metadata in pyproject.toml.
Supports the PyPA standard format (PEP 621):
@ -160,7 +155,7 @@ def infer_target_version(
return None
def parse_req_python_version(requires_python: str) -> Optional[list[TargetVersion]]:
def parse_req_python_version(requires_python: str) -> Optional[List[TargetVersion]]:
"""Parse a version string (i.e. ``"3.7"``) to a list of TargetVersion.
If parsing fails, will raise a packaging.version.InvalidVersion error.
@ -175,7 +170,7 @@ def parse_req_python_version(requires_python: str) -> Optional[list[TargetVersio
return None
def parse_req_python_specifier(requires_python: str) -> Optional[list[TargetVersion]]:
def parse_req_python_specifier(requires_python: str) -> Optional[List[TargetVersion]]:
"""Parse a specifier string (i.e. ``">=3.7,<3.10"``) to a list of TargetVersion.
If parsing fails, will raise a packaging.specifiers.InvalidSpecifier error.
@ -186,7 +181,7 @@ def parse_req_python_specifier(requires_python: str) -> Optional[list[TargetVers
return None
target_version_map = {f"3.{v.value}": v for v in TargetVersion}
compatible_versions: list[str] = list(specifier_set.filter(target_version_map))
compatible_versions: List[str] = list(specifier_set.filter(target_version_map))
if compatible_versions:
return [target_version_map[v] for v in compatible_versions]
return None
@ -234,14 +229,14 @@ def find_user_pyproject_toml() -> Path:
else:
config_root = os.environ.get("XDG_CONFIG_HOME", "~/.config")
user_config_path = Path(config_root).expanduser() / "black"
return _cached_resolve(user_config_path)
return user_config_path.resolve()
@lru_cache
def get_gitignore(root: Path) -> PathSpec:
"""Return a PathSpec matching gitignore content if present."""
gitignore = root / ".gitignore"
lines: list[str] = []
lines: List[str] = []
if gitignore.is_file():
with gitignore.open(encoding="utf-8") as gf:
lines = gf.readlines()
@ -252,47 +247,39 @@ def get_gitignore(root: Path) -> PathSpec:
raise
def resolves_outside_root_or_cannot_stat(
def normalize_path_maybe_ignore(
path: Path,
root: Path,
report: Optional[Report] = None,
) -> bool:
"""
Returns whether the path is a symbolic link that points outside the
root directory. Also returns True if we failed to resolve the path.
) -> Optional[str]:
"""Normalize `path`. May return `None` if `path` was ignored.
`report` is where "path ignored" output goes.
"""
try:
resolved_path = _cached_resolve(path)
abspath = path if path.is_absolute() else Path.cwd() / path
normalized_path = abspath.resolve()
try:
root_relative_path = normalized_path.relative_to(root).as_posix()
except ValueError:
if report:
report.path_ignored(
path, f"is a symbolic link that points outside {root}"
)
return None
except OSError as e:
if report:
report.path_ignored(path, f"cannot be read because {e}")
return True
try:
resolved_path.relative_to(root)
except ValueError:
if report:
report.path_ignored(path, f"is a symbolic link that points outside {root}")
return True
return False
return None
def best_effort_relative_path(path: Path, root: Path) -> Path:
# Precondition: resolves_outside_root_or_cannot_stat(path, root) is False
try:
return path.absolute().relative_to(root)
except ValueError:
pass
root_parent = next((p for p in path.parents if _cached_resolve(p) == root), None)
if root_parent is not None:
return path.relative_to(root_parent)
# something adversarial, fallback to path guaranteed by precondition
return _cached_resolve(path).relative_to(root)
return root_relative_path
def _path_is_ignored(
root_relative_path: str,
root: Path,
gitignore_dict: dict[Path, PathSpec],
gitignore_dict: Dict[Path, PathSpec],
) -> bool:
path = root / root_relative_path
# Note that this logic is sensitive to the ordering of gitignore_dict. Callers must
@ -300,8 +287,6 @@ def _path_is_ignored(
for gitignore_path, pattern in gitignore_dict.items():
try:
relative_path = path.relative_to(gitignore_path).as_posix()
if path.is_dir():
relative_path = relative_path + "/"
except ValueError:
break
if pattern.match_file(relative_path):
@ -325,7 +310,7 @@ def gen_python_files(
extend_exclude: Optional[Pattern[str]],
force_exclude: Optional[Pattern[str]],
report: Report,
gitignore_dict: Optional[dict[Path, PathSpec]],
gitignore_dict: Optional[Dict[Path, PathSpec]],
*,
verbose: bool,
quiet: bool,
@ -341,8 +326,7 @@ def gen_python_files(
assert root.is_absolute(), f"INTERNAL ERROR: `root` must be absolute but is {root}"
for child in paths:
assert child.is_absolute()
root_relative_path = child.relative_to(root).as_posix()
root_relative_path = child.absolute().relative_to(root).as_posix()
# First ignore files matching .gitignore, if passed
if gitignore_dict and _path_is_ignored(
@ -370,7 +354,8 @@ def gen_python_files(
report.path_ignored(child, "matches the --force-exclude regular expression")
continue
if resolves_outside_root_or_cannot_stat(child, root, report):
normalized_path = normalize_path_maybe_ignore(child, root, report)
if normalized_path is None:
continue
if child.is_dir():

154
src/black/handle_ipynb_magics.py
View File

@ -3,46 +3,51 @@
import ast
import collections
import dataclasses
import re
import secrets
import sys
from functools import lru_cache
from importlib.util import find_spec
from typing import Optional
from typing import Dict, List, Optional, Tuple
if sys.version_info >= (3, 10):
from typing import TypeGuard
else:
from typing_extensions import TypeGuard
from black.mode import Mode
from black.output import out
from black.report import NothingChanged
TRANSFORMED_MAGICS = frozenset((
"get_ipython().run_cell_magic",
"get_ipython().system",
"get_ipython().getoutput",
"get_ipython().run_line_magic",
))
TOKENS_TO_IGNORE = frozenset((
"ENDMARKER",
"NL",
"NEWLINE",
"COMMENT",
"DEDENT",
"UNIMPORTANT_WS",
"ESCAPED_NL",
))
PYTHON_CELL_MAGICS = frozenset((
"capture",
"prun",
"pypy",
"python",
"python3",
"time",
"timeit",
))
TRANSFORMED_MAGICS = frozenset(
(
"get_ipython().run_cell_magic",
"get_ipython().system",
"get_ipython().getoutput",
"get_ipython().run_line_magic",
)
)
TOKENS_TO_IGNORE = frozenset(
(
"ENDMARKER",
"NL",
"NEWLINE",
"COMMENT",
"DEDENT",
"UNIMPORTANT_WS",
"ESCAPED_NL",
)
)
PYTHON_CELL_MAGICS = frozenset(
(
"capture",
"prun",
"pypy",
"python",
"python3",
"time",
"timeit",
)
)
TOKEN_HEX = secrets.token_hex
@dataclasses.dataclass(frozen=True)
@ -65,35 +70,7 @@ def jupyter_dependencies_are_installed(*, warn: bool) -> bool:
return installed
def validate_cell(src: str, mode: Mode) -> None:
"""Check that cell does not already contain TransformerManager transformations,
or non-Python cell magics, which might cause tokenizer_rt to break because of
indentations.
If a cell contains ``!ls``, then it'll be transformed to
``get_ipython().system('ls')``. However, if the cell originally contained
``get_ipython().system('ls')``, then it would get transformed in the same way:
>>> TransformerManager().transform_cell("get_ipython().system('ls')")
"get_ipython().system('ls')\n"
>>> TransformerManager().transform_cell("!ls")
"get_ipython().system('ls')\n"
Due to the impossibility of safely roundtripping in such situations, cells
containing transformed magics will be ignored.
"""
if any(transformed_magic in src for transformed_magic in TRANSFORMED_MAGICS):
raise NothingChanged
line = _get_code_start(src)
if line.startswith("%%") and (
line.split(maxsplit=1)[0][2:]
not in PYTHON_CELL_MAGICS | mode.python_cell_magics
):
raise NothingChanged
def remove_trailing_semicolon(src: str) -> tuple[str, bool]:
def remove_trailing_semicolon(src: str) -> Tuple[str, bool]:
"""Remove trailing semicolon from Jupyter notebook cell.
For example,
@ -149,7 +126,7 @@ def put_trailing_semicolon_back(src: str, has_trailing_semicolon: bool) -> str:
return str(tokens_to_src(tokens))
def mask_cell(src: str) -> tuple[str, list[Replacement]]:
def mask_cell(src: str) -> Tuple[str, List[Replacement]]:
"""Mask IPython magics so content becomes parseable Python code.
For example,
@ -159,12 +136,12 @@ def mask_cell(src: str) -> tuple[str, list[Replacement]]:
becomes
b"25716f358c32750"
"25716f358c32750e"
'foo'
The replacements are returned, along with the transformed code.
"""
replacements: list[Replacement] = []
replacements: List[Replacement] = []
try:
ast.parse(src)
except SyntaxError:
@ -177,32 +154,18 @@ def mask_cell(src: str) -> tuple[str, list[Replacement]]:
from IPython.core.inputtransformer2 import TransformerManager
transformer_manager = TransformerManager()
# A side effect of the following transformation is that it also removes any
# empty lines at the beginning of the cell.
transformed = transformer_manager.transform_cell(src)
transformed, cell_magic_replacements = replace_cell_magics(transformed)
replacements += cell_magic_replacements
transformed = transformer_manager.transform_cell(transformed)
transformed, magic_replacements = replace_magics(transformed)
if len(transformed.strip().splitlines()) != len(src.strip().splitlines()):
if len(transformed.splitlines()) != len(src.splitlines()):
# Multi-line magic, not supported.
raise NothingChanged
replacements += magic_replacements
return transformed, replacements
def create_token(n_chars: int) -> str:
"""Create a randomly generated token that is n_chars characters long."""
assert n_chars > 0
n_bytes = max(n_chars // 2 - 1, 1)
token = secrets.token_hex(n_bytes)
if len(token) + 3 > n_chars:
token = token[:-1]
# We use a bytestring so that the string does not get interpreted
# as a docstring.
return f'b"{token}"'
def get_token(src: str, magic: str) -> str:
"""Return randomly generated token to mask IPython magic with.
@ -212,11 +175,11 @@ def get_token(src: str, magic: str) -> str:
not already present anywhere else in the cell.
"""
assert magic
n_chars = len(magic)
token = create_token(n_chars)
nbytes = max(len(magic) // 2 - 1, 1)
token = TOKEN_HEX(nbytes)
counter = 0
while token in src:
token = create_token(n_chars)
token = TOKEN_HEX(nbytes)
counter += 1
if counter > 100:
raise AssertionError(
@ -224,10 +187,12 @@ def get_token(src: str, magic: str) -> str:
"Please report a bug on https://github.com/psf/black/issues. "
f"The magic might be helpful: {magic}"
) from None
return token
if len(token) + 2 < len(magic):
token = f"{token}."
return f'"{token}"'
def replace_cell_magics(src: str) -> tuple[str, list[Replacement]]:
def replace_cell_magics(src: str) -> Tuple[str, List[Replacement]]:
"""Replace cell magic with token.
Note that 'src' will already have been processed by IPython's
@ -244,7 +209,7 @@ def replace_cell_magics(src: str) -> tuple[str, list[Replacement]]:
The replacement, along with the transformed code, is returned.
"""
replacements: list[Replacement] = []
replacements: List[Replacement] = []
tree = ast.parse(src)
@ -258,7 +223,7 @@ def replace_cell_magics(src: str) -> tuple[str, list[Replacement]]:
return f"{mask}\n{cell_magic_finder.cell_magic.body}", replacements
def replace_magics(src: str) -> tuple[str, list[Replacement]]:
def replace_magics(src: str) -> Tuple[str, List[Replacement]]:
"""Replace magics within body of cell.
Note that 'src' will already have been processed by IPython's
@ -280,7 +245,7 @@ def replace_magics(src: str) -> tuple[str, list[Replacement]]:
magic_finder = MagicFinder()
magic_finder.visit(ast.parse(src))
new_srcs = []
for i, line in enumerate(src.split("\n"), start=1):
for i, line in enumerate(src.splitlines(), start=1):
if i in magic_finder.magics:
offsets_and_magics = magic_finder.magics[i]
if len(offsets_and_magics) != 1: # pragma: nocover
@ -299,7 +264,7 @@ def replace_magics(src: str) -> tuple[str, list[Replacement]]:
return "\n".join(new_srcs), replacements
def unmask_cell(src: str, replacements: list[Replacement]) -> str:
def unmask_cell(src: str, replacements: List[Replacement]) -> str:
"""Remove replacements from cell.
For example
@ -317,21 +282,6 @@ def unmask_cell(src: str, replacements: list[Replacement]) -> str:
return src
def _get_code_start(src: str) -> str:
"""Provides the first line where the code starts.
Iterates over lines of code until it finds the first line that doesn't
contain only empty spaces and comments. It removes any empty spaces at the
start of the line and returns it. If such line doesn't exist, it returns an
empty string.
"""
for match in re.finditer(".+", src):
line = match.group(0).lstrip()
if line and not line.startswith("#"):
return line
return ""
def _is_ipython_magic(node: ast.expr) -> TypeGuard[ast.Attribute]:
"""Check if attribute is IPython magic.
@ -347,11 +297,11 @@ def _is_ipython_magic(node: ast.expr) -> TypeGuard[ast.Attribute]:
)
def _get_str_args(args: list[ast.expr]) -> list[str]:
def _get_str_args(args: List[ast.expr]) -> List[str]:
str_args = []
for arg in args:
assert isinstance(arg, ast.Constant) and isinstance(arg.value, str)
str_args.append(arg.value)
assert isinstance(arg, ast.Str)
str_args.append(arg.s)
return str_args
@ -431,7 +381,7 @@ class MagicFinder(ast.NodeVisitor):
"""
def __init__(self) -> None:
self.magics: dict[int, list[OffsetAndMagic]] = collections.defaultdict(list)
self.magics: Dict[int, List[OffsetAndMagic]] = collections.defaultdict(list)
def visit_Assign(self, node: ast.Assign) -> None:
"""Look for system assign magics.

647
src/black/linegen.py
View File

File diff suppressed because it is too large Load Diff

230
src/black/lines.py
View File

@ -1,8 +1,18 @@
import itertools
import math
from collections.abc import Callable, Iterator, Sequence
from dataclasses import dataclass, field
from typing import Optional, TypeVar, Union, cast
from typing import (
Callable,
Dict,
Iterator,
List,
Optional,
Sequence,
Tuple,
TypeVar,
Union,
cast,
)
from black.brackets import COMMA_PRIORITY, DOT_PRIORITY, BracketTracker
from black.mode import Mode, Preview
@ -14,13 +24,13 @@
TEST_DESCENDANTS,
child_towards,
is_docstring,
is_funcdef,
is_import,
is_multiline_string,
is_one_sequence_between,
is_type_comment,
is_type_ignore_comment,
is_with_or_async_with_stmt,
make_simple_prefix,
replace_child,
syms,
whitespace,
@ -42,9 +52,9 @@ class Line:
mode: Mode = field(repr=False)
depth: int = 0
leaves: list[Leaf] = field(default_factory=list)
leaves: List[Leaf] = field(default_factory=list)
# keys ordered like `leaves`
comments: dict[LeafID, list[Leaf]] = field(default_factory=dict)
comments: Dict[LeafID, List[Leaf]] = field(default_factory=dict)
bracket_tracker: BracketTracker = field(default_factory=BracketTracker)
inside_brackets: bool = False
should_split_rhs: bool = False
@ -62,12 +72,7 @@ def append(
Inline comments are put aside.
"""
has_value = (
leaf.type in BRACKETS
# empty fstring-middles must not be truncated
or leaf.type == token.FSTRING_MIDDLE
or bool(leaf.value.strip())
)
has_value = leaf.type in BRACKETS or bool(leaf.value.strip())
if not has_value:
return
@ -86,7 +91,7 @@ def append(
if self.mode.magic_trailing_comma:
if self.has_magic_trailing_comma(leaf):
self.magic_trailing_comma = leaf
elif self.has_magic_trailing_comma(leaf):
elif self.has_magic_trailing_comma(leaf, ensure_removable=True):
self.remove_trailing_comma()
if not self.append_comment(leaf):
self.leaves.append(leaf)
@ -190,27 +195,19 @@ def is_class_paren_empty(self) -> bool:
)
@property
def _is_triple_quoted_string(self) -> bool:
def is_triple_quoted_string(self) -> bool:
"""Is the line a triple quoted string?"""
if not self or self.leaves[0].type != token.STRING:
return False
value = self.leaves[0].value
if value.startswith(('"""', "'''")):
return True
if value.startswith(("r'''", 'r"""', "R'''", 'R"""')):
if Preview.accept_raw_docstrings in self.mode and value.startswith(
("r'''", 'r"""', "R'''", 'R"""')
):
return True
return False
@property
def is_docstring(self) -> bool:
"""Is the line a docstring?"""
return bool(self) and is_docstring(self.leaves[0])
@property
def is_chained_assignment(self) -> bool:
"""Is the line a chained assignment"""
return [leaf.type for leaf in self.leaves].count(token.EQUAL) > 1
@property
def opens_block(self) -> bool:
"""Does this line open a new level of indentation."""
@ -333,11 +330,16 @@ def contains_unsplittable_type_ignore(self) -> bool:
def contains_multiline_strings(self) -> bool:
return any(is_multiline_string(leaf) for leaf in self.leaves)
def has_magic_trailing_comma(self, closing: Leaf) -> bool:
def has_magic_trailing_comma(
self, closing: Leaf, ensure_removable: bool = False
) -> bool:
"""Return True if we have a magic trailing comma, that is when:
- there's a trailing comma here
- it's not from single-element square bracket indexing
- it's not a one-tuple
- it's not a single-element subscript
Additionally, if ensure_removable:
- it's not from square bracket indexing
(specifically, single-element square bracket indexing)
"""
if not (
closing.type in CLOSING_BRACKETS
@ -361,8 +363,6 @@ def has_magic_trailing_comma(self, closing: Leaf) -> bool:
brackets=(token.LSQB, token.RSQB),
)
):
assert closing.prev_sibling is not None
assert closing.prev_sibling.type == syms.subscriptlist
return False
return True
@ -414,7 +414,7 @@ def append_comment(self, comment: Leaf) -> bool:
self.comments.setdefault(id(last_leaf), []).append(comment)
return True
def comments_after(self, leaf: Leaf) -> list[Leaf]:
def comments_after(self, leaf: Leaf) -> List[Leaf]:
"""Generate comments that should appear directly after `leaf`."""
return self.comments.get(id(leaf), [])
@ -440,21 +440,20 @@ def is_complex_subscript(self, leaf: Leaf) -> bool:
if subscript_start.type == syms.subscriptlist:
subscript_start = child_towards(subscript_start, leaf)
return subscript_start is not None and any(
n.type in TEST_DESCENDANTS for n in subscript_start.pre_order()
)
def enumerate_with_length(
self, is_reversed: bool = False
) -> Iterator[tuple[Index, Leaf, int]]:
self, reversed: bool = False
) -> Iterator[Tuple[Index, Leaf, int]]:
"""Return an enumeration of leaves with their length.
Stops prematurely on multiline strings and standalone comments.
"""
op = cast(
Callable[[Sequence[Leaf]], Iterator[tuple[Index, Leaf]]],
enumerate_reversed if is_reversed else enumerate,
Callable[[Sequence[Leaf]], Iterator[Tuple[Index, Leaf]]],
enumerate_reversed if reversed else enumerate,
)
for index, leaf in op(self.leaves):
length = len(leaf.prefix) + len(leaf.value)
@ -519,14 +518,14 @@ class LinesBlock:
previous_block: Optional["LinesBlock"]
original_line: Line
before: int = 0
content_lines: list[str] = field(default_factory=list)
content_lines: List[str] = field(default_factory=list)
after: int = 0
form_feed: bool = False
def all_lines(self) -> list[str]:
def all_lines(self) -> List[str]:
empty_line = str(Line(mode=self.mode))
prefix = make_simple_prefix(self.before, self.form_feed, empty_line)
return [prefix] + self.content_lines + [empty_line * self.after]
return (
[empty_line * self.before] + self.content_lines + [empty_line * self.after]
)
@dataclass
@ -542,7 +541,7 @@ class EmptyLineTracker:
mode: Mode
previous_line: Optional[Line] = None
previous_block: Optional[LinesBlock] = None
previous_defs: list[Line] = field(default_factory=list)
previous_defs: List[Line] = field(default_factory=list)
semantic_leading_comment: Optional[LinesBlock] = None
def maybe_empty_lines(self, current_line: Line) -> LinesBlock:
@ -551,20 +550,21 @@ def maybe_empty_lines(self, current_line: Line) -> LinesBlock:
This is for separating `def`, `async def` and `class` with extra empty
lines (two on module-level).
"""
form_feed = (
current_line.depth == 0
and bool(current_line.leaves)
and "\f\n" in current_line.leaves[0].prefix
)
before, after = self._maybe_empty_lines(current_line)
previous_after = self.previous_block.after if self.previous_block else 0
before = max(0, before - previous_after)
before = (
# Black should not insert empty lines at the beginning
# of the file
0
if self.previous_line is None
else before - previous_after
)
if (
# Always have one empty line after a module docstring
self.previous_block
Preview.module_docstring_newlines in current_line.mode
and self.previous_block
and self.previous_block.previous_block is None
and len(self.previous_block.original_line.leaves) == 1
and self.previous_block.original_line.is_docstring
and self.previous_block.original_line.is_triple_quoted_string
and not (current_line.is_class or current_line.is_def)
):
before = 1
@ -575,7 +575,6 @@ def maybe_empty_lines(self, current_line: Line) -> LinesBlock:
original_line=current_line,
before=before,
after=after,
form_feed=form_feed,
)
# Maintain the semantic_leading_comment state.
@ -595,11 +594,10 @@ def maybe_empty_lines(self, current_line: Line) -> LinesBlock:
self.previous_block = block
return block
def _maybe_empty_lines(self, current_line: Line) -> tuple[int, int]: # noqa: C901
def _maybe_empty_lines(self, current_line: Line) -> Tuple[int, int]:
max_allowed = 1
if current_line.depth == 0:
max_allowed = 1 if self.mode.is_pyi else 2
if current_line.leaves:
# Consume the first leaf's extra newlines.
first_leaf = current_line.leaves[0]
@ -612,31 +610,22 @@ def _maybe_empty_lines(self, current_line: Line) -> tuple[int, int]: # noqa: C9
user_had_newline = bool(before)
depth = current_line.depth
# Mutate self.previous_defs, remainder of this function should be pure
previous_def = None
while self.previous_defs and self.previous_defs[-1].depth >= depth:
previous_def = self.previous_defs.pop()
if current_line.is_def or current_line.is_class:
self.previous_defs.append(current_line)
if self.previous_line is None:
# Don't insert empty lines before the first line in the file.
return 0, 0
if current_line.is_docstring:
if self.previous_line.is_class:
return 0, 1
if self.previous_line.opens_block and self.previous_line.is_def:
return 0, 0
if previous_def is not None:
assert self.previous_line is not None
if self.mode.is_pyi:
if previous_def.is_class and not previous_def.is_stub_class:
before = 1
elif depth and not current_line.is_def and self.previous_line.is_def:
if depth and not current_line.is_def and self.previous_line.is_def:
# Empty lines between attributes and methods should be preserved.
before = 1 if user_had_newline else 0
elif (
Preview.blank_line_after_nested_stub_class in self.mode
and previous_def.is_class
and not previous_def.is_stub_class
):
before = 1
elif depth:
before = 0
else:
@ -670,41 +659,70 @@ def _maybe_empty_lines(self, current_line: Line) -> tuple[int, int]: # noqa: C9
)
if (
self.previous_line.is_import
and self.previous_line.depth == 0
and current_line.depth == 0
and not current_line.is_import
and Preview.always_one_newline_after_import in self.mode
):
return 1, 0
if (
self.previous_line.is_import
self.previous_line
and self.previous_line.is_import
and not current_line.is_import
and not current_line.is_fmt_pass_converted(first_leaf_matches=is_import)
and depth == self.previous_line.depth
):
return (before or 1), 0
if (
self.previous_line
and self.previous_line.is_class
and current_line.is_triple_quoted_string
):
if Preview.no_blank_line_before_class_docstring in current_line.mode:
return 0, 1
return before, 1
is_empty_first_line_ok = (
Preview.allow_empty_first_line_before_new_block_or_comment
in current_line.mode
and (
# If it's a standalone comment
current_line.leaves[0].type == STANDALONE_COMMENT
# If it opens a new block
or current_line.opens_block
# If it's a triple quote comment (but not at the start of a funcdef)
or (
is_docstring(current_line.leaves[0])
and self.previous_line
and self.previous_line.leaves[0]
and self.previous_line.leaves[0].parent
and not is_funcdef(self.previous_line.leaves[0].parent)
)
)
)
if (
self.previous_line
and self.previous_line.opens_block
and not is_empty_first_line_ok
):
return 0, 0
return before, 0
def _maybe_empty_lines_for_class_or_def( # noqa: C901
self, current_line: Line, before: int, user_had_newline: bool
) -> tuple[int, int]:
assert self.previous_line is not None
) -> Tuple[int, int]:
if not current_line.is_decorator:
self.previous_defs.append(current_line)
if self.previous_line is None:
# Don't insert empty lines before the first line in the file.
return 0, 0
if self.previous_line.is_decorator:
if self.mode.is_pyi and current_line.is_stub_class:
# Insert an empty line after a decorated stub class
return 0, 1
return 0, 0
if self.previous_line.depth < current_line.depth and (
self.previous_line.is_class or self.previous_line.is_def
):
if self.mode.is_pyi:
return 0, 0
return 1 if user_had_newline else 0, 0
return 0, 0
comment_to_add_newlines: Optional[LinesBlock] = None
if (
@ -735,10 +753,16 @@ def _maybe_empty_lines_for_class_or_def( # noqa: C901
newlines = 0
else:
newlines = 1
# Remove case `self.previous_line.depth > current_line.depth` below when
# this becomes stable.
#
# Don't inspect the previous line if it's part of the body of the previous
# statement in the same level, we always want a blank line if there's
# something with a body preceding.
elif self.previous_line.depth > current_line.depth:
elif (
Preview.blank_line_between_nested_and_def_stub_file in current_line.mode
and self.previous_line.depth > current_line.depth
):
newlines = 1
elif (
current_line.is_def or current_line.is_decorator
@ -751,13 +775,19 @@ def _maybe_empty_lines_for_class_or_def( # noqa: C901
# Blank line between a block of functions (maybe with preceding
# decorators) and a block of non-functions
newlines = 1
elif self.previous_line.depth > current_line.depth:
newlines = 1
else:
newlines = 0
else:
newlines = 1 if current_line.depth else 2
# If a user has left no space after a dummy implementation, don't insert
# new lines. This is useful for instance for @overload or Protocols.
if self.previous_line.is_stub_def and not user_had_newline:
if (
Preview.dummy_implementations in self.mode
and self.previous_line.is_stub_def
and not user_had_newline
):
newlines = 0
if comment_to_add_newlines is not None:
previous_block = comment_to_add_newlines.previous_block
@ -769,7 +799,7 @@ def _maybe_empty_lines_for_class_or_def( # noqa: C901
return newlines, 0
def enumerate_reversed(sequence: Sequence[T]) -> Iterator[tuple[Index, T]]:
def enumerate_reversed(sequence: Sequence[T]) -> Iterator[Tuple[Index, T]]:
"""Like `reversed(enumerate(sequence))` if that were possible."""
index = len(sequence) - 1
for element in reversed(sequence):
@ -778,7 +808,7 @@ def enumerate_reversed(sequence: Sequence[T]) -> Iterator[tuple[Index, T]]:
def append_leaves(
new_line: Line, old_line: Line, leaves: list[Leaf], preformatted: bool = False
new_line: Line, old_line: Line, leaves: List[Leaf], preformatted: bool = False
) -> None:
"""
Append leaves (taken from @old_line) to @new_line, making sure to fix the
@ -812,9 +842,11 @@ def is_line_short_enough( # noqa: C901
if not line_str:
line_str = line_to_string(line)
width = str_width if mode.preview else len
if Preview.multiline_string_handling not in mode:
return (
str_width(line_str) <= mode.line_length
width(line_str) <= mode.line_length
and "\n" not in line_str # multiline strings
and not line.contains_standalone_comments()
)
@ -823,10 +855,10 @@ def is_line_short_enough( # noqa: C901
return False
if "\n" not in line_str:
# No multiline strings (MLS) present
return str_width(line_str) <= mode.line_length
return width(line_str) <= mode.line_length
first, *_, last = line_str.split("\n")
if str_width(first) > mode.line_length or str_width(last) > mode.line_length:
if width(first) > mode.line_length or width(last) > mode.line_length:
return False
# Traverse the AST to examine the context of the multiline string (MLS),
@ -835,10 +867,10 @@ def is_line_short_enough( # noqa: C901
# Depth (which is based on the existing bracket_depth concept)
# is needed to determine nesting level of the MLS.
# Includes special case for trailing commas.
commas: list[int] = [] # tracks number of commas per depth level
commas: List[int] = [] # tracks number of commas per depth level
multiline_string: Optional[Leaf] = None
# store the leaves that contain parts of the MLS
multiline_string_contexts: list[LN] = []
multiline_string_contexts: List[LN] = []
max_level_to_update: Union[int, float] = math.inf # track the depth of the MLS
for i, leaf in enumerate(line.leaves):
@ -860,13 +892,11 @@ def is_line_short_enough( # noqa: C901
return False
if leaf.bracket_depth <= max_level_to_update and leaf.type == token.COMMA:
# Inside brackets, ignore trailing comma
# Ignore non-nested trailing comma
# directly after MLS/MLS-containing expression
ignore_ctxs: list[Optional[LN]] = [None]
ignore_ctxs: List[Optional[LN]] = [None]
ignore_ctxs += multiline_string_contexts
if (line.inside_brackets or leaf.bracket_depth > 0) and (
i != len(line.leaves) - 1 or leaf.prev_sibling not in ignore_ctxs
):
if not (leaf.prev_sibling in ignore_ctxs and i == len(line.leaves) - 1):
commas[leaf.bracket_depth] += 1
if max_level_to_update != math.inf:
max_level_to_update = min(max_level_to_update, leaf.bracket_depth)
@ -968,7 +998,11 @@ def can_omit_invisible_parens(
return False
if delimiter_count == 1:
if max_priority == COMMA_PRIORITY and rhs.head.is_with_or_async_with_stmt:
if (
Preview.wrap_multiple_context_managers_in_parens in line.mode
and max_priority == COMMA_PRIORITY
and rhs.head.is_with_or_async_with_stmt
):
# For two context manager with statements, the optional parentheses read
# better. In this case, `rhs.body` is the context managers part of
# the with statement. `rhs.head` is the `with (` part on the previous

120
src/black/mode.py
View File

@ -8,7 +8,8 @@
from enum import Enum, auto
from hashlib import sha256
from operator import attrgetter
from typing import Final
from typing import Dict, Final, Set
from warnings import warn
from black.const import DEFAULT_LINE_LENGTH
@ -24,11 +25,6 @@ class TargetVersion(Enum):
PY310 = 10
PY311 = 11
PY312 = 12
PY313 = 13
def pretty(self) -> str:
assert self.name[:2] == "PY"
return f"Python {self.name[2]}.{self.name[3:]}"
class Feature(Enum):
@ -51,8 +47,6 @@ class Feature(Enum):
DEBUG_F_STRINGS = 16
PARENTHESIZED_CONTEXT_MANAGERS = 17
TYPE_PARAMS = 18
FSTRING_PARSING = 19
TYPE_PARAM_DEFAULTS = 20
FORCE_OPTIONAL_PARENTHESES = 50
# __future__ flags
@ -64,7 +58,7 @@ class Feature(Enum):
}
VERSION_TO_FEATURES: dict[TargetVersion, set[Feature]] = {
VERSION_TO_FEATURES: Dict[TargetVersion, Set[Feature]] = {
TargetVersion.PY33: {Feature.ASYNC_IDENTIFIERS},
TargetVersion.PY34: {Feature.ASYNC_IDENTIFIERS},
TargetVersion.PY35: {Feature.TRAILING_COMMA_IN_CALL, Feature.ASYNC_IDENTIFIERS},
@ -163,93 +157,80 @@ class Feature(Enum):
Feature.EXCEPT_STAR,
Feature.VARIADIC_GENERICS,
Feature.TYPE_PARAMS,
Feature.FSTRING_PARSING,
},
TargetVersion.PY313: {
Feature.F_STRINGS,
Feature.DEBUG_F_STRINGS,
Feature.NUMERIC_UNDERSCORES,
Feature.TRAILING_COMMA_IN_CALL,
Feature.TRAILING_COMMA_IN_DEF,
Feature.ASYNC_KEYWORDS,
Feature.FUTURE_ANNOTATIONS,
Feature.ASSIGNMENT_EXPRESSIONS,
Feature.RELAXED_DECORATORS,
Feature.POS_ONLY_ARGUMENTS,
Feature.UNPACKING_ON_FLOW,
Feature.ANN_ASSIGN_EXTENDED_RHS,
Feature.PARENTHESIZED_CONTEXT_MANAGERS,
Feature.PATTERN_MATCHING,
Feature.EXCEPT_STAR,
Feature.VARIADIC_GENERICS,
Feature.TYPE_PARAMS,
Feature.FSTRING_PARSING,
Feature.TYPE_PARAM_DEFAULTS,
},
}
def supports_feature(target_versions: set[TargetVersion], feature: Feature) -> bool:
def supports_feature(target_versions: Set[TargetVersion], feature: Feature) -> bool:
return all(feature in VERSION_TO_FEATURES[version] for version in target_versions)
class Preview(Enum):
"""Individual preview style features."""
add_trailing_comma_consistently = auto()
blank_line_after_nested_stub_class = auto()
blank_line_between_nested_and_def_stub_file = auto()
hex_codes_in_unicode_sequences = auto()
improved_async_statements_handling = auto()
multiline_string_handling = auto()
no_blank_line_before_class_docstring = auto()
prefer_splitting_right_hand_side_of_assignments = auto()
# NOTE: string_processing requires wrap_long_dict_values_in_parens
# for https://github.com/psf/black/issues/3117 to be fixed.
string_processing = auto()
hug_parens_with_braces_and_square_brackets = auto()
parenthesize_conditional_expressions = auto()
parenthesize_long_type_hints = auto()
respect_magic_trailing_comma_in_return_type = auto()
skip_magic_trailing_comma_in_subscript = auto()
wrap_long_dict_values_in_parens = auto()
multiline_string_handling = auto()
always_one_newline_after_import = auto()
fix_fmt_skip_in_one_liners = auto()
UNSTABLE_FEATURES: set[Preview] = {
# Many issues, see summary in https://github.com/psf/black/issues/4042
Preview.string_processing,
# See issue #4159
Preview.multiline_string_handling,
# See issue #4036 (crash), #4098, #4099 (proposed tweaks)
Preview.hug_parens_with_braces_and_square_brackets,
}
wrap_multiple_context_managers_in_parens = auto()
dummy_implementations = auto()
walrus_subscript = auto()
module_docstring_newlines = auto()
accept_raw_docstrings = auto()
fix_power_op_line_length = auto()
hug_parens_with_braces_and_square_brackets = auto()
allow_empty_first_line_before_new_block_or_comment = auto()
single_line_format_skip_with_multiple_comments = auto()
long_case_block_line_splitting = auto()
class Deprecated(UserWarning):
"""Visible deprecation warning."""
_MAX_CACHE_KEY_PART_LENGTH: Final = 32
@dataclass
class Mode:
target_versions: set[TargetVersion] = field(default_factory=set)
target_versions: Set[TargetVersion] = field(default_factory=set)
line_length: int = DEFAULT_LINE_LENGTH
string_normalization: bool = True
is_pyi: bool = False
is_ipynb: bool = False
skip_source_first_line: bool = False
magic_trailing_comma: bool = True
python_cell_magics: set[str] = field(default_factory=set)
experimental_string_processing: bool = False
python_cell_magics: Set[str] = field(default_factory=set)
preview: bool = False
unstable: bool = False
enabled_features: set[Preview] = field(default_factory=set)
def __post_init__(self) -> None:
if self.experimental_string_processing:
warn(
"`experimental string processing` has been included in `preview`"
" and deprecated. Use `preview` instead.",
Deprecated,
)
def __contains__(self, feature: Preview) -> bool:
"""
Provide `Preview.FEATURE in Mode` syntax that mirrors the ``preview`` flag.
In unstable mode, all features are enabled. In preview mode, all features
except those in UNSTABLE_FEATURES are enabled. Any features in
`self.enabled_features` are also enabled.
The argument is not checked and features are not differentiated.
They only exist to make development easier by clarifying intent.
"""
if self.unstable:
return True
if feature in self.enabled_features:
return True
return self.preview and feature not in UNSTABLE_FEATURES
if feature is Preview.string_processing:
return self.preview or self.experimental_string_processing
return self.preview
def get_cache_key(self) -> str:
if self.target_versions:
@ -259,19 +240,6 @@ def get_cache_key(self) -> str:
)
else:
version_str = "-"
if len(version_str) > _MAX_CACHE_KEY_PART_LENGTH:
version_str = sha256(version_str.encode()).hexdigest()[
:_MAX_CACHE_KEY_PART_LENGTH
]
features_and_magics = (
",".join(sorted(f.name for f in self.enabled_features))
+ "@"
+ ",".join(sorted(self.python_cell_magics))
)
if len(features_and_magics) > _MAX_CACHE_KEY_PART_LENGTH:
features_and_magics = sha256(features_and_magics.encode()).hexdigest()[
:_MAX_CACHE_KEY_PART_LENGTH
]
parts = [
version_str,
str(self.line_length),
@ -280,8 +248,8 @@ def get_cache_key(self) -> str:
str(int(self.is_ipynb)),
str(int(self.skip_source_first_line)),
str(int(self.magic_trailing_comma)),
str(int(self.experimental_string_processing)),
str(int(self.preview)),
str(int(self.unstable)),
features_and_magics,
sha256((",".join(sorted(self.python_cell_magics))).encode()).hexdigest(),
]
return ".".join(parts)

177
src/black/nodes.py
View File

@ -3,8 +3,7 @@
"""
import sys
from collections.abc import Iterator
from typing import Final, Generic, Literal, Optional, TypeVar, Union
from typing import Final, Generic, Iterator, List, Optional, Set, Tuple, TypeVar, Union
if sys.version_info >= (3, 10):
from typing import TypeGuard
@ -14,7 +13,7 @@
from mypy_extensions import mypyc_attr
from black.cache import CACHE_DIR
from black.mode import Mode
from black.mode import Mode, Preview
from black.strings import get_string_prefix, has_triple_quotes
from blib2to3 import pygram
from blib2to3.pgen2 import token
@ -105,7 +104,6 @@
syms.trailer,
syms.term,
syms.power,
syms.namedexpr_test,
}
TYPED_NAMES: Final = {syms.tname, syms.tname_star}
ASSIGNMENTS: Final = {
@ -123,7 +121,6 @@
">>=",
"**=",
"//=",
":",
}
IMPLICIT_TUPLE: Final = {syms.testlist, syms.testlist_star_expr, syms.exprlist}
@ -135,13 +132,7 @@
OPENING_BRACKETS: Final = set(BRACKET.keys())
CLOSING_BRACKETS: Final = set(BRACKET.values())
BRACKETS: Final = OPENING_BRACKETS | CLOSING_BRACKETS
ALWAYS_NO_SPACE: Final = CLOSING_BRACKETS | {
token.COMMA,
STANDALONE_COMMENT,
token.FSTRING_MIDDLE,
token.FSTRING_END,
token.BANG,
}
ALWAYS_NO_SPACE: Final = CLOSING_BRACKETS | {token.COMMA, STANDALONE_COMMENT}
RARROW = 55
@ -207,9 +198,6 @@ def whitespace(leaf: Leaf, *, complex_subscript: bool, mode: Mode) -> str: # no
}:
return NO
if t == token.LBRACE and p.type == syms.fstring_replacement_field:
return NO
prev = leaf.prev_sibling
if not prev:
prevp = preceding_leaf(p)
@ -244,9 +232,9 @@ def whitespace(leaf: Leaf, *, complex_subscript: bool, mode: Mode) -> str: # no
elif (
prevp.type == token.STAR
and parent_type(prevp) == syms.star_expr
and parent_type(prevp.parent) in (syms.subscriptlist, syms.tname_star)
and parent_type(prevp.parent) == syms.subscriptlist
):
# No space between typevar tuples or unpacking them.
# No space between typevar tuples.
return NO
elif prevp.type in VARARGS_SPECIALS:
@ -271,9 +259,6 @@ def whitespace(leaf: Leaf, *, complex_subscript: bool, mode: Mode) -> str: # no
elif prev.type in OPENING_BRACKETS:
return NO
elif prev.type == token.BANG:
return NO
if p.type in {syms.parameters, syms.arglist}:
# untyped function signatures or calls
if not prev or prev.type != token.COMMA:
@ -361,7 +346,9 @@ def whitespace(leaf: Leaf, *, complex_subscript: bool, mode: Mode) -> str: # no
return NO
elif t == token.COLONEQUAL or prev.type == token.COLONEQUAL:
elif Preview.walrus_subscript in mode and (
t == token.COLONEQUAL or prev.type == token.COLONEQUAL
):
return SPACE
elif not complex_subscript:
@ -395,7 +382,6 @@ def whitespace(leaf: Leaf, *, complex_subscript: bool, mode: Mode) -> str: # no
elif prevp.type == token.EQUAL and prevp_parent.type == syms.argument:
return NO
# TODO: add fstring here?
elif t in {token.NAME, token.NUMBER, token.STRING}:
return NO
@ -421,13 +407,6 @@ def whitespace(leaf: Leaf, *, complex_subscript: bool, mode: Mode) -> str: # no
return SPACE
def make_simple_prefix(nl_count: int, form_feed: bool, empty_line: str = "\n") -> str:
"""Generate a normalized prefix string."""
if form_feed:
return (empty_line * (nl_count - 1)) + "\f" + empty_line
return empty_line * nl_count
def preceding_leaf(node: Optional[LN]) -> Optional[Leaf]:
"""Return the first leaf that precedes `node`, if any."""
while node:
@ -446,7 +425,7 @@ def preceding_leaf(node: Optional[LN]) -> Optional[Leaf]:
return None
def prev_siblings_are(node: Optional[LN], tokens: list[Optional[NodeType]]) -> bool:
def prev_siblings_are(node: Optional[LN], tokens: List[Optional[NodeType]]) -> bool:
"""Return if the `node` and its previous siblings match types against the provided
list of tokens; the provided `node`has its type matched against the last element in
the list. `None` can be used as the first element to declare that the start of the
@ -545,31 +524,21 @@ def is_arith_like(node: LN) -> bool:
}
def is_docstring(node: NL) -> bool:
if isinstance(node, Leaf):
if node.type != token.STRING:
return False
def is_docstring(leaf: Leaf) -> bool:
if leaf.type != token.STRING:
return False
prefix = get_string_prefix(node.value)
if set(prefix).intersection("bBfF"):
return False
if (
node.parent
and node.parent.type == syms.simple_stmt
and not node.parent.prev_sibling
and node.parent.parent
and node.parent.parent.type == syms.file_input
):
return True
prefix = get_string_prefix(leaf.value)
if set(prefix).intersection("bBfF"):
return False
if prev_siblings_are(
node.parent, [None, token.NEWLINE, token.INDENT, syms.simple_stmt]
leaf.parent, [None, token.NEWLINE, token.INDENT, syms.simple_stmt]
):
return True
# Multiline docstring on the same line as the `def`.
if prev_siblings_are(node.parent, [syms.parameters, token.COLON, syms.simple_stmt]):
if prev_siblings_are(leaf.parent, [syms.parameters, token.COLON, syms.simple_stmt]):
# `syms.parameters` is only used in funcdefs and async_funcdefs in the Python
# grammar. We're safe to return True without further checks.
return True
@ -603,17 +572,6 @@ def is_one_tuple(node: LN) -> bool:
)
def is_tuple(node: LN) -> bool:
"""Return True if `node` holds a tuple."""
if node.type != syms.atom:
return False
gexp = unwrap_singleton_parenthesis(node)
if gexp is None or gexp.type != syms.testlist_gexp:
return False
return True
def is_tuple_containing_walrus(node: LN) -> bool:
"""Return True if `node` holds a tuple that contains a walrus operator."""
if node.type != syms.atom:
@ -625,33 +583,11 @@ def is_tuple_containing_walrus(node: LN) -> bool:
return any(child.type == syms.namedexpr_test for child in gexp.children)
def is_tuple_containing_star(node: LN) -> bool:
"""Return True if `node` holds a tuple that contains a star operator."""
if node.type != syms.atom:
return False
gexp = unwrap_singleton_parenthesis(node)
if gexp is None or gexp.type != syms.testlist_gexp:
return False
return any(child.type == syms.star_expr for child in gexp.children)
def is_generator(node: LN) -> bool:
"""Return True if `node` holds a generator."""
if node.type != syms.atom:
return False
gexp = unwrap_singleton_parenthesis(node)
if gexp is None or gexp.type != syms.testlist_gexp:
return False
return any(child.type == syms.old_comp_for for child in gexp.children)
def is_one_sequence_between(
opening: Leaf,
closing: Leaf,
leaves: list[Leaf],
brackets: tuple[int, int] = (token.LPAR, token.RPAR),
leaves: List[Leaf],
brackets: Tuple[int, int] = (token.LPAR, token.RPAR),
) -> bool:
"""Return True if content between `opening` and `closing` is a one-sequence."""
if (opening.type, closing.type) != brackets:
@ -761,7 +697,7 @@ def is_yield(node: LN) -> bool:
return False
def is_vararg(leaf: Leaf, within: set[NodeType]) -> bool:
def is_vararg(leaf: Leaf, within: Set[NodeType]) -> bool:
"""Return True if `leaf` is a star or double star in a vararg or kwarg.
If `within` includes VARARGS_PARENTS, this applies to function signatures.
@ -784,46 +720,17 @@ def is_vararg(leaf: Leaf, within: set[NodeType]) -> bool:
return p.type in within
def is_fstring(node: Node) -> bool:
"""Return True if the node is an f-string"""
return node.type == syms.fstring
def fstring_to_string(node: Node) -> Leaf:
"""Converts an fstring node back to a string node."""
string_without_prefix = str(node)[len(node.prefix) :]
string_leaf = Leaf(token.STRING, string_without_prefix, prefix=node.prefix)
string_leaf.lineno = node.get_lineno() or 0
return string_leaf
def is_multiline_string(node: LN) -> bool:
def is_multiline_string(leaf: Leaf) -> bool:
"""Return True if `leaf` is a multiline string that actually spans many lines."""
if isinstance(node, Node) and is_fstring(node):
leaf = fstring_to_string(node)
elif isinstance(node, Leaf):
leaf = node
else:
return False
return has_triple_quotes(leaf.value) and "\n" in leaf.value
def is_parent_function_or_class(node: Node) -> bool:
assert node.type in {syms.suite, syms.simple_stmt}
assert node.parent is not None
# Note this works for suites / simple_stmts in async def as well
return node.parent.type in {syms.funcdef, syms.classdef}
def is_function_or_class(node: Node) -> bool:
return node.type in {syms.funcdef, syms.classdef, syms.async_funcdef}
def is_funcdef(node: Node) -> bool:
return node.type == syms.funcdef
def is_stub_suite(node: Node) -> bool:
"""Return True if `node` is a suite with a stub body."""
if node.parent is not None and not is_parent_function_or_class(node):
return False
# If there is a comment, we want to keep it.
if node.prefix.strip():
@ -1010,26 +917,24 @@ def is_rpar_token(nl: NL) -> TypeGuard[Leaf]:
return nl.type == token.RPAR
def is_string_token(nl: NL) -> TypeGuard[Leaf]:
return nl.type == token.STRING
def is_number_token(nl: NL) -> TypeGuard[Leaf]:
return nl.type == token.NUMBER
def get_annotation_type(leaf: Leaf) -> Literal["return", "param", None]:
"""Returns the type of annotation this leaf is part of, if any."""
def is_part_of_annotation(leaf: Leaf) -> bool:
"""Returns whether this leaf is part of type annotations."""
ancestor = leaf.parent
while ancestor is not None:
if ancestor.prev_sibling and ancestor.prev_sibling.type == token.RARROW:
return "return"
return True
if ancestor.parent and ancestor.parent.type == syms.tname:
return "param"
return True
ancestor = ancestor.parent
return None
def is_part_of_annotation(leaf: Leaf) -> bool:
"""Returns whether this leaf is part of a type annotation."""
assert leaf.parent is not None
return get_annotation_type(leaf) is not None
return False
def first_leaf(node: LN) -> Optional[Leaf]:
@ -1058,21 +963,3 @@ def furthest_ancestor_with_last_leaf(leaf: Leaf) -> LN:
while node.parent and node.parent.children and node is node.parent.children[-1]:
node = node.parent
return node
def has_sibling_with_type(node: LN, type: int) -> bool:
# Check previous siblings
sibling = node.prev_sibling
while sibling is not None:
if sibling.type == type:
return True
sibling = sibling.prev_sibling
# Check next siblings
sibling = node.next_sibling
while sibling is not None:
if sibling.type == type:
return True
sibling = sibling.next_sibling
return False

2
src/black/numerics.py
View File

@ -14,7 +14,7 @@ def format_hex(text: str) -> str:
def format_scientific_notation(text: str) -> str:
"""Formats a numeric string utilizing scientific notation"""
"""Formats a numeric string utilizing scentific notation"""
before, after = text.split("e")
sign = ""
if after.startswith("-"):

21
src/black/output.py
View File

@ -4,7 +4,6 @@
"""
import json
import re
import tempfile
from typing import Any, Optional
@ -56,28 +55,12 @@ def ipynb_diff(a: str, b: str, a_name: str, b_name: str) -> str:
return "".join(diff_lines)
_line_pattern = re.compile(r"(.*?(?:\r\n|\n|\r|$))")
def _splitlines_no_ff(source: str) -> list[str]:
"""Split a string into lines ignoring form feed and other chars.
This mimics how the Python parser splits source code.
A simplified version of the function with the same name in Lib/ast.py
"""
result = [match[0] for match in _line_pattern.finditer(source)]
if result[-1] == "":
result.pop(-1)
return result
def diff(a: str, b: str, a_name: str, b_name: str) -> str:
"""Return a unified diff string between strings `a` and `b`."""
import difflib
a_lines = _splitlines_no_ff(a)
b_lines = _splitlines_no_ff(b)
a_lines = a.splitlines(keepends=True)
b_lines = b.splitlines(keepends=True)
diff_lines = []
for line in difflib.unified_diff(
a_lines, b_lines, fromfile=a_name, tofile=b_name, n=5

88
src/black/parsing.py
View File

@ -4,8 +4,7 @@
import ast
import sys
import warnings
from collections.abc import Collection, Iterator
from typing import Iterable, Iterator, List, Set, Tuple
from black.mode import VERSION_TO_FEATURES, Feature, TargetVersion, supports_feature
from black.nodes import syms
@ -21,7 +20,7 @@ class InvalidInput(ValueError):
"""Raised when input source code fails all parse attempts."""
def get_grammars(target_versions: set[TargetVersion]) -> list[Grammar]:
def get_grammars(target_versions: Set[TargetVersion]) -> List[Grammar]:
if not target_versions:
# No target_version specified, so try all grammars.
return [
@ -52,20 +51,12 @@ def get_grammars(target_versions: set[TargetVersion]) -> list[Grammar]:
return grammars
def lib2to3_parse(
src_txt: str, target_versions: Collection[TargetVersion] = ()
) -> Node:
def lib2to3_parse(src_txt: str, target_versions: Iterable[TargetVersion] = ()) -> Node:
"""Given a string with source, return the lib2to3 Node."""
if not src_txt.endswith("\n"):
src_txt += "\n"
grammars = get_grammars(set(target_versions))
if target_versions:
max_tv = max(target_versions, key=lambda tv: tv.value)
tv_str = f" for target version {max_tv.pretty()}"
else:
tv_str = ""
errors = {}
for grammar in grammars:
drv = driver.Driver(grammar)
@ -81,14 +72,14 @@ def lib2to3_parse(
except IndexError:
faulty_line = "<line number missing in source>"
errors[grammar.version] = InvalidInput(
f"Cannot parse{tv_str}: {lineno}:{column}: {faulty_line}"
f"Cannot parse: {lineno}:{column}: {faulty_line}"
)
except TokenError as te:
# In edge cases these are raised; and typically don't have a "faulty_line".
lineno, column = te.args[1]
errors[grammar.version] = InvalidInput(
f"Cannot parse{tv_str}: {lineno}:{column}: {te.args[0]}"
f"Cannot parse: {lineno}:{column}: {te.args[0]}"
)
else:
@ -118,20 +109,13 @@ def lib2to3_unparse(node: Node) -> str:
return code
class ASTSafetyError(Exception):
"""Raised when Black's generated code is not equivalent to the old AST."""
def _parse_single_version(
src: str, version: tuple[int, int], *, type_comments: bool
def parse_single_version(
src: str, version: Tuple[int, int], *, type_comments: bool
) -> ast.AST:
filename = "<unknown>"
with warnings.catch_warnings():
warnings.simplefilter("ignore", SyntaxWarning)
warnings.simplefilter("ignore", DeprecationWarning)
return ast.parse(
src, filename, feature_version=version, type_comments=type_comments
)
return ast.parse(
src, filename, feature_version=version, type_comments=type_comments
)
def parse_ast(src: str) -> ast.AST:
@ -141,7 +125,7 @@ def parse_ast(src: str) -> ast.AST:
first_error = ""
for version in sorted(versions, reverse=True):
try:
return _parse_single_version(src, version, type_comments=True)
return parse_single_version(src, version, type_comments=True)
except SyntaxError as e:
if not first_error:
first_error = str(e)
@ -149,7 +133,7 @@ def parse_ast(src: str) -> ast.AST:
# Try to parse without type comments
for version in sorted(versions, reverse=True):
try:
return _parse_single_version(src, version, type_comments=False)
return parse_single_version(src, version, type_comments=False)
except SyntaxError:
pass
@ -159,27 +143,16 @@ def parse_ast(src: str) -> ast.AST:
def _normalize(lineend: str, value: str) -> str:
# To normalize, we strip any leading and trailing space from
# each line...
stripped: list[str] = [i.strip() for i in value.splitlines()]
stripped: List[str] = [i.strip() for i in value.splitlines()]
normalized = lineend.join(stripped)
# ...and remove any blank lines at the beginning and end of
# the whole string
return normalized.strip()
def stringify_ast(node: ast.AST) -> Iterator[str]:
def stringify_ast(node: ast.AST, depth: int = 0) -> Iterator[str]:
"""Simple visitor generating strings to compare ASTs by content."""
return _stringify_ast(node, [])
def _stringify_ast_with_new_parent(
node: ast.AST, parent_stack: list[ast.AST], new_parent: ast.AST
) -> Iterator[str]:
parent_stack.append(new_parent)
yield from _stringify_ast(node, parent_stack)
parent_stack.pop()
def _stringify_ast(node: ast.AST, parent_stack: list[ast.AST]) -> Iterator[str]:
if (
isinstance(node, ast.Constant)
and isinstance(node.value, str)
@ -190,7 +163,7 @@ def _stringify_ast(node: ast.AST, parent_stack: list[ast.AST]) -> Iterator[str]:
# over the kind
node.kind = None
yield f"{' ' * len(parent_stack)}{node.__class__.__name__}("
yield f"{' ' * depth}{node.__class__.__name__}("
for field in sorted(node._fields): # noqa: F402
# TypeIgnore has only one field 'lineno' which breaks this comparison
@ -202,7 +175,7 @@ def _stringify_ast(node: ast.AST, parent_stack: list[ast.AST]) -> Iterator[str]:
except AttributeError:
continue
yield f"{' ' * (len(parent_stack) + 1)}{field}="
yield f"{' ' * (depth+1)}{field}="
if isinstance(value, list):
for item in value:
@ -213,16 +186,14 @@ def _stringify_ast(node: ast.AST, parent_stack: list[ast.AST]) -> Iterator[str]:
and isinstance(node, ast.Delete)
and isinstance(item, ast.Tuple)
):
for elt in _unwrap_tuples(item):
yield from _stringify_ast_with_new_parent(
elt, parent_stack, node
)
for elt in item.elts:
yield from stringify_ast(elt, depth + 2)
elif isinstance(item, ast.AST):
yield from _stringify_ast_with_new_parent(item, parent_stack, node)
yield from stringify_ast(item, depth + 2)
elif isinstance(value, ast.AST):
yield from _stringify_ast_with_new_parent(value, parent_stack, node)
yield from stringify_ast(value, depth + 2)
else:
normalized: object
@ -230,10 +201,6 @@ def _stringify_ast(node: ast.AST, parent_stack: list[ast.AST]) -> Iterator[str]:
isinstance(node, ast.Constant)
and field == "value"
and isinstance(value, str)
and len(parent_stack) >= 2
# Any standalone string, ideally this would
# exactly match black.nodes.is_docstring
and isinstance(parent_stack[-1], ast.Expr)
):
# Constant strings may be indented across newlines, if they are
# docstrings; fold spaces after newlines when comparing. Similarly,
@ -244,17 +211,6 @@ def _stringify_ast(node: ast.AST, parent_stack: list[ast.AST]) -> Iterator[str]:
normalized = value.rstrip()
else:
normalized = value
yield (
f"{' ' * (len(parent_stack) + 1)}{normalized!r}, #"
f" {value.__class__.__name__}"
)
yield f"{' ' * (depth+2)}{normalized!r}, # {value.__class__.__name__}"
yield f"{' ' * len(parent_stack)}) # /{node.__class__.__name__}"
def _unwrap_tuples(node: ast.Tuple) -> Iterator[ast.AST]:
for elt in node.elts:
if isinstance(elt, ast.Tuple):
yield from _unwrap_tuples(elt)
else:
yield elt
yield f"{' ' * depth}) # /{node.__class__.__name__}"

64
src/black/ranges.py
View File

@ -1,9 +1,8 @@
"""Functions related to Black's formatting by line ranges feature."""
import difflib
from collections.abc import Collection, Iterator, Sequence
from dataclasses import dataclass
from typing import Union
from typing import Collection, Iterator, List, Sequence, Set, Tuple, Union
from black.nodes import (
LN,
@ -19,8 +18,8 @@
from blib2to3.pgen2.token import ASYNC, NEWLINE
def parse_line_ranges(line_ranges: Sequence[str]) -> list[tuple[int, int]]:
lines: list[tuple[int, int]] = []
def parse_line_ranges(line_ranges: Sequence[str]) -> List[Tuple[int, int]]:
lines: List[Tuple[int, int]] = []
for lines_str in line_ranges:
parts = lines_str.split("-")
if len(parts) != 2:
@ -41,44 +40,16 @@ def parse_line_ranges(line_ranges: Sequence[str]) -> list[tuple[int, int]]:
return lines
def is_valid_line_range(lines: tuple[int, int]) -> bool:
def is_valid_line_range(lines: Tuple[int, int]) -> bool:
"""Returns whether the line range is valid."""
return not lines or lines[0] <= lines[1]
def sanitized_lines(
lines: Collection[tuple[int, int]], src_contents: str
) -> Collection[tuple[int, int]]:
"""Returns the valid line ranges for the given source.
This removes ranges that are entirely outside the valid lines.
Other ranges are normalized so that the start values are at least 1 and the
end values are at most the (1-based) index of the last source line.
"""
if not src_contents:
return []
good_lines = []
src_line_count = src_contents.count("\n")
if not src_contents.endswith("\n"):
src_line_count += 1
for start, end in lines:
if start > src_line_count:
continue
# line-ranges are 1-based
start = max(start, 1)
if end < start:
continue
end = min(end, src_line_count)
good_lines.append((start, end))
return good_lines
def adjusted_lines(
lines: Collection[tuple[int, int]],
lines: Collection[Tuple[int, int]],
original_source: str,
modified_source: str,
) -> list[tuple[int, int]]:
) -> List[Tuple[int, int]]:
"""Returns the adjusted line ranges based on edits from the original code.
This computes the new line ranges by diffing original_source and
@ -154,7 +125,7 @@ def adjusted_lines(
return new_lines
def convert_unchanged_lines(src_node: Node, lines: Collection[tuple[int, int]]) -> None:
def convert_unchanged_lines(src_node: Node, lines: Collection[Tuple[int, int]]) -> None:
"""Converts unchanged lines to STANDALONE_COMMENT.
The idea is similar to how `# fmt: on/off` is implemented. It also converts the
@ -178,7 +149,7 @@ def convert_unchanged_lines(src_node: Node, lines: Collection[tuple[int, int]])
more formatting to pass (1). However, it's hard to get it correct when
incorrect indentations are used. So we defer this to future optimizations.
"""
lines_set: set[int] = set()
lines_set: Set[int] = set()
for start, end in lines:
lines_set.update(range(start, end + 1))
visitor = _TopLevelStatementsVisitor(lines_set)
@ -201,12 +172,12 @@ class _TopLevelStatementsVisitor(Visitor[None]):
A node visitor that converts unchanged top-level statements to
STANDALONE_COMMENT.
This is used in addition to _convert_unchanged_line_by_line, to
This is used in addition to _convert_unchanged_lines_by_flatterning, to
speed up formatting when there are unchanged top-level
classes/functions/statements.
"""
def __init__(self, lines_set: set[int]):
def __init__(self, lines_set: Set[int]):
self._lines_set = lines_set
def visit_simple_stmt(self, node: Node) -> Iterator[None]:
@ -250,7 +221,7 @@ def visit_suite(self, node: Node) -> Iterator[None]:
_convert_node_to_standalone_comment(semantic_parent)
def _convert_unchanged_line_by_line(node: Node, lines_set: set[int]) -> None:
def _convert_unchanged_line_by_line(node: Node, lines_set: Set[int]) -> None:
"""Converts unchanged to STANDALONE_COMMENT line by line."""
for leaf in node.leaves():
if leaf.type != NEWLINE:
@ -262,7 +233,7 @@ def _convert_unchanged_line_by_line(node: Node, lines_set: set[int]) -> None:
# match_stmt: "match" subject_expr ':' NEWLINE INDENT case_block+ DEDENT
# Here we need to check `subject_expr`. The `case_block+` will be
# checked by their own NEWLINEs.
nodes_to_ignore: list[LN] = []
nodes_to_ignore: List[LN] = []
prev_sibling = leaf.prev_sibling
while prev_sibling:
nodes_to_ignore.insert(0, prev_sibling)
@ -331,7 +302,7 @@ def _convert_node_to_standalone_comment(node: LN) -> None:
index = node.remove()
if index is not None:
# Remove the '\n', as STANDALONE_COMMENT will have '\n' appended when
# generating the formatted code.
# genearting the formatted code.
value = str(node)[:-1]
parent.insert_child(
index,
@ -383,7 +354,7 @@ def _leaf_line_end(leaf: Leaf) -> int:
return leaf.lineno + str(leaf).count("\n")
def _get_line_range(node_or_nodes: Union[LN, list[LN]]) -> set[int]:
def _get_line_range(node_or_nodes: Union[LN, List[LN]]) -> Set[int]:
"""Returns the line range of this node or list of nodes."""
if isinstance(node_or_nodes, list):
nodes = node_or_nodes
@ -464,7 +435,7 @@ def _calculate_lines_mappings(
modified_source.splitlines(keepends=True),
)
matching_blocks = matcher.get_matching_blocks()
lines_mappings: list[_LinesMapping] = []
lines_mappings: List[_LinesMapping] = []
# matching_blocks is a sequence of "same block of code ranges", see
# https://docs.python.org/3/library/difflib.html#difflib.SequenceMatcher.get_matching_blocks
# Each block corresponds to a _LinesMapping with is_changed_block=False,
@ -516,7 +487,10 @@ def _find_lines_mapping_index(
index = start_index
while index < len(lines_mappings):
mapping = lines_mappings[index]
if mapping.original_start <= original_line <= mapping.original_end:
if (
mapping.original_start <= original_line
and original_line <= mapping.original_end
):
return index
index += 1
return index

0
src/black/resources/__init__.py
View File

148
src/black/resources/black.schema.json
View File

@ -1,148 +0,0 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://github.com/psf/black/blob/main/src/black/resources/black.schema.json",
"$comment": "tool.black table in pyproject.toml",
"type": "object",
"additionalProperties": false,
"properties": {
"code": {
"type": "string",
"description": "Format the code passed in as a string."
},
"line-length": {
"type": "integer",
"description": "How many characters per line to allow.",
"default": 88
},
"target-version": {
"type": "array",
"items": {
"enum": [
"py33",
"py34",
"py35",
"py36",
"py37",
"py38",
"py39",
"py310",
"py311",
"py312",
"py313"
]
},
"description": "Python versions that should be supported by Black's output. You should include all versions that your code supports. By default, Black will infer target versions from the project metadata in pyproject.toml. If this does not yield conclusive results, Black will use per-file auto-detection."
},
"pyi": {
"type": "boolean",
"description": "Format all input files like typing stubs regardless of file extension. This is useful when piping source on standard input.",
"default": false
},
"ipynb": {
"type": "boolean",
"description": "Format all input files like Jupyter Notebooks regardless of file extension. This is useful when piping source on standard input.",
"default": false
},
"python-cell-magics": {
"type": "array",
"items": {
"type": "string"
},
"description": "When processing Jupyter Notebooks, add the given magic to the list of known python-magics (capture, prun, pypy, python, python3, time, timeit). Useful for formatting cells with custom python magics."
},
"skip-source-first-line": {
"type": "boolean",
"description": "Skip the first line of the source code.",
"default": false
},
"skip-string-normalization": {
"type": "boolean",
"description": "Don't normalize string quotes or prefixes.",
"default": false
},
"skip-magic-trailing-comma": {
"type": "boolean",
"description": "Don't use trailing commas as a reason to split lines.",
"default": false
},
"preview": {
"type": "boolean",
"description": "Enable potentially disruptive style changes that may be added to Black's main functionality in the next major release.",
"default": false
},
"unstable": {
"type": "boolean",
"description": "Enable potentially disruptive style changes that have known bugs or are not currently expected to make it into the stable style Black's next major release. Implies --preview.",
"default": false
},
"enable-unstable-feature": {
"type": "array",
"items": {
"enum": [
"string_processing",
"hug_parens_with_braces_and_square_brackets",
"wrap_long_dict_values_in_parens",
"multiline_string_handling",
"always_one_newline_after_import",
"fix_fmt_skip_in_one_liners"
]
},
"description": "Enable specific features included in the `--unstable` style. Requires `--preview`. No compatibility guarantees are provided on the behavior or existence of any unstable features."
},
"check": {
"type": "boolean",
"description": "Don't write the files back, just return the status. Return code 0 means nothing would change. Return code 1 means some files would be reformatted. Return code 123 means there was an internal error.",
"default": false
},
"diff": {
"type": "boolean",
"description": "Don't write the files back, just output a diff to indicate what changes Black would've made. They are printed to stdout so capturing them is simple.",
"default": false
},
"color": {
"type": "boolean",
"description": "Show (or do not show) colored diff. Only applies when --diff is given.",
"default": false
},
"fast": {
"type": "boolean",
"description": "By default, Black performs an AST safety check after formatting your code. The --fast flag turns off this check and the --safe flag explicitly enables it. [default: --safe]",
"default": false
},
"required-version": {
"type": "string",
"description": "Require a specific version of Black to be running. This is useful for ensuring that all contributors to your project are using the same version, because different versions of Black may format code a little differently. This option can be set in a configuration file for consistent results across environments."
},
"exclude": {
"type": "string",
"description": "A regular expression that matches files and directories that should be excluded on recursive searches. An empty value means no paths are excluded. Use forward slashes for directories on all platforms (Windows, too). By default, Black also ignores all paths listed in .gitignore. Changing this value will override all default exclusions. [default: /(\\.direnv|\\.eggs|\\.git|\\.hg|\\.ipynb_checkpoints|\\.mypy_cache|\\.nox|\\.pytest_cache|\\.ruff_cache|\\.tox|\\.svn|\\.venv|\\.vscode|__pypackages__|_build|buck-out|build|dist|venv)/]"
},
"extend-exclude": {
"type": "string",
"description": "Like --exclude, but adds additional files and directories on top of the default values instead of overriding them."
},
"force-exclude": {
"type": "string",
"description": "Like --exclude, but files and directories matching this regex will be excluded even when they are passed explicitly as arguments. This is useful when invoking Black programmatically on changed files, such as in a pre-commit hook or editor plugin."
},
"include": {
"type": "string",
"description": "A regular expression that matches files and directories that should be included on recursive searches. An empty value means all files are included regardless of the name. Use forward slashes for directories on all platforms (Windows, too). Overrides all exclusions, including from .gitignore and command line options.",
"default": "(\\.pyi?|\\.ipynb)$"
},
"workers": {
"type": "integer",
"description": "When Black formats multiple files, it may use a process pool to speed up formatting. This option controls the number of parallel workers. This can also be specified via the BLACK_NUM_WORKERS environment variable. Defaults to the number of CPUs in the system."
},
"quiet": {
"type": "boolean",
"description": "Stop emitting all non-critical output. Error messages will still be emitted (which can silenced by 2>/dev/null).",
"default": false
},
"verbose": {
"type": "boolean",
"description": "Emit messages about files that were not changed or were ignored due to exclusion patterns. If Black is using a configuration file, a message detailing which one it is using will be emitted.",
"default": false
}
}
}

15
src/black/schema.py
View File

@ -1,15 +0,0 @@
import importlib.resources
import json
from typing import Any
def get_schema(tool_name: str = "black") -> Any:
"""Get the stored complete schema for black's settings."""
assert tool_name == "black", "Only black is supported."
pkg = "black.resources"
fname = "black.schema.json"
schema = importlib.resources.files(pkg).joinpath(fname)
with schema.open(encoding="utf-8") as f:
return json.load(f)

102
src/black/strings.py
View File

@ -5,8 +5,7 @@
import re
import sys
from functools import lru_cache
from re import Match, Pattern
from typing import Final
from typing import Final, List, Match, Pattern
from black._width_table import WIDTH_TABLE
from blib2to3.pytree import Leaf
@ -15,6 +14,7 @@
STRING_PREFIX_RE: Final = re.compile(
r"^([" + STRING_PREFIX_CHARS + r"]*)(.*)$", re.DOTALL
)
FIRST_NON_WHITESPACE_RE: Final = re.compile(r"\s*\t+\s*(\S)")
UNICODE_ESCAPE_RE: Final = re.compile(
r"(?P<backslashes>\\+)(?P<body>"
r"(u(?P<u>[a-fA-F0-9]{4}))" # Character with 16-bit hex value xxxx
@ -44,28 +44,32 @@ def has_triple_quotes(string: str) -> bool:
return raw_string[:3] in {'"""', "'''"}
def lines_with_leading_tabs_expanded(s: str) -> list[str]:
def lines_with_leading_tabs_expanded(s: str) -> List[str]:
"""
Splits string into lines and expands only leading tabs (following the normal
Python rules)
"""
lines = []
for line in s.splitlines():
stripped_line = line.lstrip()
if not stripped_line or stripped_line == line:
lines.append(line)
# Find the index of the first non-whitespace character after a string of
# whitespace that includes at least one tab
match = FIRST_NON_WHITESPACE_RE.match(line)
if match:
first_non_whitespace_idx = match.start(1)
lines.append(
line[:first_non_whitespace_idx].expandtabs()
+ line[first_non_whitespace_idx:]
)
else:
prefix_length = len(line) - len(stripped_line)
prefix = line[:prefix_length].expandtabs()
lines.append(prefix + stripped_line)
if s.endswith("\n"):
lines.append("")
lines.append(line)
return lines
def fix_multiline_docstring(docstring: str, prefix: str) -> str:
def fix_docstring(docstring: str, prefix: str) -> str:
# https://www.python.org/dev/peps/pep-0257/#handling-docstring-indentation
assert docstring, "INTERNAL ERROR: Multiline docstrings cannot be empty"
if not docstring:
return ""
lines = lines_with_leading_tabs_expanded(docstring)
# Determine minimum indentation (first line doesn't count):
indent = sys.maxsize
@ -169,7 +173,8 @@ def _cached_compile(pattern: str) -> Pattern[str]:
def normalize_string_quotes(s: str) -> str:
"""Prefer double quotes but only if it doesn't cause more escaping.
Adds or removes backslashes as appropriate.
Adds or removes backslashes as appropriate. Doesn't parse and fix
strings nested in f-strings.
"""
value = s.lstrip(STRING_PREFIX_CHARS)
if value[:3] == '"""':
@ -185,7 +190,8 @@ def normalize_string_quotes(s: str) -> str:
orig_quote = "'"
new_quote = '"'
first_quote_pos = s.find(orig_quote)
assert first_quote_pos != -1, f"INTERNAL ERROR: Malformed string {s!r}"
if first_quote_pos == -1:
return s # There's an internal error
prefix = s[:first_quote_pos]
unescaped_new_quote = _cached_compile(rf"(([^\\]|^)(\\\\)*){new_quote}")
@ -209,7 +215,6 @@ def normalize_string_quotes(s: str) -> str:
s = f"{prefix}{orig_quote}{body}{orig_quote}"
new_body = sub_twice(escaped_orig_quote, rf"\1\2{orig_quote}", new_body)
new_body = sub_twice(unescaped_new_quote, rf"\1\\{new_quote}", new_body)
if "f" in prefix.casefold():
matches = re.findall(
r"""
@ -239,71 +244,6 @@ def normalize_string_quotes(s: str) -> str:
return f"{prefix}{new_quote}{new_body}{new_quote}"
def normalize_fstring_quotes(
quote: str,
middles: list[Leaf],
is_raw_fstring: bool,
) -> tuple[list[Leaf], str]:
"""Prefer double quotes but only if it doesn't cause more escaping.
Adds or removes backslashes as appropriate.
"""
if quote == '"""':
return middles, quote
elif quote == "'''":
new_quote = '"""'
elif quote == '"':
new_quote = "'"
else:
new_quote = '"'
unescaped_new_quote = _cached_compile(rf"(([^\\]|^)(\\\\)*){new_quote}")
escaped_new_quote = _cached_compile(rf"([^\\]|^)\\((?:\\\\)*){new_quote}")
escaped_orig_quote = _cached_compile(rf"([^\\]|^)\\((?:\\\\)*){quote}")
if is_raw_fstring:
for middle in middles:
if unescaped_new_quote.search(middle.value):
# There's at least one unescaped new_quote in this raw string
# so converting is impossible
return middles, quote
# Do not introduce or remove backslashes in raw strings, just use double quote
return middles, '"'
new_segments = []
for middle in middles:
segment = middle.value
# remove unnecessary escapes
new_segment = sub_twice(escaped_new_quote, rf"\1\2{new_quote}", segment)
if segment != new_segment:
# Consider the string without unnecessary escapes as the original
middle.value = new_segment
new_segment = sub_twice(escaped_orig_quote, rf"\1\2{quote}", new_segment)
new_segment = sub_twice(unescaped_new_quote, rf"\1\\{new_quote}", new_segment)
new_segments.append(new_segment)
if new_quote == '"""' and new_segments[-1].endswith('"'):
# edge case:
new_segments[-1] = new_segments[-1][:-1] + '\\"'
for middle, new_segment in zip(middles, new_segments):
orig_escape_count = middle.value.count("\\")
new_escape_count = new_segment.count("\\")
if new_escape_count > orig_escape_count:
return middles, quote # Do not introduce more escaping
if new_escape_count == orig_escape_count and quote == '"':
return middles, quote # Prefer double quotes
for middle, new_segment in zip(middles, new_segments):
middle.value = new_segment
return middles, new_quote
def normalize_unicode_escape_sequences(leaf: Leaf) -> None:
"""Replace hex codes in Unicode escape sequences with lowercase representation."""
text = leaf.value

250
src/black/trans.py
View File

@ -5,9 +5,25 @@
import re
from abc import ABC, abstractmethod
from collections import defaultdict
from collections.abc import Callable, Collection, Iterable, Iterator, Sequence
from dataclasses import dataclass
from typing import Any, ClassVar, Final, Literal, Optional, TypeVar, Union
from typing import (
Any,
Callable,
ClassVar,
Collection,
Dict,
Final,
Iterable,
Iterator,
List,
Literal,
Optional,
Sequence,
Set,
Tuple,
TypeVar,
Union,
)
from mypy_extensions import trait
@ -52,7 +68,7 @@ class CannotTransform(Exception):
ParserState = int
StringID = int
TResult = Result[T, CannotTransform] # (T)ransform Result
TMatchResult = TResult[list[Index]]
TMatchResult = TResult[List[Index]]
SPLIT_SAFE_CHARS = frozenset(["\u3001", "\u3002", "\uff0c"]) # East Asian stops
@ -78,30 +94,43 @@ def hug_power_op(
else:
raise CannotTransform("No doublestar token was found in the line.")
def is_simple_lookup(index: int, kind: Literal[1, -1]) -> bool:
def is_simple_lookup(index: int, step: Literal[1, -1]) -> bool:
# Brackets and parentheses indicate calls, subscripts, etc. ...
# basically stuff that doesn't count as "simple". Only a NAME lookup
# or dotted lookup (eg. NAME.NAME) is OK.
if kind == -1:
return handle_is_simple_look_up_prev(line, index, {token.RPAR, token.RSQB})
if step == -1:
disallowed = {token.RPAR, token.RSQB}
else:
return handle_is_simple_lookup_forward(
line, index, {token.LPAR, token.LSQB}
)
disallowed = {token.LPAR, token.LSQB}
def is_simple_operand(index: int, kind: Literal[1, -1]) -> bool:
while 0 <= index < len(line.leaves):
current = line.leaves[index]
if current.type in disallowed:
return False
if current.type not in {token.NAME, token.DOT} or current.value == "for":
# If the current token isn't disallowed, we'll assume this is simple as
# only the disallowed tokens are semantically attached to this lookup
# expression we're checking. Also, stop early if we hit the 'for' bit
# of a comprehension.
return True
index += step
return True
def is_simple_operand(index: int, kind: Literal["base", "exponent"]) -> bool:
# An operand is considered "simple" if's a NAME, a numeric CONSTANT, a simple
# lookup (see above), with or without a preceding unary operator.
start = line.leaves[index]
if start.type in {token.NAME, token.NUMBER}:
return is_simple_lookup(index, kind)
return is_simple_lookup(index, step=(1 if kind == "exponent" else -1))
if start.type in {token.PLUS, token.MINUS, token.TILDE}:
if line.leaves[index + 1].type in {token.NAME, token.NUMBER}:
# kind is always one as bases with a preceding unary op will be checked
# step is always one as bases with a preceding unary op will be checked
# for simplicity starting from the next token (so it'll hit the check
# above).
return is_simple_lookup(index + 1, kind=1)
return is_simple_lookup(index + 1, step=1)
return False
@ -116,9 +145,9 @@ def is_simple_operand(index: int, kind: Literal[1, -1]) -> bool:
should_hug = (
(0 < idx < len(line.leaves) - 1)
and leaf.type == token.DOUBLESTAR
and is_simple_operand(idx - 1, kind=-1)
and is_simple_operand(idx - 1, kind="base")
and line.leaves[idx - 1].value != "lambda"
and is_simple_operand(idx + 1, kind=1)
and is_simple_operand(idx + 1, kind="exponent")
)
if should_hug:
new_leaf.prefix = ""
@ -133,75 +162,6 @@ def is_simple_operand(index: int, kind: Literal[1, -1]) -> bool:
yield new_line
def handle_is_simple_look_up_prev(line: Line, index: int, disallowed: set[int]) -> bool:
"""
Handling the determination of is_simple_lookup for the lines prior to the doublestar
token. This is required because of the need to isolate the chained expression
to determine the bracket or parenthesis belong to the single expression.
"""
contains_disallowed = False
chain = []
while 0 <= index < len(line.leaves):
current = line.leaves[index]
chain.append(current)
if not contains_disallowed and current.type in disallowed:
contains_disallowed = True
if not is_expression_chained(chain):
return not contains_disallowed
index -= 1
return True
def handle_is_simple_lookup_forward(
line: Line, index: int, disallowed: set[int]
) -> bool:
"""
Handling decision is_simple_lookup for the lines behind the doublestar token.
This function is simplified to keep consistent with the prior logic and the forward
case are more straightforward and do not need to care about chained expressions.
"""
while 0 <= index < len(line.leaves):
current = line.leaves[index]
if current.type in disallowed:
return False
if current.type not in {token.NAME, token.DOT} or (
current.type == token.NAME and current.value == "for"
):
# If the current token isn't disallowed, we'll assume this is simple as
# only the disallowed tokens are semantically attached to this lookup
# expression we're checking. Also, stop early if we hit the 'for' bit
# of a comprehension.
return True
index += 1
return True
def is_expression_chained(chained_leaves: list[Leaf]) -> bool:
"""
Function to determine if the variable is a chained call.
(e.g., foo.lookup, foo().lookup, (foo.lookup())) will be recognized as chained call)
"""
if len(chained_leaves) < 2:
return True
current_leaf = chained_leaves[-1]
past_leaf = chained_leaves[-2]
if past_leaf.type == token.NAME:
return current_leaf.type in {token.DOT}
elif past_leaf.type in {token.RPAR, token.RSQB}:
return current_leaf.type in {token.RSQB, token.RPAR}
elif past_leaf.type in {token.LPAR, token.LSQB}:
return current_leaf.type in {token.NAME, token.LPAR, token.LSQB}
else:
return False
class StringTransformer(ABC):
"""
An implementation of the Transformer protocol that relies on its
@ -252,7 +212,7 @@ def do_match(self, line: Line) -> TMatchResult:
@abstractmethod
def do_transform(
self, line: Line, string_indices: list[int]
self, line: Line, string_indices: List[int]
) -> Iterator[TResult[Line]]:
"""
Yields:
@ -342,8 +302,8 @@ class CustomSplitMapMixin:
the resultant substrings go over the configured max line length.
"""
_Key: ClassVar = tuple[StringID, str]
_CUSTOM_SPLIT_MAP: ClassVar[dict[_Key, tuple[CustomSplit, ...]]] = defaultdict(
_Key: ClassVar = Tuple[StringID, str]
_CUSTOM_SPLIT_MAP: ClassVar[Dict[_Key, Tuple[CustomSplit, ...]]] = defaultdict(
tuple
)
@ -367,7 +327,7 @@ def add_custom_splits(
key = self._get_key(string)
self._CUSTOM_SPLIT_MAP[key] = tuple(custom_splits)
def pop_custom_splits(self, string: str) -> list[CustomSplit]:
def pop_custom_splits(self, string: str) -> List[CustomSplit]:
"""Custom Split Map Getter Method
Returns:
@ -442,7 +402,7 @@ def do_match(self, line: Line) -> TMatchResult:
break
i += 1
if not contains_comment and not is_part_of_annotation(leaf):
if not is_part_of_annotation(leaf) and not contains_comment:
string_indices.append(idx)
# Advance to the next non-STRING leaf.
@ -466,7 +426,7 @@ def do_match(self, line: Line) -> TMatchResult:
return TErr("This line has no strings that need merging.")
def do_transform(
self, line: Line, string_indices: list[int]
self, line: Line, string_indices: List[int]
) -> Iterator[TResult[Line]]:
new_line = line
@ -497,7 +457,7 @@ def do_transform(
@staticmethod
def _remove_backslash_line_continuation_chars(
line: Line, string_indices: list[int]
line: Line, string_indices: List[int]
) -> TResult[Line]:
"""
Merge strings that were split across multiple lines using
@ -538,7 +498,7 @@ def _remove_backslash_line_continuation_chars(
return Ok(new_line)
def _merge_string_group(
self, line: Line, string_indices: list[int]
self, line: Line, string_indices: List[int]
) -> TResult[Line]:
"""
Merges string groups (i.e. set of adjacent strings).
@ -557,7 +517,7 @@ def _merge_string_group(
is_valid_index = is_valid_index_factory(LL)
# A dict of {string_idx: tuple[num_of_strings, string_leaf]}.
merged_string_idx_dict: dict[int, tuple[int, Leaf]] = {}
merged_string_idx_dict: Dict[int, Tuple[int, Leaf]] = {}
for string_idx in string_indices:
vresult = self._validate_msg(line, string_idx)
if isinstance(vresult, Err):
@ -593,8 +553,8 @@ def _merge_string_group(
return Ok(new_line)
def _merge_one_string_group(
self, LL: list[Leaf], string_idx: int, is_valid_index: Callable[[int], bool]
) -> tuple[int, Leaf]:
self, LL: List[Leaf], string_idx: int, is_valid_index: Callable[[int], bool]
) -> Tuple[int, Leaf]:
"""
Merges one string group where the first string in the group is
`LL[string_idx]`.
@ -630,10 +590,10 @@ def make_naked(string: str, string_prefix: str) -> str:
"""
assert_is_leaf_string(string)
if "f" in string_prefix:
f_expressions = [
f_expressions = (
string[span[0] + 1 : span[1] - 1] # +-1 to get rid of curly braces
for span in iter_fexpr_spans(string)
]
)
debug_expressions_contain_visible_quotes = any(
re.search(r".*[\'\"].*(?<![!:=])={1}(?!=)(?![^\s:])", expression)
for expression in f_expressions
@ -764,8 +724,6 @@ def _validate_msg(line: Line, string_idx: int) -> TResult[None]:
- The set of all string prefixes in the string group is of
length greater than one and is not equal to {"", "f"}.
- The string group consists of raw strings.
- The string group would merge f-strings with different quote types
and internal quotes.
- The string group is stringified type annotations. We don't want to
process stringified type annotations since pyright doesn't support
them spanning multiple string values. (NOTE: mypy, pytype, pyre do
@ -792,8 +750,6 @@ def _validate_msg(line: Line, string_idx: int) -> TResult[None]:
i += inc
QUOTE = line.leaves[string_idx].value[-1]
num_of_inline_string_comments = 0
set_of_prefixes = set()
num_of_strings = 0
@ -816,19 +772,6 @@ def _validate_msg(line: Line, string_idx: int) -> TResult[None]:
set_of_prefixes.add(prefix)
if (
"f" in prefix
and leaf.value[-1] != QUOTE
and (
"'" in leaf.value[len(prefix) + 1 : -1]
or '"' in leaf.value[len(prefix) + 1 : -1]
)
):
return TErr(
"StringMerger does NOT merge f-strings with different quote types"
" and internal quotes."
)
if id(leaf) in line.comments:
num_of_inline_string_comments += 1
if contains_pragma_comment(line.comments[id(leaf)]):
@ -857,7 +800,6 @@ class StringParenStripper(StringTransformer):
The line contains a string which is surrounded by parentheses and:
- The target string is NOT the only argument to a function call.
- The target string is NOT a "pointless" string.
- The target string is NOT a dictionary value.
- If the target string contains a PERCENT, the brackets are not
preceded or followed by an operator with higher precedence than
PERCENT.
@ -905,14 +847,11 @@ def do_match(self, line: Line) -> TMatchResult:
):
continue
# That LPAR should NOT be preceded by a colon (which could be a
# dictionary value), function name, or a closing bracket (which
# could be a function returning a function or a list/dictionary
# containing a function)...
# That LPAR should NOT be preceded by a function name or a closing
# bracket (which could be a function which returns a function or a
# list/dictionary that contains a function)...
if is_valid_index(idx - 2) and (
LL[idx - 2].type == token.COLON
or LL[idx - 2].type == token.NAME
or LL[idx - 2].type in CLOSING_BRACKETS
LL[idx - 2].type == token.NAME or LL[idx - 2].type in CLOSING_BRACKETS
):
continue
@ -979,11 +918,11 @@ def do_match(self, line: Line) -> TMatchResult:
return TErr("This line has no strings wrapped in parens.")
def do_transform(
self, line: Line, string_indices: list[int]
self, line: Line, string_indices: List[int]
) -> Iterator[TResult[Line]]:
LL = line.leaves
string_and_rpar_indices: list[int] = []
string_and_rpar_indices: List[int] = []
for string_idx in string_indices:
string_parser = StringParser()
rpar_idx = string_parser.parse(LL, string_idx)
@ -1006,7 +945,7 @@ def do_transform(
)
def _transform_to_new_line(
self, line: Line, string_and_rpar_indices: list[int]
self, line: Line, string_and_rpar_indices: List[int]
) -> Line:
LL = line.leaves
@ -1259,7 +1198,7 @@ def _get_max_string_length(self, line: Line, string_idx: int) -> int:
return max_string_length
@staticmethod
def _prefer_paren_wrap_match(LL: list[Leaf]) -> Optional[int]:
def _prefer_paren_wrap_match(LL: List[Leaf]) -> Optional[int]:
"""
Returns:
string_idx such that @LL[string_idx] is equal to our target (i.e.
@ -1304,14 +1243,14 @@ def _prefer_paren_wrap_match(LL: list[Leaf]) -> Optional[int]:
return None
def iter_fexpr_spans(s: str) -> Iterator[tuple[int, int]]:
def iter_fexpr_spans(s: str) -> Iterator[Tuple[int, int]]:
"""
Yields spans corresponding to expressions in a given f-string.
Spans are half-open ranges (left inclusive, right exclusive).
Assumes the input string is a valid f-string, but will not crash if the input
string is invalid.
"""
stack: list[int] = [] # our curly paren stack
stack: List[int] = [] # our curly paren stack
i = 0
while i < len(s):
if s[i] == "{":
@ -1334,7 +1273,7 @@ def iter_fexpr_spans(s: str) -> Iterator[tuple[int, int]]:
i += 1
continue
# if we're in an expression part of the f-string, fast-forward through strings
# if we're in an expression part of the f-string, fast forward through strings
# note that backslashes are not legal in the expression portion of f-strings
if stack:
delim = None
@ -1474,7 +1413,7 @@ def do_splitter_match(self, line: Line) -> TMatchResult:
return Ok([string_idx])
def do_transform(
self, line: Line, string_indices: list[int]
self, line: Line, string_indices: List[int]
) -> Iterator[TResult[Line]]:
LL = line.leaves
assert len(string_indices) == 1, (
@ -1576,7 +1515,7 @@ def more_splits_should_be_made() -> bool:
else:
return str_width(rest_value) > max_last_string_column()
string_line_results: list[Ok[Line]] = []
string_line_results: List[Ok[Line]] = []
while more_splits_should_be_made():
if use_custom_breakpoints:
# Custom User Split (manual)
@ -1705,7 +1644,7 @@ def more_splits_should_be_made() -> bool:
last_line.comments = line.comments.copy()
yield Ok(last_line)
def _iter_nameescape_slices(self, string: str) -> Iterator[tuple[Index, Index]]:
def _iter_nameescape_slices(self, string: str) -> Iterator[Tuple[Index, Index]]:
"""
Yields:
All ranges of @string which, if @string were to be split there,
@ -1736,7 +1675,7 @@ def _iter_nameescape_slices(self, string: str) -> Iterator[tuple[Index, Index]]:
raise RuntimeError(f"{self.__class__.__name__} LOGIC ERROR!")
yield begin, end
def _iter_fexpr_slices(self, string: str) -> Iterator[tuple[Index, Index]]:
def _iter_fexpr_slices(self, string: str) -> Iterator[Tuple[Index, Index]]:
"""
Yields:
All ranges of @string which, if @string were to be split there,
@ -1747,8 +1686,8 @@ def _iter_fexpr_slices(self, string: str) -> Iterator[tuple[Index, Index]]:
return
yield from iter_fexpr_spans(string)
def _get_illegal_split_indices(self, string: str) -> set[Index]:
illegal_indices: set[Index] = set()
def _get_illegal_split_indices(self, string: str) -> Set[Index]:
illegal_indices: Set[Index] = set()
iterators = [
self._iter_fexpr_slices(string),
self._iter_nameescape_slices(string),
@ -1801,7 +1740,7 @@ def passes_all_checks(i: Index) -> bool:
"""
Returns:
True iff ALL of the conditions listed in the 'Transformations'
section of this classes' docstring would be met by returning @i.
section of this classes' docstring would be be met by returning @i.
"""
is_space = string[i] == " "
is_split_safe = is_valid_index(i - 1) and string[i - 1] in SPLIT_SAFE_CHARS
@ -1874,7 +1813,7 @@ def _normalize_f_string(self, string: str, prefix: str) -> str:
else:
return string
def _get_string_operator_leaves(self, leaves: Iterable[Leaf]) -> list[Leaf]:
def _get_string_operator_leaves(self, leaves: Iterable[Leaf]) -> List[Leaf]:
LL = list(leaves)
string_op_leaves = []
@ -1983,7 +1922,7 @@ def do_splitter_match(self, line: Line) -> TMatchResult:
return TErr("This line does not contain any non-atomic strings.")
@staticmethod
def _return_match(LL: list[Leaf]) -> Optional[int]:
def _return_match(LL: List[Leaf]) -> Optional[int]:
"""
Returns:
string_idx such that @LL[string_idx] is equal to our target (i.e.
@ -1993,7 +1932,7 @@ def _return_match(LL: list[Leaf]) -> Optional[int]:
OR
None, otherwise.
"""
# If this line is a part of a return/yield statement and the first leaf
# If this line is apart of a return/yield statement and the first leaf
# contains either the "return" or "yield" keywords...
if parent_type(LL[0]) in [syms.return_stmt, syms.yield_expr] and LL[
0
@ -2008,7 +1947,7 @@ def _return_match(LL: list[Leaf]) -> Optional[int]:
return None
@staticmethod
def _else_match(LL: list[Leaf]) -> Optional[int]:
def _else_match(LL: List[Leaf]) -> Optional[int]:
"""
Returns:
string_idx such that @LL[string_idx] is equal to our target (i.e.
@ -2018,7 +1957,7 @@ def _else_match(LL: list[Leaf]) -> Optional[int]:
OR
None, otherwise.
"""
# If this line is a part of a ternary expression and the first leaf
# If this line is apart of a ternary expression and the first leaf
# contains the "else" keyword...
if (
parent_type(LL[0]) == syms.test
@ -2035,7 +1974,7 @@ def _else_match(LL: list[Leaf]) -> Optional[int]:
return None
@staticmethod
def _assert_match(LL: list[Leaf]) -> Optional[int]:
def _assert_match(LL: List[Leaf]) -> Optional[int]:
"""
Returns:
string_idx such that @LL[string_idx] is equal to our target (i.e.
@ -2045,7 +1984,7 @@ def _assert_match(LL: list[Leaf]) -> Optional[int]:
OR
None, otherwise.
"""
# If this line is a part of an assert statement and the first leaf
# If this line is apart of an assert statement and the first leaf
# contains the "assert" keyword...
if parent_type(LL[0]) == syms.assert_stmt and LL[0].value == "assert":
is_valid_index = is_valid_index_factory(LL)
@ -2070,7 +2009,7 @@ def _assert_match(LL: list[Leaf]) -> Optional[int]:
return None
@staticmethod
def _assign_match(LL: list[Leaf]) -> Optional[int]:
def _assign_match(LL: List[Leaf]) -> Optional[int]:
"""
Returns:
string_idx such that @LL[string_idx] is equal to our target (i.e.
@ -2080,7 +2019,7 @@ def _assign_match(LL: list[Leaf]) -> Optional[int]:
OR
None, otherwise.
"""
# If this line is a part of an expression statement or is a function
# If this line is apart of an expression statement or is a function
# argument AND the first leaf contains a variable name...
if (
parent_type(LL[0]) in [syms.expr_stmt, syms.argument, syms.power]
@ -2101,7 +2040,7 @@ def _assign_match(LL: list[Leaf]) -> Optional[int]:
string_parser = StringParser()
idx = string_parser.parse(LL, string_idx)
# The next leaf MAY be a comma iff this line is a part
# The next leaf MAY be a comma iff this line is apart
# of a function argument...
if (
parent_type(LL[0]) == syms.argument
@ -2117,7 +2056,7 @@ def _assign_match(LL: list[Leaf]) -> Optional[int]:
return None
@staticmethod
def _dict_or_lambda_match(LL: list[Leaf]) -> Optional[int]:
def _dict_or_lambda_match(LL: List[Leaf]) -> Optional[int]:
"""
Returns:
string_idx such that @LL[string_idx] is equal to our target (i.e.
@ -2156,7 +2095,7 @@ def _dict_or_lambda_match(LL: list[Leaf]) -> Optional[int]:
return None
def do_transform(
self, line: Line, string_indices: list[int]
self, line: Line, string_indices: List[int]
) -> Iterator[TResult[Line]]:
LL = line.leaves
assert len(string_indices) == 1, (
@ -2238,17 +2177,18 @@ def do_transform(
elif right_leaves and right_leaves[-1].type == token.RPAR:
# Special case for lambda expressions as dict's value, e.g.:
# my_dict = {
# "key": lambda x: f"formatted: {x}",
# "key": lambda x: f"formatted: {x},
# }
# After wrapping the dict's value with parentheses, the string is
# followed by a RPAR but its opening bracket is lambda's, not
# the string's:
# "key": (lambda x: f"formatted: {x}"),
# "key": (lambda x: f"formatted: {x}),
opening_bracket = right_leaves[-1].opening_bracket
if opening_bracket is not None and opening_bracket in left_leaves:
index = left_leaves.index(opening_bracket)
if (
0 < index < len(left_leaves) - 1
index > 0
and index < len(left_leaves) - 1
and left_leaves[index - 1].type == token.COLON
and left_leaves[index + 1].value == "lambda"
):
@ -2322,7 +2262,7 @@ class StringParser:
DONE: Final = 8
# Lookup Table for Next State
_goto: Final[dict[tuple[ParserState, NodeType], ParserState]] = {
_goto: Final[Dict[Tuple[ParserState, NodeType], ParserState]] = {
# A string trailer may start with '.' OR '%'.
(START, token.DOT): DOT,
(START, token.PERCENT): PERCENT,
@ -2351,13 +2291,13 @@ def __init__(self) -> None:
self._state = self.START
self._unmatched_lpars = 0
def parse(self, leaves: list[Leaf], string_idx: int) -> int:
def parse(self, leaves: List[Leaf], string_idx: int) -> int:
"""
Pre-conditions:
* @leaves[@string_idx].type == token.STRING
Returns:
The index directly after the last leaf which is a part of the string
The index directly after the last leaf which is apart of the string
trailer, if a "trailer" exists.
OR
@string_idx + 1, if no string "trailer" exists.
@ -2380,7 +2320,7 @@ def _next_state(self, leaf: Leaf) -> bool:
MUST be the leaf directly following @leaf.
Returns:
True iff @leaf is a part of the string's trailer.
True iff @leaf is apart of the string's trailer.
"""
# We ignore empty LPAR or RPAR leaves.
if is_empty_par(leaf):

121
src/blackd/__init__.py
View File

@ -2,12 +2,12 @@
import logging
from concurrent.futures import Executor, ProcessPoolExecutor
from datetime import datetime, timezone
from functools import cache, partial
from functools import partial
from multiprocessing import freeze_support
from typing import Set, Tuple
try:
from aiohttp import web
from multidict import MultiMapping
from .middlewares import cors
except ImportError as ie:
@ -34,8 +34,6 @@
SKIP_STRING_NORMALIZATION_HEADER = "X-Skip-String-Normalization"
SKIP_MAGIC_TRAILING_COMMA = "X-Skip-Magic-Trailing-Comma"
PREVIEW = "X-Preview"
UNSTABLE = "X-Unstable"
ENABLE_UNSTABLE_FEATURE = "X-Enable-Unstable-Feature"
FAST_OR_SAFE_HEADER = "X-Fast-Or-Safe"
DIFF_HEADER = "X-Diff"
@ -47,8 +45,6 @@
SKIP_STRING_NORMALIZATION_HEADER,
SKIP_MAGIC_TRAILING_COMMA,
PREVIEW,
UNSTABLE,
ENABLE_UNSTABLE_FEATURE,
FAST_OR_SAFE_HEADER,
DIFF_HEADER,
]
@ -57,10 +53,6 @@
BLACK_VERSION_HEADER = "X-Black-Version"
class HeaderError(Exception):
pass
class InvalidVariantHeader(Exception):
pass
@ -82,19 +74,17 @@ def main(bind_host: str, bind_port: int) -> None:
app = make_app()
ver = black.__version__
black.out(f"blackd version {ver} listening on {bind_host} port {bind_port}")
web.run_app(app, host=bind_host, port=bind_port, handle_signals=True, print=None)
@cache
def executor() -> Executor:
return ProcessPoolExecutor()
# TODO: aiohttp had an incorrect annotation for `print` argument,
# It'll be fixed once aiohttp releases that code
web.run_app(app, host=bind_host, port=bind_port, handle_signals=True, print=None) # type: ignore[arg-type]
def make_app() -> web.Application:
app = web.Application(
middlewares=[cors(allow_headers=(*BLACK_HEADERS, "Content-Type"))]
)
app.add_routes([web.post("/", partial(handle, executor=executor()))])
executor = ProcessPoolExecutor()
app.add_routes([web.post("/", partial(handle, executor=executor))])
return app
@ -105,21 +95,55 @@ async def handle(request: web.Request, executor: Executor) -> web.Response:
return web.Response(
status=501, text="This server only supports protocol version 1"
)
try:
line_length = int(
request.headers.get(LINE_LENGTH_HEADER, black.DEFAULT_LINE_LENGTH)
)
except ValueError:
return web.Response(status=400, text="Invalid line length header value")
if PYTHON_VARIANT_HEADER in request.headers:
value = request.headers[PYTHON_VARIANT_HEADER]
try:
pyi, versions = parse_python_variant_header(value)
except InvalidVariantHeader as e:
return web.Response(
status=400,
text=f"Invalid value for {PYTHON_VARIANT_HEADER}: {e.args[0]}",
)
else:
pyi = False
versions = set()
skip_string_normalization = bool(
request.headers.get(SKIP_STRING_NORMALIZATION_HEADER, False)
)
skip_magic_trailing_comma = bool(
request.headers.get(SKIP_MAGIC_TRAILING_COMMA, False)
)
skip_source_first_line = bool(
request.headers.get(SKIP_SOURCE_FIRST_LINE, False)
)
preview = bool(request.headers.get(PREVIEW, False))
fast = False
if request.headers.get(FAST_OR_SAFE_HEADER, "safe") == "fast":
fast = True
try:
mode = parse_mode(request.headers)
except HeaderError as e:
return web.Response(status=400, text=e.args[0])
mode = black.FileMode(
target_versions=versions,
is_pyi=pyi,
line_length=line_length,
skip_source_first_line=skip_source_first_line,
string_normalization=not skip_string_normalization,
magic_trailing_comma=not skip_magic_trailing_comma,
preview=preview,
)
req_bytes = await request.content.read()
charset = request.charset if request.charset is not None else "utf8"
req_str = req_bytes.decode(charset)
then = datetime.now(timezone.utc)
header = ""
if mode.skip_source_first_line:
if skip_source_first_line:
first_newline_position: int = req_str.find("\n") + 1
header = req_str[:first_newline_position]
req_str = req_str[first_newline_position:]
@ -168,58 +192,7 @@ async def handle(request: web.Request, executor: Executor) -> web.Response:
return web.Response(status=500, headers=headers, text=str(e))
def parse_mode(headers: MultiMapping[str]) -> black.Mode:
try:
line_length = int(headers.get(LINE_LENGTH_HEADER, black.DEFAULT_LINE_LENGTH))
except ValueError:
raise HeaderError("Invalid line length header value") from None
if PYTHON_VARIANT_HEADER in headers:
value = headers[PYTHON_VARIANT_HEADER]
try:
pyi, versions = parse_python_variant_header(value)
except InvalidVariantHeader as e:
raise HeaderError(
f"Invalid value for {PYTHON_VARIANT_HEADER}: {e.args[0]}",
) from None
else:
pyi = False
versions = set()
skip_string_normalization = bool(
headers.get(SKIP_STRING_NORMALIZATION_HEADER, False)
)
skip_magic_trailing_comma = bool(headers.get(SKIP_MAGIC_TRAILING_COMMA, False))
skip_source_first_line = bool(headers.get(SKIP_SOURCE_FIRST_LINE, False))
preview = bool(headers.get(PREVIEW, False))
unstable = bool(headers.get(UNSTABLE, False))
enable_features: set[black.Preview] = set()
enable_unstable_features = headers.get(ENABLE_UNSTABLE_FEATURE, "").split(",")
for piece in enable_unstable_features:
piece = piece.strip()
if piece:
try:
enable_features.add(black.Preview[piece])
except KeyError:
raise HeaderError(
f"Invalid value for {ENABLE_UNSTABLE_FEATURE}: {piece}",
) from None
return black.FileMode(
target_versions=versions,
is_pyi=pyi,
line_length=line_length,
skip_source_first_line=skip_source_first_line,
string_normalization=not skip_string_normalization,
magic_trailing_comma=not skip_magic_trailing_comma,
preview=preview,
unstable=unstable,
enabled_features=enable_features,
)
def parse_python_variant_header(value: str) -> tuple[bool, set[black.TargetVersion]]:
def parse_python_variant_header(value: str) -> Tuple[bool, Set[black.TargetVersion]]:
if value == "pyi":
return True, set()
else:

16
src/blackd/middlewares.py
View File

@ -1,11 +1,21 @@
from collections.abc import Awaitable, Callable, Iterable
from typing import TYPE_CHECKING, Any, Awaitable, Callable, Iterable, TypeVar
from aiohttp.typedefs import Middleware
from aiohttp.web_middlewares import middleware
from aiohttp.web_request import Request
from aiohttp.web_response import StreamResponse
if TYPE_CHECKING:
F = TypeVar("F", bound=Callable[..., Any])
middleware: Callable[[F], F]
else:
try:
from aiohttp.web_middlewares import middleware
except ImportError:
# @middleware is deprecated and its behaviour is the default since aiohttp 4.0
# so if it doesn't exist anymore, define a no-op for forward compatibility.
middleware = lambda x: x # noqa: E731
Handler = Callable[[Request], Awaitable[StreamResponse]]
Middleware = Callable[[Request, Handler], Awaitable[StreamResponse]]
def cors(allow_headers: Iterable[str]) -> Middleware:

13
src/blib2to3/Grammar.txt
View File

@ -12,9 +12,9 @@ file_input: (NEWLINE | stmt)* ENDMARKER
single_input: NEWLINE | simple_stmt | compound_stmt NEWLINE
eval_input: testlist NEWLINE* ENDMARKER
typevar: NAME [':' test] ['=' test]
paramspec: '**' NAME ['=' test]
typevartuple: '*' NAME ['=' (test|star_expr)]
typevar: NAME [':' expr]
paramspec: '**' NAME
typevartuple: '*' NAME
typeparam: typevar | paramspec | typevartuple
typeparams: '[' typeparam (',' typeparam)* [','] ']'
@ -163,7 +163,7 @@ atom: ('(' [yield_expr|testlist_gexp] ')' |
'[' [listmaker] ']' |
'{' [dictsetmaker] '}' |
'`' testlist1 '`' |
NAME | NUMBER | (STRING | fstring)+ | '.' '.' '.')
NAME | NUMBER | STRING+ | '.' '.' '.')
listmaker: (namedexpr_test|star_expr) ( old_comp_for | (',' (namedexpr_test|star_expr))* [','] )
testlist_gexp: (namedexpr_test|star_expr) ( old_comp_for | (',' (namedexpr_test|star_expr))* [','] )
lambdef: 'lambda' [varargslist] ':' test
@ -254,8 +254,3 @@ case_block: "case" patterns [guard] ':' suite
guard: 'if' namedexpr_test
patterns: pattern (',' pattern)* [',']
pattern: (expr|star_expr) ['as' expr]
fstring: FSTRING_START fstring_middle* FSTRING_END
fstring_middle: fstring_replacement_field | FSTRING_MIDDLE
fstring_replacement_field: '{' (yield_expr | testlist_star_expr) ['='] [ "!" NAME ] [ ':' fstring_format_spec* ] '}'
fstring_format_spec: FSTRING_MIDDLE | fstring_replacement_field

39
src/blib2to3/pgen2/driver.py
View File

@ -21,14 +21,13 @@
import os
import pkgutil
import sys
from collections.abc import Iterable, Iterator
from contextlib import contextmanager
from dataclasses import dataclass, field
from logging import Logger
from typing import IO, Any, Optional, Union, cast
from typing import IO, Any, Iterable, Iterator, List, Optional, Tuple, Union, cast
from blib2to3.pgen2.grammar import Grammar
from blib2to3.pgen2.tokenize import TokenInfo
from blib2to3.pgen2.tokenize import GoodTokenInfo
from blib2to3.pytree import NL
# Pgen imports
@ -41,7 +40,7 @@
class ReleaseRange:
start: int
end: Optional[int] = None
tokens: list[Any] = field(default_factory=list)
tokens: List[Any] = field(default_factory=list)
def lock(self) -> None:
total_eaten = len(self.tokens)
@ -52,7 +51,7 @@ class TokenProxy:
def __init__(self, generator: Any) -> None:
self._tokens = generator
self._counter = 0
self._release_ranges: list[ReleaseRange] = []
self._release_ranges: List[ReleaseRange] = []
@contextmanager
def release(self) -> Iterator["TokenProxy"]:
@ -112,7 +111,7 @@ def __init__(self, grammar: Grammar, logger: Optional[Logger] = None) -> None:
logger = logging.getLogger(__name__)
self.logger = logger
def parse_tokens(self, tokens: Iterable[TokenInfo], debug: bool = False) -> NL:
def parse_tokens(self, tokens: Iterable[GoodTokenInfo], debug: bool = False) -> NL:
"""Parse a series of tokens and return the syntax tree."""
# XXX Move the prefix computation into a wrapper around tokenize.
proxy = TokenProxy(tokens)
@ -122,7 +121,7 @@ def parse_tokens(self, tokens: Iterable[TokenInfo], debug: bool = False) -> NL:
lineno = 1
column = 0
indent_columns: list[int] = []
indent_columns: List[int] = []
type = value = start = end = line_text = None
prefix = ""
@ -168,9 +167,7 @@ def parse_tokens(self, tokens: Iterable[TokenInfo], debug: bool = False) -> NL:
if type in {token.INDENT, token.DEDENT}:
prefix = _prefix
lineno, column = end
# FSTRING_MIDDLE is the only token that can end with a newline, and
# `end` will point to the next line. For that case, don't increment lineno.
if value.endswith("\n") and type != token.FSTRING_MIDDLE:
if value.endswith("\n"):
lineno += 1
column = 0
else:
@ -180,21 +177,31 @@ def parse_tokens(self, tokens: Iterable[TokenInfo], debug: bool = False) -> NL:
assert p.rootnode is not None
return p.rootnode
def parse_stream_raw(self, stream: IO[str], debug: bool = False) -> NL:
"""Parse a stream and return the syntax tree."""
tokens = tokenize.generate_tokens(stream.readline, grammar=self.grammar)
return self.parse_tokens(tokens, debug)
def parse_stream(self, stream: IO[str], debug: bool = False) -> NL:
"""Parse a stream and return the syntax tree."""
return self.parse_stream_raw(stream, debug)
def parse_file(
self, filename: Path, encoding: Optional[str] = None, debug: bool = False
) -> NL:
"""Parse a file and return the syntax tree."""
with open(filename, encoding=encoding) as stream:
text = stream.read()
return self.parse_string(text, debug)
return self.parse_stream(stream, debug)
def parse_string(self, text: str, debug: bool = False) -> NL:
"""Parse a string and return the syntax tree."""
tokens = tokenize.tokenize(text, grammar=self.grammar)
tokens = tokenize.generate_tokens(
io.StringIO(text).readline, grammar=self.grammar
)
return self.parse_tokens(tokens, debug)
def _partially_consume_prefix(self, prefix: str, column: int) -> tuple[str, str]:
lines: list[str] = []
def _partially_consume_prefix(self, prefix: str, column: int) -> Tuple[str, str]:
lines: List[str] = []
current_line = ""
current_column = 0
wait_for_nl = False
@ -215,8 +222,6 @@ def _partially_consume_prefix(self, prefix: str, column: int) -> tuple[str, str]
elif char == "\n":
# unexpected empty line
current_column = 0
elif char == "\f":
current_column = 0
else:
# indent is finished
wait_for_nl = True

31
src/blib2to3/pgen2/grammar.py
View File

@ -16,15 +16,15 @@
import os
import pickle
import tempfile
from typing import Any, Optional, TypeVar, Union
from typing import Any, Dict, List, Optional, Tuple, TypeVar, Union
# Local imports
from . import token
_P = TypeVar("_P", bound="Grammar")
Label = tuple[int, Optional[str]]
DFA = list[list[tuple[int, int]]]
DFAS = tuple[DFA, dict[int, int]]
Label = Tuple[int, Optional[str]]
DFA = List[List[Tuple[int, int]]]
DFAS = Tuple[DFA, Dict[int, int]]
Path = Union[str, "os.PathLike[str]"]
@ -83,16 +83,16 @@ class Grammar:
"""
def __init__(self) -> None:
self.symbol2number: dict[str, int] = {}
self.number2symbol: dict[int, str] = {}
self.states: list[DFA] = []
self.dfas: dict[int, DFAS] = {}
self.labels: list[Label] = [(0, "EMPTY")]
self.keywords: dict[str, int] = {}
self.soft_keywords: dict[str, int] = {}
self.tokens: dict[int, int] = {}
self.symbol2label: dict[str, int] = {}
self.version: tuple[int, int] = (0, 0)
self.symbol2number: Dict[str, int] = {}
self.number2symbol: Dict[int, str] = {}
self.states: List[DFA] = []
self.dfas: Dict[int, DFAS] = {}
self.labels: List[Label] = [(0, "EMPTY")]
self.keywords: Dict[str, int] = {}
self.soft_keywords: Dict[str, int] = {}
self.tokens: Dict[int, int] = {}
self.symbol2label: Dict[str, int] = {}
self.version: Tuple[int, int] = (0, 0)
self.start = 256
# Python 3.7+ parses async as a keyword, not an identifier
self.async_keywords = False
@ -114,7 +114,7 @@ def dump(self, filename: Path) -> None:
pickle.dump(d, f, pickle.HIGHEST_PROTOCOL)
os.replace(f.name, filename)
def _update(self, attrs: dict[str, Any]) -> None:
def _update(self, attrs: Dict[str, Any]) -> None:
for k, v in attrs.items():
setattr(self, k, v)
@ -218,7 +218,6 @@ def report(self) -> None:
//= DOUBLESLASHEQUAL
-> RARROW
:= COLONEQUAL
! BANG
"""
opmap = {}

11
src/blib2to3/pgen2/literals.py
View File

@ -4,8 +4,9 @@
"""Safely evaluate Python string literals without using eval()."""
import re
from typing import Dict, Match
simple_escapes: dict[str, str] = {
simple_escapes: Dict[str, str] = {
"a": "\a",
"b": "\b",
"f": "\f",
@ -19,7 +20,7 @@
}
def escape(m: re.Match[str]) -> str:
def escape(m: Match[str]) -> str:
all, tail = m.group(0, 1)
assert all.startswith("\\")
esc = simple_escapes.get(tail)
@ -28,16 +29,16 @@ def escape(m: re.Match[str]) -> str:
if tail.startswith("x"):
hexes = tail[1:]
if len(hexes) < 2:
raise ValueError(f"invalid hex string escape ('\\{tail}')")
raise ValueError("invalid hex string escape ('\\%s')" % tail)
try:
i = int(hexes, 16)
except ValueError:
raise ValueError(f"invalid hex string escape ('\\{tail}')") from None
raise ValueError("invalid hex string escape ('\\%s')" % tail) from None
else:
try:
i = int(tail, 8)
except ValueError:
raise ValueError(f"invalid octal string escape ('\\{tail}')") from None
raise ValueError("invalid octal string escape ('\\%s')" % tail) from None
return chr(i)

49
src/blib2to3/pgen2/parse.py
View File

@ -9,9 +9,20 @@
how this parsing engine works.
"""
from collections.abc import Callable, Iterator
from contextlib import contextmanager
from typing import TYPE_CHECKING, Any, Optional, Union, cast
from typing import (
TYPE_CHECKING,
Any,
Callable,
Dict,
Iterator,
List,
Optional,
Set,
Tuple,
Union,
cast,
)
from blib2to3.pgen2.grammar import Grammar
from blib2to3.pytree import NL, Context, Leaf, Node, RawNode, convert
@ -23,10 +34,10 @@
from blib2to3.pgen2.driver import TokenProxy
Results = dict[str, NL]
Results = Dict[str, NL]
Convert = Callable[[Grammar, RawNode], Union[Node, Leaf]]
DFA = list[list[tuple[int, int]]]
DFAS = tuple[DFA, dict[int, int]]
DFA = List[List[Tuple[int, int]]]
DFAS = Tuple[DFA, Dict[int, int]]
def lam_sub(grammar: Grammar, node: RawNode) -> NL:
@ -39,24 +50,24 @@ def lam_sub(grammar: Grammar, node: RawNode) -> NL:
def stack_copy(
stack: list[tuple[DFAS, int, RawNode]],
) -> list[tuple[DFAS, int, RawNode]]:
stack: List[Tuple[DFAS, int, RawNode]]
) -> List[Tuple[DFAS, int, RawNode]]:
"""Nodeless stack copy."""
return [(dfa, label, DUMMY_NODE) for dfa, label, _ in stack]
class Recorder:
def __init__(self, parser: "Parser", ilabels: list[int], context: Context) -> None:
def __init__(self, parser: "Parser", ilabels: List[int], context: Context) -> None:
self.parser = parser
self._ilabels = ilabels
self.context = context # not really matter
self._dead_ilabels: set[int] = set()
self._dead_ilabels: Set[int] = set()
self._start_point = self.parser.stack
self._points = {ilabel: stack_copy(self._start_point) for ilabel in ilabels}
@property
def ilabels(self) -> set[int]:
def ilabels(self) -> Set[int]:
return self._dead_ilabels.symmetric_difference(self._ilabels)
@contextmanager
@ -89,12 +100,18 @@ def backtrack(self) -> Iterator[None]:
self.parser.is_backtracking = is_backtracking
def add_token(self, tok_type: int, tok_val: str, raw: bool = False) -> None:
func: Callable[..., Any]
if raw:
func = self.parser._addtoken
else:
func = self.parser.addtoken
for ilabel in self.ilabels:
with self.switch_to(ilabel):
args = [tok_type, tok_val, self.context]
if raw:
self.parser._addtoken(ilabel, tok_type, tok_val, self.context)
else:
self.parser.addtoken(tok_type, tok_val, self.context)
args.insert(0, ilabel)
func(*args)
def determine_route(
self, value: Optional[str] = None, force: bool = False
@ -216,9 +233,9 @@ def setup(self, proxy: "TokenProxy", start: Optional[int] = None) -> None:
# where children is a list of nodes or None, and context may be None.
newnode: RawNode = (start, None, None, [])
stackentry = (self.grammar.dfas[start], 0, newnode)
self.stack: list[tuple[DFAS, int, RawNode]] = [stackentry]
self.stack: List[Tuple[DFAS, int, RawNode]] = [stackentry]
self.rootnode: Optional[NL] = None
self.used_names: set[str] = set()
self.used_names: Set[str] = set()
self.proxy = proxy
self.last_token = None
@ -316,7 +333,7 @@ def _addtoken(self, ilabel: int, type: int, value: str, context: Context) -> boo
# No success finding a transition
raise ParseError("bad input", type, value, context)
def classify(self, type: int, value: str, context: Context) -> list[int]:
def classify(self, type: int, value: str, context: Context) -> List[int]:
"""Turn a token into a label. (Internal)
Depending on whether the value is a soft-keyword or not,

89
src/blib2to3/pgen2/pgen.py
View File

@ -2,11 +2,21 @@
# Licensed to PSF under a Contributor Agreement.
import os
from collections.abc import Iterator, Sequence
from typing import IO, Any, NoReturn, Optional, Union
from typing import (
IO,
Any,
Dict,
Iterator,
List,
NoReturn,
Optional,
Sequence,
Tuple,
Union,
)
from blib2to3.pgen2 import grammar, token, tokenize
from blib2to3.pgen2.tokenize import TokenInfo
from blib2to3.pgen2.tokenize import GoodTokenInfo
Path = Union[str, "os.PathLike[str]"]
@ -18,8 +28,8 @@ class PgenGrammar(grammar.Grammar):
class ParserGenerator:
filename: Path
stream: IO[str]
generator: Iterator[TokenInfo]
first: dict[str, Optional[dict[str, int]]]
generator: Iterator[GoodTokenInfo]
first: Dict[str, Optional[Dict[str, int]]]
def __init__(self, filename: Path, stream: Optional[IO[str]] = None) -> None:
close_stream = None
@ -27,7 +37,8 @@ def __init__(self, filename: Path, stream: Optional[IO[str]] = None) -> None:
stream = open(filename, encoding="utf-8")
close_stream = stream.close
self.filename = filename
self.generator = tokenize.tokenize(stream.read())
self.stream = stream
self.generator = tokenize.generate_tokens(stream.readline)
self.gettoken() # Initialize lookahead
self.dfas, self.startsymbol = self.parse()
if close_stream is not None:
@ -60,7 +71,7 @@ def make_grammar(self) -> PgenGrammar:
c.start = c.symbol2number[self.startsymbol]
return c
def make_first(self, c: PgenGrammar, name: str) -> dict[int, int]:
def make_first(self, c: PgenGrammar, name: str) -> Dict[int, int]:
rawfirst = self.first[name]
assert rawfirst is not None
first = {}
@ -133,14 +144,14 @@ def calcfirst(self, name: str) -> None:
dfa = self.dfas[name]
self.first[name] = None # dummy to detect left recursion
state = dfa[0]
totalset: dict[str, int] = {}
totalset: Dict[str, int] = {}
overlapcheck = {}
for label in state.arcs:
if label in self.dfas:
if label in self.first:
fset = self.first[label]
if fset is None:
raise ValueError(f"recursion for rule {name!r}")
raise ValueError("recursion for rule %r" % name)
else:
self.calcfirst(label)
fset = self.first[label]
@ -150,18 +161,18 @@ def calcfirst(self, name: str) -> None:
else:
totalset[label] = 1
overlapcheck[label] = {label: 1}
inverse: dict[str, str] = {}
inverse: Dict[str, str] = {}
for label, itsfirst in overlapcheck.items():
for symbol in itsfirst:
if symbol in inverse:
raise ValueError(
f"rule {name} is ambiguous; {symbol} is in the first sets of"
f" {label} as well as {inverse[symbol]}"
"rule %s is ambiguous; %s is in the first sets of %s as well"
" as %s" % (name, symbol, label, inverse[symbol])
)
inverse[symbol] = label
self.first[name] = totalset
def parse(self) -> tuple[dict[str, list["DFAState"]], str]:
def parse(self) -> Tuple[Dict[str, List["DFAState"]], str]:
dfas = {}
startsymbol: Optional[str] = None
# MSTART: (NEWLINE | RULE)* ENDMARKER
@ -186,7 +197,7 @@ def parse(self) -> tuple[dict[str, list["DFAState"]], str]:
assert startsymbol is not None
return dfas, startsymbol
def make_dfa(self, start: "NFAState", finish: "NFAState") -> list["DFAState"]:
def make_dfa(self, start: "NFAState", finish: "NFAState") -> List["DFAState"]:
# To turn an NFA into a DFA, we define the states of the DFA
# to correspond to *sets* of states of the NFA. Then do some
# state reduction. Let's represent sets as dicts with 1 for
@ -194,12 +205,12 @@ def make_dfa(self, start: "NFAState", finish: "NFAState") -> list["DFAState"]:
assert isinstance(start, NFAState)
assert isinstance(finish, NFAState)
def closure(state: NFAState) -> dict[NFAState, int]:
base: dict[NFAState, int] = {}
def closure(state: NFAState) -> Dict[NFAState, int]:
base: Dict[NFAState, int] = {}
addclosure(state, base)
return base
def addclosure(state: NFAState, base: dict[NFAState, int]) -> None:
def addclosure(state: NFAState, base: Dict[NFAState, int]) -> None:
assert isinstance(state, NFAState)
if state in base:
return
@ -210,7 +221,7 @@ def addclosure(state: NFAState, base: dict[NFAState, int]) -> None:
states = [DFAState(closure(start), finish)]
for state in states: # NB states grows while we're iterating
arcs: dict[str, dict[NFAState, int]] = {}
arcs: Dict[str, Dict[NFAState, int]] = {}
for nfastate in state.nfaset:
for label, next in nfastate.arcs:
if label is not None:
@ -237,18 +248,18 @@ def dump_nfa(self, name: str, start: "NFAState", finish: "NFAState") -> None:
j = len(todo)
todo.append(next)
if label is None:
print(f" -> {j}")
print(" -> %d" % j)
else:
print(f" {label} -> {j}")
print(" %s -> %d" % (label, j))
def dump_dfa(self, name: str, dfa: Sequence["DFAState"]) -> None:
print("Dump of DFA for", name)
for i, state in enumerate(dfa):
print(" State", i, state.isfinal and "(final)" or "")
for label, next in sorted(state.arcs.items()):
print(f" {label} -> {dfa.index(next)}")
print(" %s -> %d" % (label, dfa.index(next)))
def simplify_dfa(self, dfa: list["DFAState"]) -> None:
def simplify_dfa(self, dfa: List["DFAState"]) -> None:
# This is not theoretically optimal, but works well enough.
# Algorithm: repeatedly look for two states that have the same
# set of arcs (same labels pointing to the same nodes) and
@ -269,7 +280,7 @@ def simplify_dfa(self, dfa: list["DFAState"]) -> None:
changes = True
break
def parse_rhs(self) -> tuple["NFAState", "NFAState"]:
def parse_rhs(self) -> Tuple["NFAState", "NFAState"]:
# RHS: ALT ('|' ALT)*
a, z = self.parse_alt()
if self.value != "|":
@ -286,7 +297,7 @@ def parse_rhs(self) -> tuple["NFAState", "NFAState"]:
z.addarc(zz)
return aa, zz
def parse_alt(self) -> tuple["NFAState", "NFAState"]:
def parse_alt(self) -> Tuple["NFAState", "NFAState"]:
# ALT: ITEM+
a, b = self.parse_item()
while self.value in ("(", "[") or self.type in (token.NAME, token.STRING):
@ -295,7 +306,7 @@ def parse_alt(self) -> tuple["NFAState", "NFAState"]:
b = d
return a, b
def parse_item(self) -> tuple["NFAState", "NFAState"]:
def parse_item(self) -> Tuple["NFAState", "NFAState"]:
# ITEM: '[' RHS ']' | ATOM ['+' | '*']
if self.value == "[":
self.gettoken()
@ -315,7 +326,7 @@ def parse_item(self) -> tuple["NFAState", "NFAState"]:
else:
return a, a
def parse_atom(self) -> tuple["NFAState", "NFAState"]:
def parse_atom(self) -> Tuple["NFAState", "NFAState"]:
# ATOM: '(' RHS ')' | NAME | STRING
if self.value == "(":
self.gettoken()
@ -330,12 +341,15 @@ def parse_atom(self) -> tuple["NFAState", "NFAState"]:
return a, z
else:
self.raise_error(
f"expected (...) or NAME or STRING, got {self.type}/{self.value}"
"expected (...) or NAME or STRING, got %s/%s", self.type, self.value
)
raise AssertionError
def expect(self, type: int, value: Optional[Any] = None) -> str:
if self.type != type or (value is not None and self.value != value):
self.raise_error(f"expected {type}/{value}, got {self.type}/{self.value}")
self.raise_error(
"expected %s/%s, got %s/%s", type, value, self.type, self.value
)
value = self.value
self.gettoken()
return value
@ -347,14 +361,17 @@ def gettoken(self) -> None:
self.type, self.value, self.begin, self.end, self.line = tup
# print token.tok_name[self.type], repr(self.value)
def raise_error(self, msg: str) -> NoReturn:
raise SyntaxError(
msg, (str(self.filename), self.end[0], self.end[1], self.line)
)
def raise_error(self, msg: str, *args: Any) -> NoReturn:
if args:
try:
msg = msg % args
except Exception:
msg = " ".join([msg] + list(map(str, args)))
raise SyntaxError(msg, (self.filename, self.end[0], self.end[1], self.line))
class NFAState:
arcs: list[tuple[Optional[str], "NFAState"]]
arcs: List[Tuple[Optional[str], "NFAState"]]
def __init__(self) -> None:
self.arcs = [] # list of (label, NFAState) pairs
@ -366,11 +383,11 @@ def addarc(self, next: "NFAState", label: Optional[str] = None) -> None:
class DFAState:
nfaset: dict[NFAState, Any]
nfaset: Dict[NFAState, Any]
isfinal: bool
arcs: dict[str, "DFAState"]
arcs: Dict[str, "DFAState"]
def __init__(self, nfaset: dict[NFAState, Any], final: NFAState) -> None:
def __init__(self, nfaset: Dict[NFAState, Any], final: NFAState) -> None:
assert isinstance(nfaset, dict)
assert isinstance(next(iter(nfaset)), NFAState)
assert isinstance(final, NFAState)

10
src/blib2to3/pgen2/token.py
View File

@ -1,6 +1,6 @@
"""Token constants (from "token.h")."""
from typing import Final
from typing import Dict, Final
# Taken from Python (r53757) and modified to include some tokens
# originally monkeypatched in by pgen2.tokenize
@ -66,15 +66,11 @@
ASYNC: Final = 57
ERRORTOKEN: Final = 58
COLONEQUAL: Final = 59
FSTRING_START: Final = 60
FSTRING_MIDDLE: Final = 61
FSTRING_END: Final = 62
BANG: Final = 63
N_TOKENS: Final = 64
N_TOKENS: Final = 60
NT_OFFSET: Final = 256
# --end constants--
tok_name: Final[dict[int, str]] = {}
tok_name: Final[Dict[int, str]] = {}
for _name, _value in list(globals().items()):
if type(_value) is int:
tok_name[_value] = _name

758
src/blib2to3/pgen2/tokenize.py
View File

@ -28,8 +28,19 @@
each time a new token is found."""
import sys
from collections.abc import Iterator
from typing import Optional
from typing import (
Callable,
Final,
Iterable,
Iterator,
List,
Optional,
Pattern,
Set,
Tuple,
Union,
cast,
)
from blib2to3.pgen2.grammar import Grammar
from blib2to3.pgen2.token import (
@ -39,9 +50,6 @@
DEDENT,
ENDMARKER,
ERRORTOKEN,
FSTRING_END,
FSTRING_MIDDLE,
FSTRING_START,
INDENT,
NAME,
NEWLINE,
@ -55,150 +63,141 @@
__author__ = "Ka-Ping Yee <ping@lfw.org>"
__credits__ = "GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip Montanaro"
import pytokens
from pytokens import TokenType
import re
from codecs import BOM_UTF8, lookup
from . import token as _token
from . import token
__all__ = [x for x in dir(_token) if x[0] != "_"] + [
__all__ = [x for x in dir(token) if x[0] != "_"] + [
"tokenize",
"generate_tokens",
"untokenize",
]
del _token
del token
Coord = tuple[int, int]
TokenInfo = tuple[int, str, Coord, Coord, str]
TOKEN_TYPE_MAP = {
TokenType.indent: INDENT,
TokenType.dedent: DEDENT,
TokenType.newline: NEWLINE,
TokenType.nl: NL,
TokenType.comment: COMMENT,
TokenType.semicolon: OP,
TokenType.lparen: OP,
TokenType.rparen: OP,
TokenType.lbracket: OP,
TokenType.rbracket: OP,
TokenType.lbrace: OP,
TokenType.rbrace: OP,
TokenType.colon: OP,
TokenType.op: OP,
TokenType.identifier: NAME,
TokenType.number: NUMBER,
TokenType.string: STRING,
TokenType.fstring_start: FSTRING_START,
TokenType.fstring_middle: FSTRING_MIDDLE,
TokenType.fstring_end: FSTRING_END,
TokenType.endmarker: ENDMARKER,
def group(*choices: str) -> str:
return "(" + "|".join(choices) + ")"
def any(*choices: str) -> str:
return group(*choices) + "*"
def maybe(*choices: str) -> str:
return group(*choices) + "?"
def _combinations(*l: str) -> Set[str]:
return {x + y for x in l for y in l + ("",) if x.casefold() != y.casefold()}
Whitespace = r"[ \f\t]*"
Comment = r"#[^\r\n]*"
Ignore = Whitespace + any(r"\\\r?\n" + Whitespace) + maybe(Comment)
Name = ( # this is invalid but it's fine because Name comes after Number in all groups
r"[^\s#\(\)\[\]\{\}+\-*/!@$%^&=|;:'\",\.<>/?`~\\]+"
)
Binnumber = r"0[bB]_?[01]+(?:_[01]+)*"
Hexnumber = r"0[xX]_?[\da-fA-F]+(?:_[\da-fA-F]+)*[lL]?"
Octnumber = r"0[oO]?_?[0-7]+(?:_[0-7]+)*[lL]?"
Decnumber = group(r"[1-9]\d*(?:_\d+)*[lL]?", "0[lL]?")
Intnumber = group(Binnumber, Hexnumber, Octnumber, Decnumber)
Exponent = r"[eE][-+]?\d+(?:_\d+)*"
Pointfloat = group(r"\d+(?:_\d+)*\.(?:\d+(?:_\d+)*)?", r"\.\d+(?:_\d+)*") + maybe(
Exponent
)
Expfloat = r"\d+(?:_\d+)*" + Exponent
Floatnumber = group(Pointfloat, Expfloat)
Imagnumber = group(r"\d+(?:_\d+)*[jJ]", Floatnumber + r"[jJ]")
Number = group(Imagnumber, Floatnumber, Intnumber)
# Tail end of ' string.
Single = r"[^'\\]*(?:\\.[^'\\]*)*'"
# Tail end of " string.
Double = r'[^"\\]*(?:\\.[^"\\]*)*"'
# Tail end of ''' string.
Single3 = r"[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''"
# Tail end of """ string.
Double3 = r'[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""'
_litprefix = r"(?:[uUrRbBfF]|[rR][fFbB]|[fFbBuU][rR])?"
Triple = group(_litprefix + "'''", _litprefix + '"""')
# Single-line ' or " string.
String = group(
_litprefix + r"'[^\n'\\]*(?:\\.[^\n'\\]*)*'",
_litprefix + r'"[^\n"\\]*(?:\\.[^\n"\\]*)*"',
)
# Because of leftmost-then-longest match semantics, be sure to put the
# longest operators first (e.g., if = came before ==, == would get
# recognized as two instances of =).
Operator = group(
r"\*\*=?",
r">>=?",
r"<<=?",
r"<>",
r"!=",
r"//=?",
r"->",
r"[+\-*/%&@|^=<>:]=?",
r"~",
)
Bracket = "[][(){}]"
Special = group(r"\r?\n", r"[:;.,`@]")
Funny = group(Operator, Bracket, Special)
# First (or only) line of ' or " string.
ContStr = group(
_litprefix + r"'[^\n'\\]*(?:\\.[^\n'\\]*)*" + group("'", r"\\\r?\n"),
_litprefix + r'"[^\n"\\]*(?:\\.[^\n"\\]*)*' + group('"', r"\\\r?\n"),
)
PseudoExtras = group(r"\\\r?\n", Comment, Triple)
PseudoToken = Whitespace + group(PseudoExtras, Number, Funny, ContStr, Name)
pseudoprog: Final = re.compile(PseudoToken, re.UNICODE)
single3prog = re.compile(Single3)
double3prog = re.compile(Double3)
_strprefixes = (
_combinations("r", "R", "f", "F")
| _combinations("r", "R", "b", "B")
| {"u", "U", "ur", "uR", "Ur", "UR"}
)
endprogs: Final = {
"'": re.compile(Single),
'"': re.compile(Double),
"'''": single3prog,
'"""': double3prog,
**{f"{prefix}'''": single3prog for prefix in _strprefixes},
**{f'{prefix}"""': double3prog for prefix in _strprefixes},
}
triple_quoted: Final = (
{"'''", '"""'}
| {f"{prefix}'''" for prefix in _strprefixes}
| {f'{prefix}"""' for prefix in _strprefixes}
)
single_quoted: Final = (
{"'", '"'}
| {f"{prefix}'" for prefix in _strprefixes}
| {f'{prefix}"' for prefix in _strprefixes}
)
class TokenError(Exception): ...
tabsize = 8
def transform_whitespace(
token: pytokens.Token, source: str, prev_token: Optional[pytokens.Token]
) -> pytokens.Token:
r"""
Black treats `\\\n` at the end of a line as a 'NL' token, while it
is ignored as whitespace in the regular Python parser.
But, only the first one. If there's a `\\\n` following it
(as in, a \ just by itself on a line), that is not made into NL.
"""
if (
token.type == TokenType.whitespace
and prev_token is not None
and prev_token.type not in (TokenType.nl, TokenType.newline)
):
token_str = source[token.start_index : token.end_index]
if token_str.startswith("\\\r\n"):
return pytokens.Token(
TokenType.nl,
token.start_index,
token.start_index + 3,
token.start_line,
token.start_col,
token.start_line,
token.start_col + 3,
)
elif token_str.startswith("\\\n") or token_str.startswith("\\\r"):
return pytokens.Token(
TokenType.nl,
token.start_index,
token.start_index + 2,
token.start_line,
token.start_col,
token.start_line,
token.start_col + 2,
)
return token
class TokenError(Exception):
pass
def tokenize(source: str, grammar: Optional[Grammar] = None) -> Iterator[TokenInfo]:
lines = source.split("\n")
lines += [""] # For newline tokens in files that don't end in a newline
line, column = 1, 0
class StopTokenizing(Exception):
pass
prev_token: Optional[pytokens.Token] = None
try:
for token in pytokens.tokenize(source):
token = transform_whitespace(token, source, prev_token)
line, column = token.start_line, token.start_col
if token.type == TokenType.whitespace:
continue
token_str = source[token.start_index : token.end_index]
if token.type == TokenType.newline and token_str == "":
# Black doesn't yield empty newline tokens at the end of a file
# if there's no newline at the end of a file.
prev_token = token
continue
source_line = lines[token.start_line - 1]
if token.type == TokenType.identifier and token_str in ("async", "await"):
# Black uses `async` and `await` token types just for those two keywords
yield (
ASYNC if token_str == "async" else AWAIT,
token_str,
(token.start_line, token.start_col),
(token.end_line, token.end_col),
source_line,
)
elif token.type == TokenType.op and token_str == "...":
# Black doesn't have an ellipsis token yet, yield 3 DOTs instead
assert token.start_line == token.end_line
assert token.end_col == token.start_col + 3
token_str = "."
for start_col in range(token.start_col, token.start_col + 3):
end_col = start_col + 1
yield (
TOKEN_TYPE_MAP[token.type],
token_str,
(token.start_line, start_col),
(token.end_line, end_col),
source_line,
)
else:
yield (
TOKEN_TYPE_MAP[token.type],
token_str,
(token.start_line, token.start_col),
(token.end_line, token.end_col),
source_line,
)
prev_token = token
except pytokens.UnexpectedEOF:
raise TokenError("Unexpected EOF in multi-line statement", (line, column))
except pytokens.TokenizeError as exc:
raise TokenError(f"Failed to parse: {type(exc).__name__}", (line, column))
Coord = Tuple[int, int]
def printtoken(
@ -206,14 +205,497 @@ def printtoken(
) -> None: # for testing
(srow, scol) = srow_col
(erow, ecol) = erow_col
print(f"{srow},{scol}-{erow},{ecol}:\t{tok_name[type]}\t{token!r}")
print(
"%d,%d-%d,%d:\t%s\t%s" % (srow, scol, erow, ecol, tok_name[type], repr(token))
)
TokenEater = Callable[[int, str, Coord, Coord, str], None]
def tokenize(readline: Callable[[], str], tokeneater: TokenEater = printtoken) -> None:
"""
The tokenize() function accepts two parameters: one representing the
input stream, and one providing an output mechanism for tokenize().
The first parameter, readline, must be a callable object which provides
the same interface as the readline() method of built-in file objects.
Each call to the function should return one line of input as a string.
The second parameter, tokeneater, must also be a callable object. It is
called once for each token, with five arguments, corresponding to the
tuples generated by generate_tokens().
"""
try:
tokenize_loop(readline, tokeneater)
except StopTokenizing:
pass
# backwards compatible interface
def tokenize_loop(readline: Callable[[], str], tokeneater: TokenEater) -> None:
for token_info in generate_tokens(readline):
tokeneater(*token_info)
GoodTokenInfo = Tuple[int, str, Coord, Coord, str]
TokenInfo = Union[Tuple[int, str], GoodTokenInfo]
class Untokenizer:
tokens: List[str]
prev_row: int
prev_col: int
def __init__(self) -> None:
self.tokens = []
self.prev_row = 1
self.prev_col = 0
def add_whitespace(self, start: Coord) -> None:
row, col = start
assert row <= self.prev_row
col_offset = col - self.prev_col
if col_offset:
self.tokens.append(" " * col_offset)
def untokenize(self, iterable: Iterable[TokenInfo]) -> str:
for t in iterable:
if len(t) == 2:
self.compat(cast(Tuple[int, str], t), iterable)
break
tok_type, token, start, end, line = cast(
Tuple[int, str, Coord, Coord, str], t
)
self.add_whitespace(start)
self.tokens.append(token)
self.prev_row, self.prev_col = end
if tok_type in (NEWLINE, NL):
self.prev_row += 1
self.prev_col = 0
return "".join(self.tokens)
def compat(self, token: Tuple[int, str], iterable: Iterable[TokenInfo]) -> None:
startline = False
indents = []
toks_append = self.tokens.append
toknum, tokval = token
if toknum in (NAME, NUMBER):
tokval += " "
if toknum in (NEWLINE, NL):
startline = True
for tok in iterable:
toknum, tokval = tok[:2]
if toknum in (NAME, NUMBER, ASYNC, AWAIT):
tokval += " "
if toknum == INDENT:
indents.append(tokval)
continue
elif toknum == DEDENT:
indents.pop()
continue
elif toknum in (NEWLINE, NL):
startline = True
elif startline and indents:
toks_append(indents[-1])
startline = False
toks_append(tokval)
cookie_re = re.compile(r"^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)", re.ASCII)
blank_re = re.compile(rb"^[ \t\f]*(?:[#\r\n]|$)", re.ASCII)
def _get_normal_name(orig_enc: str) -> str:
"""Imitates get_normal_name in tokenizer.c."""
# Only care about the first 12 characters.
enc = orig_enc[:12].lower().replace("_", "-")
if enc == "utf-8" or enc.startswith("utf-8-"):
return "utf-8"
if enc in ("latin-1", "iso-8859-1", "iso-latin-1") or enc.startswith(
("latin-1-", "iso-8859-1-", "iso-latin-1-")
):
return "iso-8859-1"
return orig_enc
def detect_encoding(readline: Callable[[], bytes]) -> Tuple[str, List[bytes]]:
"""
The detect_encoding() function is used to detect the encoding that should
be used to decode a Python source file. It requires one argument, readline,
in the same way as the tokenize() generator.
It will call readline a maximum of twice, and return the encoding used
(as a string) and a list of any lines (left as bytes) it has read
in.
It detects the encoding from the presence of a utf-8 bom or an encoding
cookie as specified in pep-0263. If both a bom and a cookie are present, but
disagree, a SyntaxError will be raised. If the encoding cookie is an invalid
charset, raise a SyntaxError. Note that if a utf-8 bom is found,
'utf-8-sig' is returned.
If no encoding is specified, then the default of 'utf-8' will be returned.
"""
bom_found = False
encoding = None
default = "utf-8"
def read_or_stop() -> bytes:
try:
return readline()
except StopIteration:
return b""
def find_cookie(line: bytes) -> Optional[str]:
try:
line_string = line.decode("ascii")
except UnicodeDecodeError:
return None
match = cookie_re.match(line_string)
if not match:
return None
encoding = _get_normal_name(match.group(1))
try:
codec = lookup(encoding)
except LookupError:
# This behaviour mimics the Python interpreter
raise SyntaxError("unknown encoding: " + encoding)
if bom_found:
if codec.name != "utf-8":
# This behaviour mimics the Python interpreter
raise SyntaxError("encoding problem: utf-8")
encoding += "-sig"
return encoding
first = read_or_stop()
if first.startswith(BOM_UTF8):
bom_found = True
first = first[3:]
default = "utf-8-sig"
if not first:
return default, []
encoding = find_cookie(first)
if encoding:
return encoding, [first]
if not blank_re.match(first):
return default, [first]
second = read_or_stop()
if not second:
return default, [first]
encoding = find_cookie(second)
if encoding:
return encoding, [first, second]
return default, [first, second]
def untokenize(iterable: Iterable[TokenInfo]) -> str:
"""Transform tokens back into Python source code.
Each element returned by the iterable must be a token sequence
with at least two elements, a token number and token value. If
only two tokens are passed, the resulting output is poor.
Round-trip invariant for full input:
Untokenized source will match input source exactly
Round-trip invariant for limited input:
# Output text will tokenize the back to the input
t1 = [tok[:2] for tok in generate_tokens(f.readline)]
newcode = untokenize(t1)
readline = iter(newcode.splitlines(1)).next
t2 = [tok[:2] for tokin generate_tokens(readline)]
assert t1 == t2
"""
ut = Untokenizer()
return ut.untokenize(iterable)
def generate_tokens(
readline: Callable[[], str], grammar: Optional[Grammar] = None
) -> Iterator[GoodTokenInfo]:
"""
The generate_tokens() generator requires one argument, readline, which
must be a callable object which provides the same interface as the
readline() method of built-in file objects. Each call to the function
should return one line of input as a string. Alternately, readline
can be a callable function terminating with StopIteration:
readline = open(myfile).next # Example of alternate readline
The generator produces 5-tuples with these members: the token type; the
token string; a 2-tuple (srow, scol) of ints specifying the row and
column where the token begins in the source; a 2-tuple (erow, ecol) of
ints specifying the row and column where the token ends in the source;
and the line on which the token was found. The line passed is the
logical line; continuation lines are included.
"""
lnum = parenlev = continued = 0
numchars: Final[str] = "0123456789"
contstr, needcont = "", 0
contline: Optional[str] = None
indents = [0]
# If we know we're parsing 3.7+, we can unconditionally parse `async` and
# `await` as keywords.
async_keywords = False if grammar is None else grammar.async_keywords
# 'stashed' and 'async_*' are used for async/await parsing
stashed: Optional[GoodTokenInfo] = None
async_def = False
async_def_indent = 0
async_def_nl = False
strstart: Tuple[int, int]
endprog: Pattern[str]
while 1: # loop over lines in stream
try:
line = readline()
except StopIteration:
line = ""
lnum += 1
pos, max = 0, len(line)
if contstr: # continued string
assert contline is not None
if not line:
raise TokenError("EOF in multi-line string", strstart)
endmatch = endprog.match(line)
if endmatch:
pos = end = endmatch.end(0)
yield (
STRING,
contstr + line[:end],
strstart,
(lnum, end),
contline + line,
)
contstr, needcont = "", 0
contline = None
elif needcont and line[-2:] != "\\\n" and line[-3:] != "\\\r\n":
yield (
ERRORTOKEN,
contstr + line,
strstart,
(lnum, len(line)),
contline,
)
contstr = ""
contline = None
continue
else:
contstr = contstr + line
contline = contline + line
continue
elif parenlev == 0 and not continued: # new statement
if not line:
break
column = 0
while pos < max: # measure leading whitespace
if line[pos] == " ":
column += 1
elif line[pos] == "\t":
column = (column // tabsize + 1) * tabsize
elif line[pos] == "\f":
column = 0
else:
break
pos += 1
if pos == max:
break
if stashed:
yield stashed
stashed = None
if line[pos] in "\r\n": # skip blank lines
yield (NL, line[pos:], (lnum, pos), (lnum, len(line)), line)
continue
if line[pos] == "#": # skip comments
comment_token = line[pos:].rstrip("\r\n")
nl_pos = pos + len(comment_token)
yield (
COMMENT,
comment_token,
(lnum, pos),
(lnum, nl_pos),
line,
)
yield (NL, line[nl_pos:], (lnum, nl_pos), (lnum, len(line)), line)
continue
if column > indents[-1]: # count indents
indents.append(column)
yield (INDENT, line[:pos], (lnum, 0), (lnum, pos), line)
while column < indents[-1]: # count dedents
if column not in indents:
raise IndentationError(
"unindent does not match any outer indentation level",
("<tokenize>", lnum, pos, line),
)
indents = indents[:-1]
if async_def and async_def_indent >= indents[-1]:
async_def = False
async_def_nl = False
async_def_indent = 0
yield (DEDENT, "", (lnum, pos), (lnum, pos), line)
if async_def and async_def_nl and async_def_indent >= indents[-1]:
async_def = False
async_def_nl = False
async_def_indent = 0
else: # continued statement
if not line:
raise TokenError("EOF in multi-line statement", (lnum, 0))
continued = 0
while pos < max:
pseudomatch = pseudoprog.match(line, pos)
if pseudomatch: # scan for tokens
start, end = pseudomatch.span(1)
spos, epos, pos = (lnum, start), (lnum, end), end
token, initial = line[start:end], line[start]
if initial in numchars or (
initial == "." and token != "."
): # ordinary number
yield (NUMBER, token, spos, epos, line)
elif initial in "\r\n":
newline = NEWLINE
if parenlev > 0:
newline = NL
elif async_def:
async_def_nl = True
if stashed:
yield stashed
stashed = None
yield (newline, token, spos, epos, line)
elif initial == "#":
assert not token.endswith("\n")
if stashed:
yield stashed
stashed = None
yield (COMMENT, token, spos, epos, line)
elif token in triple_quoted:
endprog = endprogs[token]
endmatch = endprog.match(line, pos)
if endmatch: # all on one line
pos = endmatch.end(0)
token = line[start:pos]
if stashed:
yield stashed
stashed = None
yield (STRING, token, spos, (lnum, pos), line)
else:
strstart = (lnum, start) # multiple lines
contstr = line[start:]
contline = line
break
elif (
initial in single_quoted
or token[:2] in single_quoted
or token[:3] in single_quoted
):
if token[-1] == "\n": # continued string
strstart = (lnum, start)
maybe_endprog = (
endprogs.get(initial)
or endprogs.get(token[1])
or endprogs.get(token[2])
)
assert (
maybe_endprog is not None
), f"endprog not found for {token}"
endprog = maybe_endprog
contstr, needcont = line[start:], 1
contline = line
break
else: # ordinary string
if stashed:
yield stashed
stashed = None
yield (STRING, token, spos, epos, line)
elif initial.isidentifier(): # ordinary name
if token in ("async", "await"):
if async_keywords or async_def:
yield (
ASYNC if token == "async" else AWAIT,
token,
spos,
epos,
line,
)
continue
tok = (NAME, token, spos, epos, line)
if token == "async" and not stashed:
stashed = tok
continue
if token in ("def", "for"):
if stashed and stashed[0] == NAME and stashed[1] == "async":
if token == "def":
async_def = True
async_def_indent = indents[-1]
yield (
ASYNC,
stashed[1],
stashed[2],
stashed[3],
stashed[4],
)
stashed = None
if stashed:
yield stashed
stashed = None
yield tok
elif initial == "\\": # continued stmt
# This yield is new; needed for better idempotency:
if stashed:
yield stashed
stashed = None
yield (NL, token, spos, (lnum, pos), line)
continued = 1
else:
if initial in "([{":
parenlev += 1
elif initial in ")]}":
parenlev -= 1
if stashed:
yield stashed
stashed = None
yield (OP, token, spos, epos, line)
else:
yield (ERRORTOKEN, line[pos], (lnum, pos), (lnum, pos + 1), line)
pos += 1
if stashed:
yield stashed
stashed = None
for _indent in indents[1:]: # pop remaining indent levels
yield (DEDENT, "", (lnum, 0), (lnum, 0), "")
yield (ENDMARKER, "", (lnum, 0), (lnum, 0), "")
if __name__ == "__main__": # testing
if len(sys.argv) > 1:
token_iterator = tokenize(open(sys.argv[1]).read())
tokenize(open(sys.argv[1]).readline)
else:
token_iterator = tokenize(sys.stdin.read())
for tok in token_iterator:
printtoken(*tok)
tokenize(sys.stdin.readline)

4
src/blib2to3/pygram.py
View File

@ -70,10 +70,6 @@ class _python_symbols(Symbols):
file_input: int
flow_stmt: int
for_stmt: int
fstring: int
fstring_format_spec: int
fstring_middle: int
fstring_replacement_field: int
funcdef: int
global_stmt: int
guard: int

92
src/blib2to3/pytree.py
View File

@ -12,8 +12,18 @@
# mypy: allow-untyped-defs, allow-incomplete-defs
from collections.abc import Iterable, Iterator
from typing import Any, Optional, TypeVar, Union
from typing import (
Any,
Dict,
Iterable,
Iterator,
List,
Optional,
Set,
Tuple,
TypeVar,
Union,
)
from blib2to3.pgen2.grammar import Grammar
@ -24,21 +34,18 @@
HUGE: int = 0x7FFFFFFF # maximum repeat count, default max
_type_reprs: dict[int, Union[str, int]] = {}
_type_reprs: Dict[int, Union[str, int]] = {}
def type_repr(type_num: int) -> Union[str, int]:
global _type_reprs
if not _type_reprs:
from . import pygram
if not hasattr(pygram, "python_symbols"):
pygram.initialize(cache_dir=None)
from .pygram import python_symbols
# printing tokens is possible but not as useful
# from .pgen2 import token // token.__dict__.items():
for name in dir(pygram.python_symbols):
val = getattr(pygram.python_symbols, name)
for name in dir(python_symbols):
val = getattr(python_symbols, name)
if type(val) == int:
_type_reprs[val] = name
return _type_reprs.setdefault(type_num, type_num)
@ -47,8 +54,8 @@ def type_repr(type_num: int) -> Union[str, int]:
_P = TypeVar("_P", bound="Base")
NL = Union["Node", "Leaf"]
Context = tuple[str, tuple[int, int]]
RawNode = tuple[int, Optional[str], Optional[Context], Optional[list[NL]]]
Context = Tuple[str, Tuple[int, int]]
RawNode = Tuple[int, Optional[str], Optional[Context], Optional[List[NL]]]
class Base:
@ -64,7 +71,7 @@ class Base:
# Default values for instance variables
type: int # int: token number (< 256) or symbol number (>= 256)
parent: Optional["Node"] = None # Parent node pointer, or None
children: list[NL] # List of subnodes
children: List[NL] # List of subnodes
was_changed: bool = False
was_checked: bool = False
@ -125,7 +132,7 @@ def pre_order(self) -> Iterator[NL]:
"""
raise NotImplementedError
def replace(self, new: Union[NL, list[NL]]) -> None:
def replace(self, new: Union[NL, List[NL]]) -> None:
"""Replace this node with a new one in the parent."""
assert self.parent is not None, str(self)
assert new is not None
@ -232,16 +239,16 @@ def get_suffix(self) -> str:
class Node(Base):
"""Concrete implementation for interior nodes."""
fixers_applied: Optional[list[Any]]
used_names: Optional[set[str]]
fixers_applied: Optional[List[Any]]
used_names: Optional[Set[str]]
def __init__(
self,
type: int,
children: list[NL],
children: List[NL],
context: Optional[Any] = None,
prefix: Optional[str] = None,
fixers_applied: Optional[list[Any]] = None,
fixers_applied: Optional[List[Any]] = None,
) -> None:
"""
Initializer.
@ -268,7 +275,11 @@ def __init__(
def __repr__(self) -> str:
"""Return a canonical string representation."""
assert self.type is not None
return f"{self.__class__.__name__}({type_repr(self.type)}, {self.children!r})"
return "{}({}, {!r})".format(
self.__class__.__name__,
type_repr(self.type),
self.children,
)
def __str__(self) -> str:
"""
@ -349,12 +360,12 @@ def append_child(self, child: NL) -> None:
self.invalidate_sibling_maps()
def invalidate_sibling_maps(self) -> None:
self.prev_sibling_map: Optional[dict[int, Optional[NL]]] = None
self.next_sibling_map: Optional[dict[int, Optional[NL]]] = None
self.prev_sibling_map: Optional[Dict[int, Optional[NL]]] = None
self.next_sibling_map: Optional[Dict[int, Optional[NL]]] = None
def update_sibling_maps(self) -> None:
_prev: dict[int, Optional[NL]] = {}
_next: dict[int, Optional[NL]] = {}
_prev: Dict[int, Optional[NL]] = {}
_next: Dict[int, Optional[NL]] = {}
self.prev_sibling_map = _prev
self.next_sibling_map = _next
previous: Optional[NL] = None
@ -370,11 +381,11 @@ class Leaf(Base):
# Default values for instance variables
value: str
fixers_applied: list[Any]
fixers_applied: List[Any]
bracket_depth: int
# Changed later in brackets.py
opening_bracket: Optional["Leaf"] = None
used_names: Optional[set[str]]
used_names: Optional[Set[str]]
_prefix = "" # Whitespace and comments preceding this token in the input
lineno: int = 0 # Line where this token starts in the input
column: int = 0 # Column where this token starts in the input
@ -389,7 +400,7 @@ def __init__(
value: str,
context: Optional[Context] = None,
prefix: Optional[str] = None,
fixers_applied: list[Any] = [],
fixers_applied: List[Any] = [],
opening_bracket: Optional["Leaf"] = None,
fmt_pass_converted_first_leaf: Optional["Leaf"] = None,
) -> None:
@ -407,7 +418,7 @@ def __init__(
self.value = value
if prefix is not None:
self._prefix = prefix
self.fixers_applied: Optional[list[Any]] = fixers_applied[:]
self.fixers_applied: Optional[List[Any]] = fixers_applied[:]
self.children = []
self.opening_bracket = opening_bracket
self.fmt_pass_converted_first_leaf = fmt_pass_converted_first_leaf
@ -417,9 +428,10 @@ def __repr__(self) -> str:
from .pgen2.token import tok_name
assert self.type is not None
return (
f"{self.__class__.__name__}({tok_name.get(self.type, self.type)},"
f" {self.value!r})"
return "{}({}, {!r})".format(
self.__class__.__name__,
tok_name.get(self.type, self.type),
self.value,
)
def __str__(self) -> str:
@ -488,7 +500,7 @@ def convert(gr: Grammar, raw_node: RawNode) -> NL:
return Leaf(type, value or "", context=context)
_Results = dict[str, NL]
_Results = Dict[str, NL]
class BasePattern:
@ -522,7 +534,7 @@ def __repr__(self) -> str:
args = [type_repr(self.type), self.content, self.name]
while args and args[-1] is None:
del args[-1]
return f"{self.__class__.__name__}({', '.join(map(repr, args))})"
return "{}({})".format(self.__class__.__name__, ", ".join(map(repr, args)))
def _submatch(self, node, results=None) -> bool:
raise NotImplementedError
@ -561,7 +573,7 @@ def match(self, node: NL, results: Optional[_Results] = None) -> bool:
results[self.name] = node
return True
def match_seq(self, nodes: list[NL], results: Optional[_Results] = None) -> bool:
def match_seq(self, nodes: List[NL], results: Optional[_Results] = None) -> bool:
"""
Does this pattern exactly match a sequence of nodes?
@ -571,7 +583,7 @@ def match_seq(self, nodes: list[NL], results: Optional[_Results] = None) -> bool
return False
return self.match(nodes[0], results)
def generate_matches(self, nodes: list[NL]) -> Iterator[tuple[int, _Results]]:
def generate_matches(self, nodes: List[NL]) -> Iterator[Tuple[int, _Results]]:
"""
Generator yielding all matches for this pattern.
@ -801,7 +813,7 @@ def match_seq(self, nodes, results=None) -> bool:
return True
return False
def generate_matches(self, nodes) -> Iterator[tuple[int, _Results]]:
def generate_matches(self, nodes) -> Iterator[Tuple[int, _Results]]:
"""
Generator yielding matches for a sequence of nodes.
@ -846,7 +858,7 @@ def generate_matches(self, nodes) -> Iterator[tuple[int, _Results]]:
if hasattr(sys, "getrefcount"):
sys.stderr = save_stderr
def _iterative_matches(self, nodes) -> Iterator[tuple[int, _Results]]:
def _iterative_matches(self, nodes) -> Iterator[Tuple[int, _Results]]:
"""Helper to iteratively yield the matches."""
nodelen = len(nodes)
if 0 >= self.min:
@ -875,7 +887,7 @@ def _iterative_matches(self, nodes) -> Iterator[tuple[int, _Results]]:
new_results.append((c0 + c1, r))
results = new_results
def _bare_name_matches(self, nodes) -> tuple[int, _Results]:
def _bare_name_matches(self, nodes) -> Tuple[int, _Results]:
"""Special optimized matcher for bare_name."""
count = 0
r = {} # type: _Results
@ -892,7 +904,7 @@ def _bare_name_matches(self, nodes) -> tuple[int, _Results]:
r[self.name] = nodes[:count]
return count, r
def _recursive_matches(self, nodes, count) -> Iterator[tuple[int, _Results]]:
def _recursive_matches(self, nodes, count) -> Iterator[Tuple[int, _Results]]:
"""Helper to recursively yield the matches."""
assert self.content is not None
if count >= self.min:
@ -929,7 +941,7 @@ def match_seq(self, nodes, results=None) -> bool:
# We only match an empty sequence of nodes in its entirety
return len(nodes) == 0
def generate_matches(self, nodes: list[NL]) -> Iterator[tuple[int, _Results]]:
def generate_matches(self, nodes: List[NL]) -> Iterator[Tuple[int, _Results]]:
if self.content is None:
# Return a match if there is an empty sequence
if len(nodes) == 0:
@ -942,8 +954,8 @@ def generate_matches(self, nodes: list[NL]) -> Iterator[tuple[int, _Results]]:
def generate_matches(
patterns: list[BasePattern], nodes: list[NL]
) -> Iterator[tuple[int, _Results]]:
patterns: List[BasePattern], nodes: List[NL]
) -> Iterator[Tuple[int, _Results]]:
"""
Generator yielding matches for a sequence of patterns and nodes.

17
tests/data/cases/annotations.py
View File

@ -1,17 +0,0 @@
# regression test for #1765
class Foo:
def foo(self):
if True:
content_ids: Mapping[
str, Optional[ContentId]
] = self.publisher_content_store.store_config_contents(files)
# output
# regression test for #1765
class Foo:
def foo(self):
if True:
content_ids: Mapping[str, Optional[ContentId]] = (
self.publisher_content_store.store_config_contents(files)
)

24
tests/data/cases/backslash_before_indent.py
View File

@ -1,24 +0,0 @@
# flags: --minimum-version=3.10
class Plotter:
\
pass
class AnotherCase:
\
"""Some
\
Docstring
"""
# output
class Plotter:
pass
class AnotherCase:
"""Some
\
Docstring
"""

Some files were not shown because too many files have changed in this diff Show More