Switch build backend to Hatchling (#3233)

This implements PEP 621, obviating the need for `setup.py`, `setup.cfg`,
and `MANIFEST.in`. The build backend Hatchling (of which I am a
maintainer in the PyPA) is now used as that is the default in the
official Python packaging tutorial. Hatchling is available on all the
major distribution channels such as Debian, Fedora, and many more.

## Python support

The earliest supported Python 3 version of Hatchling is 3.7, therefore
I've also set that as the minimum here. Python 3.6 is EOL and other
build backends like flit-core and setuptools also dropped support.
Python 3.6 accounted for 3-4% of downloads in the last month.

## Plugins 

Configuration is now completely static with the help of 3 plugins:

### Readme

hynek's hatch-fancy-pypi-readme allows for the dynamic construction of
the readme which was previously coded up in `setup.py`. Now it's simply:

```toml
[tool.hatch.metadata.hooks.fancy-pypi-readme]
content-type = "text/markdown"
fragments = [
  { path = "README.md" },
  { path = "CHANGES.md" },
]
```

### Versioning

hatch-vcs is currently just a wrapper around setuptools-scm (which
despite the legacy naming is actually now decoupled from setuptools):

```toml
[tool.hatch.version]
source = "vcs"

[tool.hatch.build.hooks.vcs]
version-file = "src/_black_version.py"
template = '''
version = "{version}"
'''
```

### mypyc

hatch-mypyc offers many benefits over the existing approach:

- No need to manually select files for inclusion
- Avoids the need for the current CI workaround for https://github.com/mypyc/mypyc/issues/946
- Intermediate artifacts (like `build/`) from setuptools and mypyc
  itself no longer clutter the project directory
- Runtime dependencies required at build time no longer need to be
  manually redeclared as this is a built-in option of Hatchling

Co-authored-by: Richard Si <63936253+ichard26@users.noreply.github.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

.github/mypyc-requirements.txt

@@ -1,14 +0,0 @@
-mypy == 0.971
-
-# A bunch of packages for type information
-mypy-extensions >= 0.4.3
-tomli >= 0.10.2
-types-typed-ast >= 1.4.2
-types-dataclasses >= 0.1.3
-typing-extensions > 3.10.0.1
-click >= 8.0.0
-platformdirs >= 2.1.0
-
-# And because build isolation is disabled, we'll need to pull this too
-setuptools-scm[toml] >= 6.3.1
-wheel

.github/workflows/diff_shades.yml

@@ -3,10 +3,10 @@ name: diff-shades
 on:
   push:
     branches: [main]
-    paths: ["src/**", "setup.*", "pyproject.toml", ".github/workflows/*"]
+    paths: ["src/**", "pyproject.toml", ".github/workflows/*"]
 
   pull_request:
-    paths: ["src/**", "setup.*", "pyproject.toml", ".github/workflows/*"]
+    paths: ["src/**", "pyproject.toml", ".github/workflows/*"]
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
@@ -41,6 +41,7 @@ jobs:
     needs: configure
     runs-on: ubuntu-latest
     env:
+      HATCH_BUILD_HOOKS_ENABLE: "1"
       # Clang is less picky with the C code it's given than gcc (and may
       # generate faster binaries too).
       CC: clang-12
@@ -64,7 +65,6 @@ jobs:
         run: |
           python -m pip install https://github.com/ichard26/diff-shades/archive/stable.zip
           python -m pip install click packaging urllib3
-          python -m pip install -r .github/mypyc-requirements.txt
           # After checking out old revisions, this might not exist so we'll use a copy.
           cat scripts/diff_shades_gha_helper.py > helper.py
           git config user.name "diff-shades-gha"
@@ -83,8 +83,7 @@ jobs:
           GITHUB_TOKEN: ${{ github.token }}
         run: >
           ${{ matrix.baseline-setup-cmd }}
-          && python setup.py --use-mypyc bdist_wheel
-          && python -m pip install dist/*.whl && rm build dist -r
+          && python -m pip install .
 
       - name: Analyze baseline revision
         if: steps.baseline-cache.outputs.cache-hit != 'true'
@@ -97,8 +96,7 @@ jobs:
           GITHUB_TOKEN: ${{ github.token }}
         run: >
           ${{ matrix.target-setup-cmd }}
-          && python setup.py --use-mypyc bdist_wheel
-          && python -m pip install dist/*.whl
+          && python -m pip install .
 
       - name: Analyze target revision
         run: >

.github/workflows/fuzz.yml

@@ -22,7 +22,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.6", "3.7", "3.8", "3.9", "3.10"]
+        python-version: ["3.7", "3.8", "3.9", "3.10"]
 
     steps:
       - uses: actions/checkout@v3

.github/workflows/pypi_upload.yml

@@ -61,8 +61,6 @@ jobs:
         uses: pypa/cibuildwheel@v2.10.0
         env:
           CIBW_ARCHS_MACOS: "${{ matrix.macos_arch }}"
-          # This isn't supported in pyproject.toml which makes sense (but is annoying).
-          CIBW_PROJECT_REQUIRES_PYTHON: ">=3.6.2"
 
       - name: Upload wheels as workflow artifacts
         uses: actions/upload-artifact@v2

.github/workflows/test.yml

@@ -31,7 +31,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.6", "3.7", "3.8", "3.9", "3.10", "pypy-3.7", "pypy-3.8"]
+        python-version: ["3.7", "3.8", "3.9", "3.10", "pypy-3.7", "pypy-3.8"]
         os: [ubuntu-latest, macOS-latest, windows-latest]
 
     steps:

CHANGES.md

@@ -6,6 +6,9 @@
 
 <!-- Include any especially major or disruptive changes here -->
 
+- Runtime support for Python 3.6 has been removed. Formatting 3.6 code will still be
+  supported until further notice.
+
 ### Stable style
 
 <!-- Changes that affect Black's stable style -->
@@ -28,6 +31,8 @@
 
 <!-- Changes to how Black is packaged, such as dependency requirements -->
 
+- Hatchling is now used as the build backend. This will not have any effect for users
+  who install Black with its wheels from PyPI. (#3233)
 - Faster compiled wheels are now available for CPython 3.11 (#3276)
 
 ### Parser

MANIFEST.in

@@ -1 +0,0 @@
-prune profiling

docs/faq.md

@@ -86,8 +86,8 @@ disabled-by-default counterpart W504. E203 should be disabled while changes are
 
 ## Which Python versions does Black support?
 
-Currently the runtime requires Python 3.6-3.10. Formatting is supported for files
-containing syntax from Python 3.3 to 3.10. We promise to support at least all Python
+Currently the runtime requires Python 3.7-3.11. Formatting is supported for files
+containing syntax from Python 3.3 to 3.11. We promise to support at least all Python
 versions that have not reached their end of life. This is the case for both running
 _Black_ and formatting code.
 
@@ -95,6 +95,8 @@ Support for formatting Python 2 code was removed in version 22.0. While we've ma
 plans to stop supporting older Python 3 minor versions immediately, their support might
 also be removed some time in the future without a deprecation period.
 
+Runtime support for 3.6 was removed in version 22.9.0.
+
 ## Why does my linter or typechecker complain after I format my code?
 
 Some linters and other tools use magical comments (e.g., `# noqa`, `# type: ignore`) to

docs/usage_and_configuration/the_basics.md

@@ -204,9 +204,10 @@ code in compliance with many other _Black_ formatted projects.
 
 [PEP 518](https://www.python.org/dev/peps/pep-0518/) defines `pyproject.toml` as a
 configuration file to store build system requirements for Python projects. With the help
-of tools like [Poetry](https://python-poetry.org/) or
-[Flit](https://flit.readthedocs.io/en/latest/) it can fully replace the need for
-`setup.py` and `setup.cfg` files.
+of tools like [Poetry](https://python-poetry.org/),
+[Flit](https://flit.readthedocs.io/en/latest/), or
+[Hatch](https://hatch.pypa.io/latest/) it can fully replace the need for `setup.py` and
+`setup.cfg` files.
 
 ### Where _Black_ looks for the file
 

pyproject.toml

@@ -7,7 +7,7 @@
 
 [tool.black]
 line-length = 88
-target-version = ['py36', 'py37', 'py38']
+target-version = ['py37', 'py38']
 include = '\.pyi?$'
 extend-exclude = '''
 /(
@@ -26,18 +26,128 @@ preview = true
 # NOTE: You don't need this in your own Black configuration.
 
 [build-system]
-requires = ["setuptools>=45.0", "setuptools_scm[toml]>=6.3.1", "wheel"]
-build-backend = "setuptools.build_meta"
+requires = ["hatchling>=1.8.0", "hatch-vcs", "hatch-fancy-pypi-readme"]
+build-backend = "hatchling.build"
+
+[project]
+name = "black"
+description = "The uncompromising code formatter."
+license = "MIT"
+requires-python = ">=3.7"
+authors = [
+  { name = "Łukasz Langa", email = "lukasz@langa.pl" },
+]
+keywords = [
+  "automation",
+  "autopep8",
+  "formatter",
+  "gofmt",
+  "pyfmt",
+  "rustfmt",
+  "yapf",
+]
+classifiers = [
+  "Development Status :: 5 - Production/Stable",
+  "Environment :: Console",
+  "Intended Audience :: Developers",
+  "License :: OSI Approved :: MIT License",
+  "Operating System :: OS Independent",
+  "Programming Language :: Python",
+  "Programming Language :: Python :: 3 :: Only",
+  "Programming Language :: Python :: 3.7",
+  "Programming Language :: Python :: 3.8",
+  "Programming Language :: Python :: 3.9",
+  "Programming Language :: Python :: 3.10",
+  "Topic :: Software Development :: Libraries :: Python Modules",
+  "Topic :: Software Development :: Quality Assurance",
+]
+dependencies = [
+  "click>=8.0.0",
+  "mypy_extensions>=0.4.3",
+  "pathspec>=0.9.0",
+  "platformdirs>=2",
+  "tomli>=1.1.0; python_full_version < '3.11.0a7'",
+  "typed-ast>=1.4.2; python_version < '3.8' and implementation_name == 'cpython'",
+  "typing_extensions>=3.10.0.0; python_version < '3.10'",
+]
+dynamic = ["readme", "version"]
+
+[project.optional-dependencies]
+colorama = ["colorama>=0.4.3"]
+uvloop = ["uvloop>=0.15.2"]
+d = [
+  "aiohttp>=3.7.4",
+]
+jupyter = [
+  "ipython>=7.8.0",
+  "tokenize-rt>=3.2.0",
+]
+
+[project.scripts]
+black = "black:patched_main"
+blackd = "blackd:patched_main [d]"
+
+[project.urls]
+Changelog = "https://github.com/psf/black/blob/main/CHANGES.md"
+Homepage = "https://github.com/psf/black"
+
+[tool.hatch.metadata.hooks.fancy-pypi-readme]
+content-type = "text/markdown"
+fragments = [
+  { path = "README.md" },
+  { path = "CHANGES.md" },
+]
+
+[tool.hatch.version]
+source = "vcs"
+
+[tool.hatch.build.hooks.vcs]
+version-file = "src/_black_version.py"
+template = '''
+version = "{version}"
+'''
+
+[tool.hatch.build.targets.sdist]
+exclude = ["/profiling"]
+
+[tool.hatch.build.targets.wheel]
+only-include = ["src"]
+sources = ["src"]
+
+[tool.hatch.build.targets.wheel.hooks.mypyc]
+enable-by-default = false
+dependencies = [
+  "hatch-mypyc>=0.13.0",
+  "mypy==0.971",
+  # Required stubs to be removed when the packages support PEP 561 themselves
+  "types-typed-ast>=1.4.2",
+]
+require-runtime-dependencies = true
+exclude = [
+  # There's no good reason for blackd to be compiled.
+  "/src/blackd",
+  # Not performance sensitive, so save bytes + compilation time:
+  "/src/blib2to3/__init__.py",
+  "/src/blib2to3/pgen2/__init__.py",
+  "/src/black/output.py",
+  "/src/black/concurrency.py",
+  "/src/black/files.py",
+  "/src/black/report.py",
+  # Breaks the test suite when compiled (and is also useless):
+  "/src/black/debug.py",
+  # Compiled modules can't be run directly and that's a problem here:
+  "/src/black/__main__.py",
+]
+options = { debug_level = "0" }
 
 [tool.cibuildwheel]
 build-verbosity = 1
 # So these are the environments we target:
-# - Python: CPython 3.6+ only
+# - Python: CPython 3.7+ only
 # - Architecture (64-bit only): amd64 / x86_64, universal2, and arm64
 # - OS: Linux (no musl), Windows, and macOS
 build = "cp3*-*"
 skip = ["*-manylinux_i686", "*-musllinux_*", "*-win32", "pp-*"]
-before-build = ["pip install -r .github/mypyc-requirements.txt"]
 # This is the bare minimum needed to run the test suite. Pulling in the full
 # test_requirements.txt would download a bunch of other packages not necessary
 # here and would slow down the testing step a fair bit.
@@ -49,7 +159,7 @@ test-extras = ["d"," jupyter"]
 test-skip = ["*-macosx_arm64", "*-macosx_universal2:arm64"]
 
 [tool.cibuildwheel.environment]
-BLACK_USE_MYPYC = "1"
+HATCH_BUILD_HOOKS_ENABLE = "1"
 MYPYC_OPT_LEVEL = "3"
 MYPYC_DEBUG_LEVEL = "0"
 # The dependencies required to build wheels with mypyc aren't specified in
@@ -61,28 +171,17 @@ AIOHTTP_NO_EXTENSIONS = "1"
 
 [tool.cibuildwheel.linux]
 before-build = [
-    "pip install -r .github/mypyc-requirements.txt",
     "yum install -y clang gcc",
 ]
 
 [tool.cibuildwheel.linux.environment]
-BLACK_USE_MYPYC = "1"
+HATCH_BUILD_HOOKS_ENABLE = "1"
 MYPYC_OPT_LEVEL = "3"
 MYPYC_DEBUG_LEVEL = "0"
-PIP_NO_BUILD_ISOLATION = "no"
 # Black needs Clang to compile successfully on Linux.
 CC = "clang"
 AIOHTTP_NO_EXTENSIONS = "1"
 
-[tool.cibuildwheel.windows]
-# For some reason, (compiled) mypyc is failing to start up with "ImportError: DLL load
-# failed: A dynamic link library (DLL) initialization routine failed." on Windows for
-# at least 3.6. Let's just use interpreted mypy[c].
-# See also: https://github.com/mypyc/mypyc/issues/819.
-before-build = [
-    "pip install -r .github/mypyc-requirements.txt --no-binary mypy"
-]
-
 [tool.isort]
 atomic = true
 profile = "black"

setup.cfg

@@ -1,3 +0,0 @@
-[options]
-setup_requires =
-  setuptools_scm[toml]>=6.3.1

setup.py

@@ -1,141 +0,0 @@
-# Copyright (C) 2020 Łukasz Langa
-import os
-import sys
-
-from setuptools import find_packages, setup
-
-assert sys.version_info >= (3, 6, 2), "black requires Python 3.6.2+"
-from pathlib import Path  # noqa E402
-from typing import List  # noqa: E402
-
-CURRENT_DIR = Path(__file__).parent
-sys.path.insert(0, str(CURRENT_DIR))  # for setuptools.build_meta
-
-
-def get_long_description() -> str:
-    return (
-        (CURRENT_DIR / "README.md").read_text(encoding="utf8")
-        + "\n\n"
-        + (CURRENT_DIR / "CHANGES.md").read_text(encoding="utf8")
-    )
-
-
-def find_python_files(base: Path) -> List[Path]:
-    files = []
-    for entry in base.iterdir():
-        if entry.is_file() and entry.suffix == ".py":
-            files.append(entry)
-        elif entry.is_dir():
-            files.extend(find_python_files(entry))
-
-    return files
-
-
-USE_MYPYC = False
-# To compile with mypyc, a mypyc checkout must be present on the PYTHONPATH
-if len(sys.argv) > 1 and sys.argv[1] == "--use-mypyc":
-    sys.argv.pop(1)
-    USE_MYPYC = True
-if os.getenv("BLACK_USE_MYPYC", None) == "1":
-    USE_MYPYC = True
-
-if USE_MYPYC:
-    from mypyc.build import mypycify
-
-    src = CURRENT_DIR / "src"
-    # TIP: filepaths are normalized to use forward slashes and are relative to ./src/
-    # before being checked against.
-    blocklist = [
-        # Not performance sensitive, so save bytes + compilation time:
-        "blib2to3/__init__.py",
-        "blib2to3/pgen2/__init__.py",
-        "black/output.py",
-        "black/concurrency.py",
-        "black/files.py",
-        "black/report.py",
-        # Breaks the test suite when compiled (and is also useless):
-        "black/debug.py",
-        # Compiled modules can't be run directly and that's a problem here:
-        "black/__main__.py",
-    ]
-    discovered = []
-    # There's no good reason for blackd to be compiled.
-    discovered.extend(find_python_files(src / "black"))
-    discovered.extend(find_python_files(src / "blib2to3"))
-    mypyc_targets = [
-        str(p) for p in discovered if p.relative_to(src).as_posix() not in blocklist
-    ]
-
-    opt_level = os.getenv("MYPYC_OPT_LEVEL", "3")
-    debug_level = os.getenv("MYPYC_DEBUG_LEVEL", "3")
-    ext_modules = mypycify(
-        mypyc_targets, opt_level=opt_level, debug_level=debug_level, verbose=True
-    )
-else:
-    ext_modules = []
-
-setup(
-    name="black",
-    use_scm_version={
-        "write_to": "src/_black_version.py",
-        "write_to_template": 'version = "{version}"\n',
-    },
-    description="The uncompromising code formatter.",
-    long_description=get_long_description(),
-    long_description_content_type="text/markdown",
-    keywords="automation formatter yapf autopep8 pyfmt gofmt rustfmt",
-    author="Łukasz Langa",
-    author_email="lukasz@langa.pl",
-    url="https://github.com/psf/black",
-    project_urls={"Changelog": "https://github.com/psf/black/blob/main/CHANGES.md"},
-    license="MIT",
-    py_modules=["_black_version"],
-    ext_modules=ext_modules,
-    packages=find_packages(where="src"),
-    package_dir={"": "src"},
-    package_data={
-        "blib2to3": ["*.txt"],
-        "black": ["py.typed"],
-    },
-    python_requires=">=3.6.2",
-    zip_safe=False,
-    install_requires=[
-        "click>=8.0.0",
-        "platformdirs>=2",
-        "tomli>=1.1.0; python_full_version < '3.11.0a7'",
-        "typed-ast>=1.4.2; python_version < '3.8' and implementation_name == 'cpython'",
-        "pathspec>=0.9.0",
-        "dataclasses>=0.6; python_version < '3.7'",
-        "typing_extensions>=3.10.0.0; python_version < '3.10'",
-        "mypy_extensions>=0.4.3",
-    ],
-    extras_require={
-        "d": ["aiohttp>=3.7.4"],
-        "colorama": ["colorama>=0.4.3"],
-        "uvloop": ["uvloop>=0.15.2"],
-        "jupyter": ["ipython>=7.8.0", "tokenize-rt>=3.2.0"],
-    },
-    classifiers=[
-        "Development Status :: 5 - Production/Stable",
-        "Environment :: Console",
-        "Intended Audience :: Developers",
-        "License :: OSI Approved :: MIT License",
-        "Operating System :: OS Independent",
-        "Programming Language :: Python",
-        "Programming Language :: Python :: 3.6",
-        "Programming Language :: Python :: 3.7",
-        "Programming Language :: Python :: 3.8",
-        "Programming Language :: Python :: 3.9",
-        "Programming Language :: Python :: 3.10",
-        "Programming Language :: Python :: 3.11",
-        "Programming Language :: Python :: 3 :: Only",
-        "Topic :: Software Development :: Libraries :: Python Modules",
-        "Topic :: Software Development :: Quality Assurance",
-    ],
-    entry_points={
-        "console_scripts": [
-            "black=black:patched_main",
-            "blackd=blackd:patched_main [d]",
-        ]
-    },
-)

tox.ini

@@ -1,5 +1,6 @@
 [tox]
-envlist = {,ci-}py{36,37,38,39,310,311,py3},fuzz,run_self
+isolated_build = true
+envlist = {,ci-}py{37,38,39,310,311,py3},fuzz,run_self
 
 [testenv]
 setenv = PYTHONPATH = {toxinidir}/src
@@ -96,4 +97,4 @@ setenv = PYTHONPATH = {toxinidir}/src
 skip_install = True
 commands =
     pip install -e .[d]
-    black --check {toxinidir}/src {toxinidir}/tests {toxinidir}/setup.py
+    black --check {toxinidir}/src {toxinidir}/tests