actions/black
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add release tool (#3974)

Browse Source 
* Add release tool

- Add tool for release managers to use to generate commits
  - I'm trying to only use stdlib so we have no depdencies ...
- Default is to change date strings in hard coded documentation files + CHANGES.md
  - I write directly to files cause we have SCM to fix any screw ups ...
- We hackily convert calver to ints to sort (all for better ideas here)
  - If we hit a ValueError we just set to 0 for sorting - This is alhpa + beta release we can safely ignore these days
- Add new CI to only run release unittests in 3.12 only on all platforms
- Update release docs

- Checked with `mypy --strict` + ensure we are `black --preview` formatted :D

Tests:
- Run it to generate template PR
  - `python3.12 release.py --debug --add-changes-template`
- Run it to cleanup CHANGE.md + change version in specified doc files
```
crl-m1:black cooper$ python3.12 release.py -d
[2023-10-23 23:39:38,414] INFO: Current version detected to be 23.10.1 (release.py:221)
[2023-10-23 23:39:38,414] INFO: Next version will be 23.10.2 (release.py:222)
[2023-10-23 23:39:38,414] INFO: Cleaning up /Users/cooper/repos/black/CHANGES.md (release.py:127)
[2023-10-23 23:39:38,416] DEBUG: Finished Cleaning up /Users/cooper/repos/black/CHANGES.md (release.py:147)
[2023-10-23 23:39:38,416] INFO: Updating black version to 23.10.2 in /Users/cooper/repos/black/docs/integrations/source_version_control.md (release.py:173)
[2023-10-23 23:39:38,416] DEBUG: Finished updating black version to 23.10.2 in /Users/cooper/repos/black/docs/integrations/source_version_control.md (release.py:185)
[2023-10-23 23:39:38,416] INFO: Updating black version to 23.10.2 in /Users/cooper/repos/black/docs/usage_and_configuration/the_basics.md (release.py:173)
[2023-10-23 23:39:38,417] DEBUG: Finished updating black version to 23.10.2 in /Users/cooper/repos/black/docs/usage_and_configuration/the_basics.md (release.py:185)
```
- Add tests around some key logic

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix lints + add git to release CI

- Remove black + mypy as linting already runs it ...
- Ignore delete param to TemporaryDirectory as we can't set mypy to 3.12 :D

* Only run CI on linux/ubuntu for now

* Add lots of debug printing + directly run unitests (not via coverage)

* Overloading __str__ is bad on a TestCase

* Add more logging around git tag

* Print where git is in a step

* Rollback creating a fake black repo as we were not using it - I did plan to but I can't get it working on GitHub actions

* Do a deep checkout

* Add noqa for E701,E761 ... maybe we need this in our flake8 config now?

* Fix action to have correct workflow yaml to action on
- Also add fix to not double run when we push directly to psf/black

* All jelle suggestions
- Fix bug missing lines ending with --> in CHANGES.md to delete ...
- Update ci to run out of scripts dir too
- Update test_tuple_calver

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
This commit is contained in:
Cooper Lees 2023-10-29 15:34:38 -07:00 committed by GitHub
parent 5515a5ac7a
commit f7cbe4ae1b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
4 changed files with 383 additions and 55 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

56
.github/workflows/release_tests.yml vendored Normal file
View File

@ -0,0 +1,56 @@
name: Release tool CI
on:
push:
paths:
- .github/workflows/release_tests.yml
- release.py
- release_tests.py
pull_request:
paths:
- .github/workflows/release_tests.yml
- release.py
- release_tests.py
jobs:
build:
# We want to run on external PRs, but not on our own internal PRs as they'll be run
# by the push to the branch. Without this if check, checks are duplicated since
# internal PRs match both the push and pull_request events.
if:
github.event_name == 'push' || github.event.pull_request.head.repo.full_name !=
github.repository
name: Running python ${{ matrix.python-version }} on ${{matrix.os}}
runs-on: ${{ matrix.os }}
strategy:
matrix:
python-version: ["3.12"]
os: [macOS-latest, ubuntu-latest, windows-latest]
steps:
- uses: actions/checkout@v4
with:
# Give us all history, branches and tags
fetch-depth: 0
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
allow-prereleases: true
- name: Print Python Version
run: python --version --version && which python
- name: Print Git Version
run: git --version && which git
- name: Update pip, setuptools + wheels
run: |
python -m pip install --upgrade pip setuptools wheel
- name: Run unit tests via coverage + print report
run: |
python -m pip install coverage
coverage run scripts/release_tests.py
coverage report --show-missing

70
docs/contributing/release_process.md
View File

@ -32,21 +32,29 @@ The 10,000 foot view of the release process is that you prepare a release PR and
publish a [GitHub Release]. This triggers [release automation](#release-workflows) that
builds all release artifacts and publishes them to the various platforms we publish to.
We now have a `scripts/release.py` script to help with cutting the release PRs.
- `python3 scripts/release.py --help` is your friend.
- `release.py` has only been tested in Python 3.12 (so get with the times :D)
To cut a release:
1. Determine the release's version number
- **_Black_ follows the [CalVer] versioning standard using the `YY.M.N` format**
- So unless there already has been a release during this month, `N` should be `0`
- Example: the first release in January, 2022 → `22.1.0`
- `release.py` will calculate this and log to stderr for you copy paste pleasure
1. File a PR editing `CHANGES.md` and the docs to version the latest changes
- Run `python3 scripts/release.py [--debug]` to generate most changes
- Sub headings in the template, if they have no bullet points need manual removal
_PR welcome to improve :D_
1. If `release.py` fail manually edit; otherwise, yay, skip this step!
1. Replace the `## Unreleased` header with the version number
1. Remove any empty sections for the current release
1. (_optional_) Read through and copy-edit the changelog (eg. by moving entries,
fixing typos, or rephrasing entries)
1. Double-check that no changelog entries since the last release were put in the
wrong section (e.g., run `git diff <last release> CHANGES.md`)
1. Add a new empty template for the next release above
([template below](#changelog-template))
1. Update references to the latest version in
{doc}`/integrations/source_version_control` and
{doc}`/usage_and_configuration/the_basics`
@ -63,6 +71,11 @@ To cut a release:
description box
1. Publish the GitHub Release, triggering [release automation](#release-workflows) that
will handle the rest
1. Once CI is done add + commit (git push - No review) a new empty template for the next
release to CHANGES.md _(Template is able to be copy pasted from release.py should we
fail)_
1. `python3 scripts/release.py --add-changes-template|-a [--debug]`
1. Should that fail, please return to copy + paste
1. At this point, you're basically done. It's good practice to go and [watch and verify
that all the release workflows pass][black-actions], although you will receive a
GitHub notification should something fail.
@ -81,59 +94,6 @@ release is probably unnecessary.
In the end, use your best judgement and ask other maintainers for their thoughts.
```
### Changelog template
Use the following template for a clean changelog after the release:
```
## Unreleased
### Highlights
<!-- Include any especially major or disruptive changes here -->
### Stable style
<!-- Changes that affect Black's stable style -->
### Preview style
<!-- Changes that affect Black's preview style -->
### Configuration
<!-- Changes to how Black can be configured -->
### Packaging
<!-- Changes to how Black is packaged, such as dependency requirements -->
### Parser
<!-- Changes to the parser or to version autodetection -->
### Performance
<!-- Changes that improve Black's performance. -->
### Output
<!-- Changes to Black's terminal output and error messages -->
### _Blackd_
<!-- Changes to blackd -->
### Integrations
<!-- For example, Docker, GitHub Actions, pre-commit, editors -->
### Documentation
<!-- Major changes to documentation and policies. Small docs changes
don't need a changelog entry. -->
```
## Release workflows
All of _Black_'s release automation uses [GitHub Actions]. All workflows are therefore

243
scripts/release.py Executable file
View File

@ -0,0 +1,243 @@
#!/usr/bin/env python3
from __future__ import annotations
"""
Tool to help automate changes needed in commits during and after releases
"""
import argparse
import logging
import sys
from datetime import datetime
from pathlib import Path
from subprocess import PIPE, run
from typing import List
LOG = logging.getLogger(__name__)
NEW_VERSION_CHANGELOG_TEMPLATE = """\
## Unreleased
### Highlights
<!-- Include any especially major or disruptive changes here -->
### Stable style
<!-- Changes that affect Black's stable style -->
### Preview style
<!-- Changes that affect Black's preview style -->
### Configuration
<!-- Changes to how Black can be configured -->
### Packaging
<!-- Changes to how Black is packaged, such as dependency requirements -->
### Parser
<!-- Changes to the parser or to version autodetection -->
### Performance
<!-- Changes that improve Black's performance. -->
### Output
<!-- Changes to Black's terminal output and error messages -->
### _Blackd_
<!-- Changes to blackd -->
### Integrations
<!-- For example, Docker, GitHub Actions, pre-commit, editors -->
### Documentation
<!-- Major changes to documentation and policies. Small docs changes
don't need a changelog entry. -->
"""
class NoGitTagsError(Exception): ... # noqa: E701,E761
# TODO: Do better with alpha + beta releases
# Maybe we vendor packaging library
def get_git_tags(versions_only: bool = True) -> List[str]:
"""Pull out all tags or calvers only"""
cp = run(["git", "tag"], stdout=PIPE, stderr=PIPE, check=True, encoding="utf8")
if not cp.stdout:
LOG.error(f"Returned no git tags stdout: {cp.stderr}")
raise NoGitTagsError
git_tags = cp.stdout.splitlines()
if versions_only:
return [t for t in git_tags if t[0].isdigit()]
return git_tags
# TODO: Support sorting alhpa/beta releases correctly
def tuple_calver(calver: str) -> tuple[int, ...]: # mypy can't notice maxsplit below
"""Convert a calver string into a tuple of ints for sorting"""
try:
return tuple(map(int, calver.split(".", maxsplit=2)))
except ValueError:
return (0, 0, 0)
class SourceFiles:
def __init__(self, black_repo_dir: Path):
# File path fun all pathlib to be platform agnostic
self.black_repo_path = black_repo_dir
self.changes_path = self.black_repo_path / "CHANGES.md"
self.docs_path = self.black_repo_path / "docs"
self.version_doc_paths = (
self.docs_path / "integrations" / "source_version_control.md",
self.docs_path / "usage_and_configuration" / "the_basics.md",
)
self.current_version = self.get_current_version()
self.next_version = self.get_next_version()
def __str__(self) -> str:
return f"""\
> SourceFiles ENV:
Repo path: {self.black_repo_path}
CHANGES.md path: {self.changes_path}
docs path: {self.docs_path}
Current version: {self.current_version}
Next version: {self.next_version}
"""
def add_template_to_changes(self) -> int:
"""Add the template to CHANGES.md if it does not exist"""
LOG.info(f"Adding template to {self.changes_path}")
with self.changes_path.open("r") as cfp:
changes_string = cfp.read()
if "## Unreleased" in changes_string:
LOG.error(f"{self.changes_path} already has unreleased template")
return 1
templated_changes_string = changes_string.replace(
"# Change Log\n",
f"# Change Log\n\n{NEW_VERSION_CHANGELOG_TEMPLATE}",
)
with self.changes_path.open("w") as cfp:
cfp.write(templated_changes_string)
LOG.info(f"Added template to {self.changes_path}")
return 0
def cleanup_changes_template_for_release(self) -> None:
LOG.info(f"Cleaning up {self.changes_path}")
with self.changes_path.open("r") as cfp:
changes_string = cfp.read()
# Change Unreleased to next version
versioned_changes = changes_string.replace(
"## Unreleased", f"## {self.next_version}"
)
# Remove all comments (subheadings are harder - Human required still)
no_comments_changes = []
for line in versioned_changes.splitlines():
if line.startswith("<!--") or line.endswith("-->"):
continue
no_comments_changes.append(line)
with self.changes_path.open("w") as cfp:
cfp.write("\n".join(no_comments_changes) + "\n")
LOG.debug(f"Finished Cleaning up {self.changes_path}")
def get_current_version(self) -> str:
"""Get the latest git (version) tag as latest version"""
return sorted(get_git_tags(), key=lambda k: tuple_calver(k))[-1]
def get_next_version(self) -> str:
"""Workout the year and month + version number we need to move to"""
base_calver = datetime.today().strftime("%y.%m")
calver_parts = base_calver.split(".")
base_calver = f"{calver_parts[0]}.{int(calver_parts[1])}" # Remove leading 0
git_tags = get_git_tags()
same_month_releases = [t for t in git_tags if t.startswith(base_calver)]
if len(same_month_releases) < 1:
return f"{base_calver}.0"
same_month_version = same_month_releases[-1].split(".", 2)[-1]
return f"{base_calver}.{int(same_month_version) + 1}"
def update_repo_for_release(self) -> int:
"""Update CHANGES.md + doc files ready for release"""
self.cleanup_changes_template_for_release()
self.update_version_in_docs()
return 0 # return 0 if no exceptions hit
def update_version_in_docs(self) -> None:
for doc_path in self.version_doc_paths:
LOG.info(f"Updating black version to {self.next_version} in {doc_path}")
with doc_path.open("r") as dfp:
doc_string = dfp.read()
next_version_doc = doc_string.replace(
self.current_version, self.next_version
)
with doc_path.open("w") as dfp:
dfp.write(next_version_doc)
LOG.debug(
f"Finished updating black version to {self.next_version} in {doc_path}"
)
def _handle_debug(debug: bool) -> None:
"""Turn on debugging if asked otherwise INFO default"""
log_level = logging.DEBUG if debug else logging.INFO
logging.basicConfig(
format="[%(asctime)s] %(levelname)s: %(message)s (%(filename)s:%(lineno)d)",
level=log_level,
)
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser()
parser.add_argument(
"-a",
"--add-changes-template",
action="store_true",
help="Add the Unreleased template to CHANGES.md",
)
parser.add_argument(
"-d", "--debug", action="store_true", help="Verbose debug output"
)
args = parser.parse_args()
_handle_debug(args.debug)
return args
def main() -> int:
args = parse_args()
# Need parent.parent cause script is in scripts/ directory
sf = SourceFiles(Path(__file__).parent.parent)
if args.add_changes_template:
return sf.add_template_to_changes()
LOG.info(f"Current version detected to be {sf.current_version}")
LOG.info(f"Next version will be {sf.next_version}")
return sf.update_repo_for_release()
if __name__ == "__main__": # pragma: no cover
sys.exit(main())

69
scripts/release_tests.py Normal file
View File

@ -0,0 +1,69 @@
#!/usr/bin/env python3
import unittest
from pathlib import Path
from shutil import rmtree
from tempfile import TemporaryDirectory
from typing import Any
from unittest.mock import Mock, patch
from release import SourceFiles, tuple_calver # type: ignore
class FakeDateTime:
"""Used to mock the date to test generating next calver function"""
def today(*args: Any, **kwargs: Any) -> "FakeDateTime": # noqa
return FakeDateTime()
# Add leading 0 on purpose to ensure we remove it
def strftime(*args: Any, **kwargs: Any) -> str: # noqa
return "69.01"
class TestRelease(unittest.TestCase):
def setUp(self) -> None:
# We only test on >= 3.12
self.tempdir = TemporaryDirectory(delete=False) # type: ignore
self.tempdir_path = Path(self.tempdir.name)
self.sf = SourceFiles(self.tempdir_path)
def tearDown(self) -> None:
rmtree(self.tempdir.name)
return super().tearDown()
@patch("release.get_git_tags")
def test_get_current_version(self, mocked_git_tags: Mock) -> None:
mocked_git_tags.return_value = ["1.1.0", "69.1.0", "69.1.1", "2.2.0"]
self.assertEqual("69.1.1", self.sf.get_current_version())
@patch("release.get_git_tags")
@patch("release.datetime", FakeDateTime)
def test_get_next_version(self, mocked_git_tags: Mock) -> None:
# test we handle no args
mocked_git_tags.return_value = []
self.assertEqual(
"69.1.0",
self.sf.get_next_version(),
"Unable to get correct next version with no git tags",
)
# test we handle
mocked_git_tags.return_value = ["1.1.0", "69.1.0", "69.1.1", "2.2.0"]
self.assertEqual(
"69.1.2",
self.sf.get_next_version(),
"Unable to get correct version with 2 previous versions released this"
" month",
)
def test_tuple_calver(self) -> None:
first_month_release = tuple_calver("69.1.0")
second_month_release = tuple_calver("69.1.1")
self.assertEqual((69, 1, 0), first_month_release)
self.assertEqual((0, 0, 0), tuple_calver("69.1.1a0")) # Hack for alphas/betas
self.assertTrue(first_month_release < second_month_release)
if __name__ == "__main__":
unittest.main()