Refactor docs / Maintenance of docs (#1456)

* Split code style and components documentation

Splits 'the_black_code_style', 'pragmatism', 'blackd',and 'black_primer'
into their own files. The exception being 'the_black_code_style' and
'pragmatism'. They have been merged into one 'the_black_code_style_and_pragmatism'
file.

These changes are being made because the README is becoming very long. And
a README isn't great if it dissuades its reader because of its length.

* Update the doc generation logic and configuration

With the moving of several sections in the README and the renaming of a
few files, 'conf.py' needs to be able to support custom sections.

This commit introduces DocSection which can be used to specify custom
sections of documentation. The information stored in DocSection will be
used by the process_sections function to read, process, and write the section
to CURRENT_DIR.

A large change has been made to the how the docs are prepared to be built.
Instead of just generating the files needed by reading the README, this
has a full chain of operations so custom sections are supported. First,
it reads the README and spits out a list of DocSection objects representing
the sections to be generated by process_sections. This is done since most
of the docs still live in README. Then along with the defined custom_sections
, the process_sections will be begin to process the DocSection objects.
It reads the information it needs to generate the section. Then fetches
the section's contents, calls processors required by the section to process
the section's contents, and finally writes the section to CURRENT_DIR.

This large change is so processing of the documentation can be done just
for the versions hosted on ReadTheDocs.org. An example processor using this
feature is a 'replace_links' processor. It will replace documentation
links that point to the docs hosted on GitHub with links that point to the
version hosted on ReadTheDocs.org. (I won't be coding that ATM)

This also means that files will be overwritten or created once the docs
have been built. It is annoying, since you have to 'git reset --hard'
and 'git clean -f -d' after each build, but there's nothing better. The old
system had the same side effects, so yeah :(

* Update filenames and delete unnecessary files

Update the filenames since 'the_black_code_style' and 'pragmatism' were
merged and 'contributing' was deleted in favor of 'contributing_to_black'.

All symlinks were deleted since their home (_build/generated) is no longer
used.

* Fix broken links and a few redirections

* Merge master into refactor_docs (manually done)

* Add my and most of @hugovk suggestions

Co-Authored-By: Hugo van Kemenade <hugovk@users.noreply.github.com>

* Add logging and improve configurability

Just some cleaning up up of the DocSection dataclass and added logging
support so you know what's going on.

* Rename a section and please the grammar gods of Black

Thanks @hugovk for the suggestion!

* Fix Markdown comments

* Add myself as an author :P

Seems like the right time.

Co-authored-by: Hugo van Kemenade <hugovk@users.noreply.github.com>

docs/authors.md

@@ -1 +0,0 @@
-_build/generated/authors.md

docs/black_primer.md

@@ -0,0 +1,117 @@
+# black-primer
+
+`black-primer` is a tool built for CI (and humans) to have _Black_ `--check` a number of
+(configured in `primer.json`) Git accessible projects in parallel. _(A PR will be
+accepted to add Mercurial support.)_
+
+## Run flow
+
+- Ensure we have a `black` + `git` in PATH
+- Load projects from `primer.json`
+- Run projects in parallel with `--worker` workers (defaults to CPU count / 2)
+  - Checkout projects
+  - Run black and record result
+  - Clean up repository checkout _(can optionally be disabled via `--keep`)_
+- Display results summary to screen
+- Default to cleaning up `--work-dir` (which defaults to tempfile schemantics)
+- Return
+  - 0 for successful run
+  - < 0 for environment / internal error
+  - \> 0 for each project with an error
+
+## Speed up runs 🏎
+
+If you're running locally yourself to test black on lots of code try:
+
+- Using `-k` / `--keep` + `-w` / `--work-dir` so you don't have to re-checkout the repo
+  each run
+
+## CLI arguments
+
+```text
+Usage: black-primer [OPTIONS]
+
+  primer - prime projects for blackening... 🏴
+
+Options:
+  -c, --config PATH      JSON config file path  [default: /Users/cooper/repos/
+                         black/src/black_primer/primer.json]
+
+  --debug                Turn on debug logging  [default: False]
+  -k, --keep             Keep workdir + repos post run  [default: False]
+  -L, --long-checkouts   Pull big projects to test  [default: False]
+  -R, --rebase           Rebase project if already checked out  [default:
+                         False]
+
+  -w, --workdir PATH     Directory path for repo checkouts  [default: /var/fol
+                         ders/tc/hbwxh76j1hn6gqjd2n2sjn4j9k1glp/T/primer.20200
+                         517125229]
+
+  -W, --workers INTEGER  Number of parallel worker coroutines  [default: 2]
+  -h, --help             Show this message and exit.
+```
+
+## primer config file
+
+The config is JSON format. Its main element is the `"projects"` dictionary. Below
+explains each parameter:
+
+```json
+{
+  "projects": {
+    "00_Example": {
+      "cli_arguments": "List of extra CLI arguments to pass Black for this project",
+      "expect_formatting_changes": "Boolean to indicate that the version of Black is expected to cause changes",
+      "git_clone_url": "URL you would pass `git clone` to check out this repo",
+      "long_checkout": "Boolean to have repo skipped by default unless `--long-checkouts` is specified",
+      "py_versions": "List of major Python versions to run this project with - all will do as you'd expect - run on ALL versions"
+    },
+    "aioexabgp": {
+      "cli_arguments": [],
+      "expect_formatting_changes": true,
+      "git_clone_url": "https://github.com/cooperlees/aioexabgp.git",
+      "long_checkout": false,
+      "py_versions": ["all", "3.8"] // "all" ignores all other versions
+    }
+  }
+}
+```
+
+## Example run
+
+```console
+cooper-mbp:black cooper$ ~/venvs/b/bin/black-primer
+[2020-05-17 13:06:40,830] INFO: 4 projects to run Black over (lib.py:270)
+[2020-05-17 13:06:44,215] INFO: Analyzing results (lib.py:285)
+-- primer results 📊 --
+
+3 / 4 succeeded (75.0%) ✅
+1 / 4 FAILED (25.0%) 💩
+ - 0 projects disabled by config
+ - 0 projects skipped due to Python version
+ - 0 skipped due to long checkout
+
+Failed projects:
+
+## flake8-bugbear:
+ - Returned 1
+ - stdout:
+--- tests/b303_b304.py	2020-05-17 20:04:09.991227 +0000
++++ tests/b303_b304.py	2020-05-17 20:06:42.753851 +0000
+@@ -26,11 +26,11 @@
+     maxint = 5  # this is okay
+     # the following shouldn't crash
+     (a, b, c) = list(range(3))
+     # it's different than this
+     a, b, c = list(range(3))
+-    a, b, c, = list(range(3))
++    a, b, c = list(range(3))
+     # and different than this
+     (a, b), c = list(range(3))
+     a, *b, c = [1, 2, 3, 4, 5]
+     b[1:3] = [0, 0]
+
+would reformat tests/b303_b304.py
+Oh no! 💥 💔 💥
+1 file would be reformatted, 22 files would be left unchanged.
+```

docs/blackd.md

@@ -1 +0,0 @@
-_build/generated/blackd.md

docs/blackd.md

@@ -0,0 +1,81 @@
+## blackd
+
+`blackd` is a small HTTP server that exposes _Black_'s functionality over a simple
+protocol. The main benefit of using it is to avoid paying the cost of starting up a new
+_Black_ process every time you want to blacken a file.
+
+### Usage
+
+`blackd` is not packaged alongside _Black_ by default because it has additional
+dependencies. You will need to do `pip install black[d]` to install it.
+
+You can start the server on the default port, binding only to the local interface by
+running `blackd`. You will see a single line mentioning the server's version, and the
+host and port it's listening on. `blackd` will then print an access log similar to most
+web servers on standard output, merged with any exception traces caused by invalid
+formatting requests.
+
+`blackd` provides even less options than _Black_. You can see them by running
+`blackd --help`:
+
+```text
+Usage: blackd [OPTIONS]
+
+Options:
+  --bind-host TEXT                Address to bind the server to.
+  --bind-port INTEGER             Port to listen on
+  --version                       Show the version and exit.
+  -h, --help                      Show this message and exit.
+```
+
+There is no official blackd client tool (yet!). You can test that blackd is working
+using `curl`:
+
+```sh
+blackd --bind-port 9090 &  # or let blackd choose a port
+curl -s -XPOST "localhost:9090" -d "print('valid')"
+```
+
+### Protocol
+
+`blackd` only accepts `POST` requests at the `/` path. The body of the request should
+contain the python source code to be formatted, encoded according to the `charset` field
+in the `Content-Type` request header. If no `charset` is specified, `blackd` assumes
+`UTF-8`.
+
+There are a few HTTP headers that control how the source is formatted. These correspond
+to command line flags for _Black_. There is one exception to this: `X-Protocol-Version`
+which if present, should have the value `1`, otherwise the request is rejected with
+`HTTP 501` (Not Implemented).
+
+The headers controlling how code is formatted are:
+
+- `X-Line-Length`: corresponds to the `--line-length` command line flag.
+- `X-Skip-String-Normalization`: corresponds to the `--skip-string-normalization`
+  command line flag. If present and its value is not the empty string, no string
+  normalization will be performed.
+- `X-Fast-Or-Safe`: if set to `fast`, `blackd` will act as _Black_ does when passed the
+  `--fast` command line flag.
+- `X-Python-Variant`: if set to `pyi`, `blackd` will act as _Black_ does when passed the
+  `--pyi` command line flag. Otherwise, its value must correspond to a Python version or
+  a set of comma-separated Python versions, optionally prefixed with `py`. For example,
+  to request code that is compatible with Python 3.5 and 3.6, set the header to
+  `py3.5,py3.6`.
+- `X-Diff`: corresponds to the `--diff` command line flag. If present, a diff of the
+  formats will be output.
+
+If any of these headers are set to invalid values, `blackd` returns a `HTTP 400` error
+response, mentioning the name of the problematic header in the message body.
+
+Apart from the above, `blackd` can produce the following response codes:
+
+- `HTTP 204`: If the input is already well-formatted. The response body is empty.
+- `HTTP 200`: If formatting was needed on the input. The response body contains the
+  blackened Python code, and the `Content-Type` header is set accordingly.
+- `HTTP 400`: If the input contains a syntax error. Details of the error are returned in
+  the response body.
+- `HTTP 500`: If there was any kind of error while trying to format the input. The
+  response body contains a textual representation of the error.
+
+The response headers include a `X-Black-Version` header containing the version of
+_Black_.

docs/change_log.md

@@ -1 +0,0 @@
-_build/generated/change_log.md

docs/conf.py

@@ -14,28 +14,76 @@
 #
 from pathlib import Path
 import re
-import shutil
 import string
+from typing import Callable, List, Optional, Pattern, Tuple, Set
+from dataclasses import dataclass
+import os
+import logging
 
 from pkg_resources import get_distribution
 from recommonmark.parser import CommonMarkParser
 
+logging.basicConfig(format="%(levelname)s: %(message)s", level=logging.INFO)
 
-CURRENT_DIR = Path(__file__).parent
+LOG = logging.getLogger(__name__)
+
+# Get a relative path so logs printing out SRC isn't too long.
+CURRENT_DIR = Path(__file__).parent.relative_to(os.getcwd())
+README = CURRENT_DIR / ".." / "README.md"
+REFERENCE_DIR = CURRENT_DIR / "reference"
+STATIC_DIR = CURRENT_DIR / "_static"
 
 
-def make_pypi_svg(version):
-    template = CURRENT_DIR / "_static" / "pypi_template.svg"
-    target = CURRENT_DIR / "_static" / "pypi.svg"
+@dataclass
+class SrcRange:
+    """Tracks which part of a file to get a section's content.
+
+    Data:
+        start_line: The line where the section starts (i.e. its sub-header) (inclusive).
+        end_line: The line where the section ends (usually next sub-header) (exclusive).
+    """
+
+    start_line: int
+    end_line: int
+
+
+@dataclass
+class DocSection:
+    """Tracks information about a section of documentation.
+
+    Data:
+        name: The section's name. This will used to detect duplicate sections.
+        src: The filepath to get its contents.
+        processors: The processors to run before writing the section to CURRENT_DIR.
+        out_filename: The filename to use when writing the section to CURRENT_DIR.
+        src_range: The line range of SRC to gets its contents.
+    """
+
+    name: str
+    src: Path
+    src_range: SrcRange = SrcRange(0, 1_000_000)
+    out_filename: str = ""
+    processors: Tuple[Callable, ...] = ()
+
+    def get_out_filename(self) -> str:
+        if not self.out_filename:
+            return self.name + ".md"
+        else:
+            return self.out_filename
+
+
+def make_pypi_svg(version: str) -> None:
+    template: Path = CURRENT_DIR / "_static" / "pypi_template.svg"
+    target: Path = CURRENT_DIR / "_static" / "pypi.svg"
     with open(str(template), "r", encoding="utf8") as f:
-        svg = string.Template(f.read()).substitute(version=version)
+        svg: str = string.Template(f.read()).substitute(version=version)
     with open(str(target), "w", encoding="utf8") as f:
         f.write(svg)
 
 
-def make_filename(line):
-    non_letters = re.compile(r"[^a-z]+")
-    filename = line[3:].rstrip().lower()
+def make_filename(line: str) -> str:
+    non_letters: Pattern = re.compile(r"[^a-z]+")
+    filename: str = line[3:].rstrip().lower()
     filename = non_letters.sub("_", filename)
     if filename.startswith("_"):
         filename = filename[1:]
@@ -44,36 +92,109 @@ def make_filename(line):
     return filename + ".md"
 
 
-def generate_sections_from_readme():
-    target_dir = CURRENT_DIR / "_build" / "generated"
-    readme = CURRENT_DIR / ".." / "README.md"
-    shutil.rmtree(str(target_dir), ignore_errors=True)
-    target_dir.mkdir(parents=True)
+def get_contents(section: DocSection) -> str:
+    """Gets the contents for the DocSection."""
+    contents: List[str] = []
+    src: Path = section.src
+    start_line: int = section.src_range.start_line
+    end_line: int = section.src_range.end_line
+    with open(src, "r", encoding="utf-8") as f:
+        for lineno, line in enumerate(f, start=1):
+            if lineno >= start_line and lineno < end_line:
+                contents.append(line)
+    return "".join(contents)
 
-    output = None
-    target_dir = target_dir.relative_to(CURRENT_DIR)
-    with open(str(readme), "r", encoding="utf8") as f:
-        for line in f:
+
+def get_sections_from_readme() -> List[DocSection]:
+    """Gets the sections from README so they can be processed by process_sections.
+
+    It opens README and goes down line by line looking for sub-header lines which
+    denotes a section. Once it finds a sub-header line, it will create a DocSection
+    object with all of the information currently available. Then on every line, it will
+    track the ending line index of the section. And it repeats this for every sub-header
+    line it finds.
+    """
+    sections: List[DocSection] = []
+    section: Optional[DocSection] = None
+    with open(README, "r", encoding="utf-8") as f:
+        for lineno, line in enumerate(f, start=1):
             if line.startswith("## "):
-                if output is not None:
-                    output.close()
                 filename = make_filename(line)
-                output_path = CURRENT_DIR / filename
-                if output_path.is_symlink() or output_path.is_file():
-                    output_path.unlink()
-                output_path.symlink_to(target_dir / filename)
-                output = open(str(output_path), "w", encoding="utf8")
-                output.write(
-                    "[//]: # (NOTE: THIS FILE IS AUTOGENERATED FROM README.md)\n\n"
+                section_name = filename[:-3]
+                section = DocSection(
+                    name=str(section_name),
+                    src=README,
+                    src_range=SrcRange(lineno, lineno),
+                    out_filename=filename,
+                    processors=(fix_headers,),
                 )
+                sections.append(section)
+            if section is not None:
+                section.src_range.end_line += 1
+    return sections
 
-            if output is None:
-                continue
 
+def fix_headers(contents: str) -> str:
+    """Fixes the headers of sections copied from README.
+
+    Removes one octothorpe (#) from all headers since the contents are no longer nested
+    in a root document (i.e. the README).
+    """
+    lines: List[str] = contents.splitlines()
+    fixed_contents: List[str] = []
+    for line in lines:
         if line.startswith("##"):
             line = line[1:]
+        fixed_contents.append(line + "\n")  # splitlines strips the leading newlines
+    return "".join(fixed_contents)
 
-            output.write(line)
+
+def process_sections(
+    custom_sections: List[DocSection], readme_sections: List[DocSection]
+) -> None:
+    """Reads, processes, and writes sections to CURRENT_DIR.
+
+    For each section, the contents will be fetched, processed by processors
+    required by the section, and written to CURRENT_DIR. If it encounters duplicate
+    sections (i.e. shares the same name attribute), it will skip processing the
+    duplicates.
+
+    It processes custom sections before the README generated sections so sections in the
+    README can be overwritten with custom options.
+    """
+    processed_sections: Set[str] = set()
+    modified_files: Set[Path] = set()
+    sections: List[DocSection] = custom_sections
+    sections.extend(readme_sections)
+    for section in sections:
+        LOG.info(f"Processing '{section.name}' from {section.src}")
+        if section.name in processed_sections:
+            LOG.info(
+                f"Skipping '{section.name}' from '{section.src}' as it is a duplicate"
+            )
+            continue
+
+        target_path: Path = CURRENT_DIR / section.get_out_filename()
+        if target_path in modified_files:
+            LOG.warning(
+                f"{target_path} has been already written to, its contents will be"
+                " OVERWRITTEN and notices will be duplicated"
+            )
+        contents: str = get_contents(section)
+
+        # processors goes here
+        if fix_headers in section.processors:
+            contents = fix_headers(contents)
+
+        with open(target_path, "w", encoding="utf-8") as f:
+            if section.src.suffix == ".md":
+                f.write(
+                    "[//]: # (NOTE: THIS FILE WAS AUTOGENERATED FROM"
+                    f" {section.src})\n\n"
+                )
+            f.write(contents)
+        processed_sections.add(section.name)
+        modified_files.add(target_path)
 
 
 # -- Project information -----------------------------------------------------
@@ -89,8 +210,21 @@ def generate_sections_from_readme():
 version = release
 for sp in "abcfr":
     version = version.split(sp)[0]
+
+custom_sections = [
+    DocSection("the_black_code_style", CURRENT_DIR / "the_black_code_style.md",),
+    DocSection("pragmatism", CURRENT_DIR / "the_black_code_style.md",),
+    DocSection("editor_integration", CURRENT_DIR / "editor_integration.md"),
+    DocSection("blackd", CURRENT_DIR / "blackd.md"),
+    DocSection("black_primer", CURRENT_DIR / "black_primer.md"),
+    DocSection("contributing_to_black", CURRENT_DIR / ".." / "CONTRIBUTING.md"),
+    DocSection("change_log", CURRENT_DIR / ".." / "CHANGES.md"),
+]
+
+
 make_pypi_svg(release)
-generate_sections_from_readme()
+readme_sections = get_sections_from_readme()
+process_sections(custom_sections, readme_sections)
 
 
 # -- General configuration ---------------------------------------------------
@@ -126,6 +260,7 @@ def generate_sections_from_readme():
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path .
+
 exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
 
 # The name of the Pygments (syntax highlighting) style to use.

docs/contributing.md

@@ -1 +0,0 @@
-../CONTRIBUTING.md

docs/contributing_to_black.md

@@ -1 +0,0 @@
-_build/generated/contributing_to_black.md

docs/editor_integration.md

@@ -1 +0,0 @@
-_build/generated/editor_integration.md

docs/editor_integration.md

@@ -0,0 +1,284 @@
+# Editor integration
+
+## Emacs
+
+Options include the following:
+
+- [purcell/reformatter.el](https://github.com/purcell/reformatter.el)
+- [proofit404/blacken](https://github.com/pythonic-emacs/blacken)
+- [Elpy](https://github.com/jorgenschaefer/elpy).
+
+## PyCharm/IntelliJ IDEA
+
+1. Install `black`.
+
+```console
+$ pip install black
+```
+
+2. Locate your `black` installation folder.
+
+On macOS / Linux / BSD:
+
+```console
+$ which black
+/usr/local/bin/black  # possible location
+```
+
+On Windows:
+
+```console
+$ where black
+%LocalAppData%\Programs\Python\Python36-32\Scripts\black.exe  # possible location
+```
+
+Note that if you are using a virtual environment detected by PyCharm, this is an
+unneeded step. In this case the path to `black` is `$PyInterpreterDirectory$/black`.
+
+3. Open External tools in PyCharm/IntelliJ IDEA
+
+On macOS:
+
+`PyCharm -> Preferences -> Tools -> External Tools`
+
+On Windows / Linux / BSD:
+
+`File -> Settings -> Tools -> External Tools`
+
+4. Click the + icon to add a new external tool with the following values:
+
+   - Name: Black
+   - Description: Black is the uncompromising Python code formatter.
+   - Program: <install_location_from_step_2>
+   - Arguments: `"$FilePath$"`
+
+5. Format the currently opened file by selecting `Tools -> External Tools -> black`.
+
+   - Alternatively, you can set a keyboard shortcut by navigating to
+     `Preferences or Settings -> Keymap -> External Tools -> External Tools - Black`.
+
+6. Optionally, run _Black_ on every file save:
+
+   1. Make sure you have the
+      [File Watchers](https://plugins.jetbrains.com/plugin/7177-file-watchers) plugin
+      installed.
+   2. Go to `Preferences or Settings -> Tools -> File Watchers` and click `+` to add a
+      new watcher:
+      - Name: Black
+      - File type: Python
+      - Scope: Project Files
+      - Program: <install_location_from_step_2>
+      - Arguments: `$FilePath$`
+      - Output paths to refresh: `$FilePath$`
+      - Working directory: `$ProjectFileDir$`
+
+   - Uncheck "Auto-save edited files to trigger the watcher" in Advanced Options
+
+## Wing IDE
+
+Wing supports black via the OS Commands tool, as explained in the Wing documentation on
+[pep8 formatting](https://wingware.com/doc/edit/pep8). The detailed procedure is:
+
+1. Install `black`.
+
+```console
+$ pip install black
+```
+
+2. Make sure it runs from the command line, e.g.
+
+```console
+$ black --help
+```
+
+3. In Wing IDE, activate the **OS Commands** panel and define the command **black** to
+   execute black on the currently selected file:
+
+- Use the Tools -> OS Commands menu selection
+- click on **+** in **OS Commands** -> New: Command line..
+  - Title: black
+  - Command Line: black %s
+  - I/O Encoding: Use Default
+  - Key Binding: F1
+  - [x] Raise OS Commands when executed
+  - [x] Auto-save files before execution
+  - [x] Line mode
+
+4. Select a file in the editor and press **F1** , or whatever key binding you selected
+   in step 3, to reformat the file.
+
+## Vim
+
+Commands and shortcuts:
+
+- `:Black` to format the entire file (ranges not supported);
+- `:BlackUpgrade` to upgrade _Black_ inside the virtualenv;
+- `:BlackVersion` to get the current version of _Black_ inside the virtualenv.
+
+Configuration:
+
+- `g:black_fast` (defaults to `0`)
+- `g:black_linelength` (defaults to `88`)
+- `g:black_skip_string_normalization` (defaults to `0`)
+- `g:black_virtualenv` (defaults to `~/.vim/black` or `~/.local/share/nvim/black`)
+
+To install with [vim-plug](https://github.com/junegunn/vim-plug):
+
+```
+Plug 'psf/black', { 'branch': 'stable' }
+```
+
+or with [Vundle](https://github.com/VundleVim/Vundle.vim):
+
+```
+Plugin 'psf/black'
+```
+
+and execute the following in a terminal:
+
+```console
+$ cd ~/.vim/bundle/black
+$ git checkout origin/stable -b stable
+```
+
+or you can copy the plugin from
+[plugin/black.vim](https://github.com/psf/black/blob/stable/plugin/black.vim).
+
+```
+mkdir -p ~/.vim/pack/python/start/black/plugin
+curl https://raw.githubusercontent.com/psf/black/master/plugin/black.vim -o ~/.vim/pack/python/start/black/plugin/black.vim
+```
+
+Let me know if this requires any changes to work with Vim 8's builtin `packadd`, or
+Pathogen, and so on.
+
+This plugin **requires Vim 7.0+ built with Python 3.6+ support**. It needs Python 3.6 to
+be able to run _Black_ inside the Vim process which is much faster than calling an
+external command.
+
+On first run, the plugin creates its own virtualenv using the right Python version and
+automatically installs _Black_. You can upgrade it later by calling `:BlackUpgrade` and
+restarting Vim.
+
+If you need to do anything special to make your virtualenv work and install _Black_ (for
+example you want to run a version from master), create a virtualenv manually and point
+`g:black_virtualenv` to it. The plugin will use it.
+
+To run _Black_ on save, add the following line to `.vimrc` or `init.vim`:
+
+```
+autocmd BufWritePre *.py execute ':Black'
+```
+
+To run _Black_ on a key press (e.g. F9 below), add this:
+
+```
+nnoremap <F9> :Black<CR>
+```
+
+**How to get Vim with Python 3.6?** On Ubuntu 17.10 Vim comes with Python 3.6 by
+default. On macOS with Homebrew run: `brew install vim`. When building Vim from source,
+use: `./configure --enable-python3interp=yes`. There's many guides online how to do
+this.
+
+**I get an import error when using _Black_ from a virtual environment**: If you get an
+error message like this:
+
+```text
+Traceback (most recent call last):
+  File "<string>", line 63, in <module>
+  File "/home/gui/.vim/black/lib/python3.7/site-packages/black.py", line 45, in <module>
+    from typed_ast import ast3, ast27
+  File "/home/gui/.vim/black/lib/python3.7/site-packages/typed_ast/ast3.py", line 40, in <module>
+    from typed_ast import _ast3
+ImportError: /home/gui/.vim/black/lib/python3.7/site-packages/typed_ast/_ast3.cpython-37m-x86_64-linux-gnu.so: undefined symbool: PyExc_KeyboardInterrupt
+```
+
+Then you need to install `typed_ast` and `regex` directly from the source code. The
+error happens because `pip` will download [Python wheels](https://pythonwheels.com/) if
+they are available. Python wheels are a new standard of distributing Python packages and
+packages that have Cython and extensions written in C are already compiled, so the
+installation is much more faster. The problem here is that somehow the Python
+environment inside Vim does not match with those already compiled C extensions and these
+kind of errors are the result. Luckily there is an easy fix: installing the packages
+from the source code.
+
+The two packages that cause the problem are:
+
+- [regex](https://pypi.org/project/regex/)
+- [typed-ast](https://pypi.org/project/typed-ast/)
+
+Now remove those two packages:
+
+```console
+$ pip uninstall regex typed-ast -y
+```
+
+And now you can install them with:
+
+```console
+$ pip install --no-binary :all: regex typed-ast
+```
+
+The C extensions will be compiled and now Vim's Python environment will match. Note that
+you need to have the GCC compiler and the Python development files installed (on
+Ubuntu/Debian do `sudo apt-get install build-essential python3-dev`).
+
+If you later want to update _Black_, you should do it like this:
+
+```console
+$ pip install -U black --no-binary regex,typed-ast
+```
+
+## Visual Studio Code
+
+Use the
+[Python extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python)
+([instructions](https://code.visualstudio.com/docs/python/editing#_formatting)).
+
+## SublimeText 3
+
+Use [sublack plugin](https://github.com/jgirardet/sublack).
+
+## Jupyter Notebook Magic
+
+Use [blackcellmagic](https://github.com/csurfer/blackcellmagic).
+
+## Python Language Server
+
+If your editor supports the [Language Server Protocol](https://langserver.org/) (Atom,
+Sublime Text, Visual Studio Code and many more), you can use the
+[Python Language Server](https://github.com/palantir/python-language-server) with the
+[pyls-black](https://github.com/rupert/pyls-black) plugin.
+
+## Atom/Nuclide
+
+Use [python-black](https://atom.io/packages/python-black).
+
+## Kakoune
+
+Add the following hook to your kakrc, then run _Black_ with `:format`.
+
+```
+hook global WinSetOption filetype=python %{
+    set-option window formatcmd 'black -q  -'
+}
+```
+
+## Thonny
+
+Use [Thonny-black-code-format](https://github.com/Franccisco/thonny-black-code-format).
+
+## Other editors
+
+Other editors will require external contributions.
+
+Patches welcome! ✨ 🍰 ✨
+
+Any tool that can pipe code through _Black_ using its stdio mode (just
+[use `-` as the file name](https://www.tldp.org/LDP/abs/html/special-chars.html#DASHREF2)).
+The formatted code will be returned on stdout (unless `--check` was passed). _Black_
+will still emit messages on stderr but that shouldn't affect your use case.
+
+This can be used for example with PyCharm's or IntelliJ's
+[File Watchers](https://www.jetbrains.com/help/pycharm/file-watchers.html).

docs/ignoring_unmodified_files.md

@@ -1 +0,0 @@
-_build/generated/ignoring_unmodified_files.md

docs/index.rst

@@ -50,14 +50,13 @@ Contents
 
    installation_and_usage
    the_black_code_style
-   pragmatism
    pyproject_toml
    editor_integration
    blackd
-   black-primer
+   black_primer
    version_control_integration
    ignoring_unmodified_files
-   contributing
+   contributing_to_black
    show_your_style
    change_log
    reference/reference_summary

docs/installation_and_usage.md

@@ -1 +0,0 @@
-_build/generated/installation_and_usage.md

docs/license.md

@@ -1 +0,0 @@
-_build/generated/license.md

docs/pyproject_toml.md

@@ -1 +0,0 @@
-_build/generated/pyproject_toml.md

docs/show_your_style.md

@@ -1 +0,0 @@
-_build/generated/show_your_style.md

docs/testimonials.md

@@ -1 +0,0 @@
-_build/generated/testimonials.md

docs/the_black_code_style.md

@@ -1 +0,0 @@
-_build/generated/the_black_code_style.md

docs/the_black_code_style.md

@@ -0,0 +1,457 @@
+# The _Black_ code style
+
+## Code style
+
+_Black_ reformats entire files in place. It is not configurable. It doesn't take
+previous formatting into account. It doesn't reformat blocks that start with
+`# fmt: off` and end with `# fmt: on`. `# fmt: on/off` have to be on the same level of
+indentation. It also recognizes [YAPF](https://github.com/google/yapf)'s block comments
+to the same effect, as a courtesy for straddling code.
+
+### How _Black_ wraps lines
+
+_Black_ ignores previous formatting and applies uniform horizontal and vertical
+whitespace to your code. The rules for horizontal whitespace can be summarized as: do
+whatever makes `pycodestyle` happy. The coding style used by _Black_ can be viewed as a
+strict subset of PEP 8.
+
+As for vertical whitespace, _Black_ tries to render one full expression or simple
+statement per line. If this fits the allotted line length, great.
+
+```py3
+# in:
+
+j = [1,
+     2,
+     3
+]
+
+# out:
+
+j = [1, 2, 3]
+```
+
+If not, _Black_ will look at the contents of the first outer matching brackets and put
+that in a separate indented line.
+
+```py3
+# in:
+
+ImportantClass.important_method(exc, limit, lookup_lines, capture_locals, extra_argument)
+
+# out:
+
+ImportantClass.important_method(
+    exc, limit, lookup_lines, capture_locals, extra_argument
+)
+```
+
+If that still doesn't fit the bill, it will decompose the internal expression further
+using the same rule, indenting matching brackets every time. If the contents of the
+matching brackets pair are comma-separated (like an argument list, or a dict literal,
+and so on) then _Black_ will first try to keep them on the same line with the matching
+brackets. If that doesn't work, it will put all of them in separate lines.
+
+```py3
+# in:
+
+def very_important_function(template: str, *variables, file: os.PathLike, engine: str, header: bool = True, debug: bool = False):
+    """Applies `variables` to the `template` and writes to `file`."""
+    with open(file, 'w') as f:
+        ...
+
+# out:
+
+def very_important_function(
+    template: str,
+    *variables,
+    file: os.PathLike,
+    engine: str,
+    header: bool = True,
+    debug: bool = False,
+):
+    """Applies `variables` to the `template` and writes to `file`."""
+    with open(file, "w") as f:
+        ...
+```
+
+_Black_ prefers parentheses over backslashes, and will remove backslashes if found.
+
+```py3
+# in:
+
+if some_short_rule1 \
+  and some_short_rule2:
+      ...
+
+# out:
+
+if some_short_rule1 and some_short_rule2:
+  ...
+
+
+# in:
+
+if some_long_rule1 \
+  and some_long_rule2:
+    ...
+
+# out:
+
+if (
+    some_long_rule1
+    and some_long_rule2
+):
+    ...
+
+```
+
+Backslashes and multiline strings are one of the two places in the Python grammar that
+break significant indentation. You never need backslashes, they are used to force the
+grammar to accept breaks that would otherwise be parse errors. That makes them confusing
+to look at and brittle to modify. This is why _Black_ always gets rid of them.
+
+If you're reaching for backslashes, that's a clear signal that you can do better if you
+slightly refactor your code. I hope some of the examples above show you that there are
+many ways in which you can do it.
+
+However there is one exception: `with` statements using multiple context managers.
+Python's grammar does not allow organizing parentheses around the series of context
+managers.
+
+We don't want formatting like:
+
+```py3
+with make_context_manager1() as cm1, make_context_manager2() as cm2, make_context_manager3() as cm3, make_context_manager4() as cm4:
+    ...  # nothing to split on - line too long
+```
+
+So _Black_ will now format it like this:
+
+```py3
+with \
+     make_context_manager(1) as cm1, \
+     make_context_manager(2) as cm2, \
+     make_context_manager(3) as cm3, \
+     make_context_manager(4) as cm4 \
+:
+    ...  # backslashes and an ugly stranded colon
+```
+
+You might have noticed that closing brackets are always dedented and that a trailing
+comma is always added. Such formatting produces smaller diffs; when you add or remove an
+element, it's always just one line. Also, having the closing bracket dedented provides a
+clear delimiter between two distinct sections of the code that otherwise share the same
+indentation level (like the arguments list and the docstring in the example above).
+
+If a data structure literal (tuple, list, set, dict) or a line of "from" imports cannot
+fit in the allotted length, it's always split into one element per line. This minimizes
+diffs as well as enables readers of code to find which commit introduced a particular
+entry. This also makes _Black_ compatible with [isort](https://pypi.org/p/isort/) with
+the following configuration.
+
+<details>
+<summary>A compatible `.isort.cfg`</summary>
+
+```
+[settings]
+multi_line_output=3
+include_trailing_comma=True
+force_grid_wrap=0
+use_parentheses=True
+line_length=88
+```
+
+The equivalent command line is:
+
+```
+$ isort --multi-line=3 --trailing-comma --force-grid-wrap=0 --use-parentheses --line-width=88 [ file.py ]
+```
+
+</details>
+
+### Line length
+
+You probably noticed the peculiar default line length. _Black_ defaults to 88 characters
+per line, which happens to be 10% over 80. This number was found to produce
+significantly shorter files than sticking with 80 (the most popular), or even 79 (used
+by the standard library). In general,
+[90-ish seems like the wise choice](https://youtu.be/wf-BqAjZb8M?t=260).
+
+If you're paid by the line of code you write, you can pass `--line-length` with a lower
+number. _Black_ will try to respect that. However, sometimes it won't be able to without
+breaking other rules. In those rare cases, auto-formatted code will exceed your allotted
+limit.
+
+You can also increase it, but remember that people with sight disabilities find it
+harder to work with line lengths exceeding 100 characters. It also adversely affects
+side-by-side diff review on typical screen resolutions. Long lines also make it harder
+to present code neatly in documentation or talk slides.
+
+If you're using Flake8, you can bump `max-line-length` to 88 and forget about it.
+Alternatively, use [Bugbear](https://github.com/PyCQA/flake8-bugbear)'s B950 warning
+instead of E501 and keep the max line length at 80 which you are probably already using.
+You'd do it like this:
+
+```ini
+[flake8]
+max-line-length = 80
+...
+select = C,E,F,W,B,B950
+ignore = E203, E501, W503
+```
+
+You'll find _Black_'s own .flake8 config file is configured like this. Explanation of
+why W503 and E203 are disabled can be found further in this documentation. And if you're
+curious about the reasoning behind B950,
+[Bugbear's documentation](https://github.com/PyCQA/flake8-bugbear#opinionated-warnings)
+explains it. The tl;dr is "it's like highway speed limits, we won't bother you if you
+overdo it by a few km/h".
+
+**If you're looking for a minimal, black-compatible flake8 configuration:**
+
+```ini
+[flake8]
+max-line-length = 88
+extend-ignore = E203
+```
+
+### Empty lines
+
+_Black_ avoids spurious vertical whitespace. This is in the spirit of PEP 8 which says
+that in-function vertical whitespace should only be used sparingly.
+
+_Black_ will allow single empty lines inside functions, and single and double empty
+lines on module level left by the original editors, except when they're within
+parenthesized expressions. Since such expressions are always reformatted to fit minimal
+space, this whitespace is lost.
+
+It will also insert proper spacing before and after function definitions. It's one line
+before and after inner functions and two lines before and after module-level functions
+and classes. _Black_ will not put empty lines between function/class definitions and
+standalone comments that immediately precede the given function/class.
+
+_Black_ will enforce single empty lines between a class-level docstring and the first
+following field or method. This conforms to
+[PEP 257](https://www.python.org/dev/peps/pep-0257/#multi-line-docstrings).
+
+_Black_ won't insert empty lines after function docstrings unless that empty line is
+required due to an inner function starting immediately after.
+
+### Trailing commas
+
+_Black_ will add trailing commas to expressions that are split by comma where each
+element is on its own line. This includes function signatures.
+
+Unnecessary trailing commas are removed if an expression fits in one line. This makes it
+1% more likely that your line won't exceed the allotted line length limit. Moreover, in
+this scenario, if you added another argument to your call, you'd probably fit it in the
+same line anyway. That doesn't make diffs any larger.
+
+One exception to removing trailing commas is tuple expressions with just one element. In
+this case _Black_ won't touch the single trailing comma as this would unexpectedly
+change the underlying data type. Note that this is also the case when commas are used
+while indexing. This is a tuple in disguise: `numpy_array[3, ]`.
+
+One exception to adding trailing commas is function signatures containing `*`, `*args`,
+or `**kwargs`. In this case a trailing comma is only safe to use on Python 3.6. _Black_
+will detect if your file is already 3.6+ only and use trailing commas in this situation.
+If you wonder how it knows, it looks for f-strings and existing use of trailing commas
+in function signatures that have stars in them. In other words, if you'd like a trailing
+comma in this situation and _Black_ didn't recognize it was safe to do so, put it there
+manually and _Black_ will keep it.
+
+### Strings
+
+_Black_ prefers double quotes (`"` and `"""`) over single quotes (`'` and `'''`). It
+will replace the latter with the former as long as it does not result in more backslash
+escapes than before.
+
+_Black_ also standardizes string prefixes, making them always lowercase. On top of that,
+if your code is already Python 3.6+ only or it's using the `unicode_literals` future
+import, _Black_ will remove `u` from the string prefix as it is meaningless in those
+scenarios.
+
+The main reason to standardize on a single form of quotes is aesthetics. Having one kind
+of quotes everywhere reduces reader distraction. It will also enable a future version of
+_Black_ to merge consecutive string literals that ended up on the same line (see
+[#26](https://github.com/psf/black/issues/26) for details).
+
+Why settle on double quotes? They anticipate apostrophes in English text. They match the
+docstring standard described in
+[PEP 257](https://www.python.org/dev/peps/pep-0257/#what-is-a-docstring). An empty
+string in double quotes (`""`) is impossible to confuse with a one double-quote
+regardless of fonts and syntax highlighting used. On top of this, double quotes for
+strings are consistent with C which Python interacts a lot with.
+
+On certain keyboard layouts like US English, typing single quotes is a bit easier than
+double quotes. The latter requires use of the Shift key. My recommendation here is to
+keep using whatever is faster to type and let _Black_ handle the transformation.
+
+If you are adopting _Black_ in a large project with pre-existing string conventions
+(like the popular
+["single quotes for data, double quotes for human-readable strings"](https://stackoverflow.com/a/56190)),
+you can pass `--skip-string-normalization` on the command line. This is meant as an
+adoption helper, avoid using this for new projects.
+
+### Numeric literals
+
+_Black_ standardizes most numeric literals to use lowercase letters for the syntactic
+parts and uppercase letters for the digits themselves: `0xAB` instead of `0XAB` and
+`1e10` instead of `1E10`. Python 2 long literals are styled as `2L` instead of `2l` to
+avoid confusion between `l` and `1`.
+
+### Line breaks & binary operators
+
+_Black_ will break a line before a binary operator when splitting a block of code over
+multiple lines. This is so that _Black_ is compliant with the recent changes in the
+[PEP 8](https://www.python.org/dev/peps/pep-0008/#should-a-line-break-before-or-after-a-binary-operator)
+style guide, which emphasizes that this approach improves readability.
+
+This behaviour may raise `W503 line break before binary operator` warnings in style
+guide enforcement tools like Flake8. Since `W503` is not PEP 8 compliant, you should
+tell Flake8 to ignore these warnings.
+
+### Slices
+
+PEP 8
+[recommends](https://www.python.org/dev/peps/pep-0008/#whitespace-in-expressions-and-statements)
+to treat `:` in slices as a binary operator with the lowest priority, and to leave an
+equal amount of space on either side, except if a parameter is omitted (e.g.
+`ham[1 + 1 :]`). It recommends no spaces around `:` operators for "simple expressions"
+(`ham[lower:upper]`), and extra space for "complex expressions"
+(`ham[lower : upper + offset]`). _Black_ treats anything more than variable names as
+"complex" (`ham[lower : upper + 1]`). It also states that for extended slices, both `:`
+operators have to have the same amount of spacing, except if a parameter is omitted
+(`ham[1 + 1 ::]`). _Black_ enforces these rules consistently.
+
+This behaviour may raise `E203 whitespace before ':'` warnings in style guide
+enforcement tools like Flake8. Since `E203` is not PEP 8 compliant, you should tell
+Flake8 to ignore these warnings.
+
+### Parentheses
+
+Some parentheses are optional in the Python grammar. Any expression can be wrapped in a
+pair of parentheses to form an atom. There are a few interesting cases:
+
+- `if (...):`
+- `while (...):`
+- `for (...) in (...):`
+- `assert (...), (...)`
+- `from X import (...)`
+- assignments like:
+  - `target = (...)`
+  - `target: type = (...)`
+  - `some, *un, packing = (...)`
+  - `augmented += (...)`
+
+In those cases, parentheses are removed when the entire statement fits in one line, or
+if the inner expression doesn't have any delimiters to further split on. If there is
+only a single delimiter and the expression starts or ends with a bracket, the
+parenthesis can also be successfully omitted since the existing bracket pair will
+organize the expression neatly anyway. Otherwise, the parentheses are added.
+
+Please note that _Black_ does not add or remove any additional nested parentheses that
+you might want to have for clarity or further code organization. For example those
+parentheses are not going to be removed:
+
+```py3
+return not (this or that)
+decision = (maybe.this() and values > 0) or (maybe.that() and values < 0)
+```
+
+### Call chains
+
+Some popular APIs, like ORMs, use call chaining. This API style is known as a
+[fluent interface](https://en.wikipedia.org/wiki/Fluent_interface). _Black_ formats
+those by treating dots that follow a call or an indexing operation like a very low
+priority delimiter. It's easier to show the behavior than to explain it. Look at the
+example:
+
+```py3
+def example(session):
+    result = (
+        session.query(models.Customer.id)
+        .filter(
+            models.Customer.account_id == account_id,
+            models.Customer.email == email_address,
+        )
+        .order_by(models.Customer.id.asc())
+        .all()
+    )
+```
+
+### Typing stub files
+
+PEP 484 describes the syntax for type hints in Python. One of the use cases for typing
+is providing type annotations for modules which cannot contain them directly (they might
+be written in C, or they might be third-party, or their implementation may be overly
+dynamic, and so on).
+
+To solve this,
+[stub files with the `.pyi` file extension](https://www.python.org/dev/peps/pep-0484/#stub-files)
+can be used to describe typing information for an external module. Those stub files omit
+the implementation of classes and functions they describe, instead they only contain the
+structure of the file (listing globals, functions, and classes with their members). The
+recommended code style for those files is more terse than PEP 8:
+
+- prefer `...` on the same line as the class/function signature;
+- avoid vertical whitespace between consecutive module-level functions, names, or
+  methods and fields within a single class;
+- use a single blank line between top-level class definitions, or none if the classes
+  are very small.
+
+_Black_ enforces the above rules. There are additional guidelines for formatting `.pyi`
+file that are not enforced yet but might be in a future version of the formatter:
+
+- all function bodies should be empty (contain `...` instead of the body);
+- do not use docstrings;
+- prefer `...` over `pass`;
+- for arguments with a default, use `...` instead of the actual default;
+- avoid using string literals in type annotations, stub files support forward references
+  natively (like Python 3.7 code with `from __future__ import annotations`);
+- use variable annotations instead of type comments, even for stubs that target older
+  versions of Python;
+- for arguments that default to `None`, use `Optional[]` explicitly;
+- use `float` instead of `Union[int, float]`.
+
+## Pragmatism
+
+Early versions of _Black_ used to be absolutist in some respects. They took after its
+initial author. This was fine at the time as it made the implementation simpler and
+there were not many users anyway. Not many edge cases were reported. As a mature tool,
+_Black_ does make some exceptions to rules it otherwise holds. This section documents
+what those exceptions are and why this is the case.
+
+### The magic trailing comma
+
+_Black_ in general does not take existing formatting into account.
+
+However, there are cases where you put a short collection or function call in your code
+but you anticipate it will grow in the future.
+
+For example:
+
+```py3
+TRANSLATIONS = {
+    "en_us": "English (US)",
+    "pl_pl": "polski",
+}
+```
+
+Early versions of _Black_ used to ruthlessly collapse those into one line (it fits!).
+Now, you can communicate that you don't want that by putting a trailing comma in the
+collection yourself. When you do, _Black_ will know to always explode your collection
+into one item per line.
+
+How do you make it stop? Just delete that trailing comma and _Black_ will collapse your
+collection into one line if it fits.
+
+### r"strings" and R"strings"
+
+_Black_ normalizes string quotes as well as string prefixes, making them lowercase. One
+exception to this rule is r-strings. It turns out that the very popular
+[MagicPython](https://github.com/MagicStack/MagicPython/) syntax highlighter, used by
+default by (among others) GitHub and Visual Studio Code, differentiates between
+r-strings and R-strings. The former are syntax highlighted as regular expressions while
+the latter are treated as true raw strings with no special semantics.

docs/version_control_integration.md

@@ -1 +0,0 @@
-_build/generated/version_control_integration.md