PyProject

The Python package build system is split into frontends and backends according to protocols found in several PEPs; chiefly 517 and PEP 660. In modern usage, this system is orchestrated through the pyproject.toml file.

The future of Python builds, distribution, and packaging lies in pyproject.toml.

Contents

PyProject

Legacy

The original packaging system depended on specially-crafted setup.py scripts.

from setuptools import setup
from pathlib import Path

this_directory = Path(__file__).parent
long_description = (this_directory / "README.md").read_text()

setup(
  name='my-project',
  version=1.0.2,
  description='This is the short description',
  long_description=long_description,
  long_description_content_type='text/markdown',
  license='GPL',
  author='John Doe',
  author_email='[email protected]',
  url='example.com/my-project',
  install_requires=[
      'toml>=0.10.2',
  ],
  entry_points={
    'console_scripts': [
      'my-project-cli = my-project:main',
    ],
  },
)

To pull the version from a source control mechanism, snippets like this were used:

import subprocess

ver = subprocess.run(['git', 'describe', '--tags'], stdout=subprocess.PIPE).stdout.decode().strip()

setup(
  ...
  version=ver,
  ...
)

To reduce the size of built packages, source code files to be included were sometimes dynamically selected with snippets like:

from glob import glob

setup(
  ...
  package_data={
    'my-project': [
      'Makefile',
      'README.md',
    ] + [f[5:] for f in glob('static/**', recursive=True)]
  },
  ...
)

PEPs 517, 660, and 518

PEP 518 introduced the pyproject.toml file, intended to declare a package's build backend.

To continue using the legacy build system (i.e., setup.py), the following was sufficient.

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

New packaging tools were encouraged to adopt this file specification and read from separately-named tables. (Note: in the TOML specification, [table-name] declares a table and any lines following it are attributes to that table.)

Setup.cfg

While pyproject.toml was adopted, setuptools did not embrace it. As a half-way step, a declarative configuration language was introduced for the setup.cfg file.

A minimal setup.cfg looked like:

[metadata]
name = my-project
version = 1.0.2
description = This is the short description
long_description = file: path/to/my/long/description
license = GPL
author = John Doe
author_email = [email protected]
url = example.com/my-project

[options]
packages = my-project
python_requires = >= 3.6
install_requires =
    toml >= 0.10.2

[options.entry_points]
console_scripts =
    my-project-cli = my-project:main

[options.package_data]
my-project =
    Makefile
    README.md
    static/*

This enabled the use of minimal shim setup.py files:

import setuptools
if __name__ == "__main__":
    setuptools.setup()

With this pair of files, setup.py install remained a functional build command.

Note that the ability to include globbing patterns under [options.package_data] was added much later. Previously, the recommendation was to not migrate off of setup.py.

PEPs 631 and 621

PEP 631 designed the [project] table for pyproject.toml.

[project]
name = "my-project"
description = "This is the short description"
readme = "path/to/my/long/description"
version = "1.0.2"
authors = [ { name = "John Doe", email = "[email protected]" } ]
urls = { homepage = "example.com/my-project" }
license = { file = "path/to/my/license" }
requires-python = ">=3.6"
dependencies = [
    "toml >= 0.10.1",
]

[project.scripts]
my-project-cli = "my-project:main"

Note the similarities to the setup.cfg syntax. Some notable changes to highlight:

long_description
- a long standing practice was to read a project's README file and provide this as a long_description
- now, just provide a path to the README file
author, author_email, maintainer, maintainer_email
- authors and maintainers have replaced these
- this design greatly simplifies the specification of multiple individuals
license
- while a license may continue to be provided as a string, the syntax has changed (license = { text = "GPL" })
- the current recommendation is to provide a path to the license file
- future PEPs may build on this design
entry_points
- project.scripts exists as a more-or-less perfect mapping
- an analogous [projects.gui-scripts] exists for scripts that should only be called in a graphical setting

This PEP was ultimately superseded by the specification of PEP 621.

Dependencies

The dependencies attribute can be heavily customized. As an example, this was the pyproject.toml of docker-compose at one point:

[project]
dependencies = [
  'cached-property >= 1.2.0, < 2',
  'distro >= 1.5.0, < 2',
  'docker[ssh] >= 4.2.2, < 5',
  'dockerpty >= 0.4.1, < 1',
  'docopt >= 0.6.1, < 1',
  'jsonschema >= 2.5.1, < 4',
  'PyYAML >= 3.10, < 6',
  'python-dotenv >= 0.13.0, < 1',
  'requests >= 2.20.0, < 3',
  'texttable >= 0.9.0, < 2',
  'websocket-client >= 0.32.0, < 1',

  # Conditional
  'backports.shutil_get_terminal_size == 1.0.0; python_version < "3.3"',
  'backports.ssl_match_hostname >= 3.5, < 4; python_version < "3.5"',
  'colorama >= 0.4, < 1; sys_platform == "win32"',
  'enum34 >= 1.0.4, < 2; python_version < "3.4"',
  'ipaddress >= 1.0.16, < 2; python_version < "3.3"',
  'subprocess32 >= 3.5.4, < 4; python_version < "3.2"',
]

[project.optional-dependencies]
socks = [ 'PySocks >= 1.5.6, != 1.5.7, < 2' ]
tests = [
  'ddt >= 1.2.2, < 2',
  'pytest < 6',
  'mock >= 1.0.1, < 4; python_version < "3.4"',
]

Note that the syntax for version specification is inherited from PEP 440.

Package Data

setuptools continues to support limiting package data. This functionality is moved to a separately-named table.

[tool.setuptools.package-data]
my-project = ["Makefile", "README.md", "static/*"]

Note that globbing is now supported. It is necessary to use Unix-style paths for all glob patterns, even when building on Windows.

Setuptools SCM

setuptools_scm is an extension of setuptools for pulling version information dynamically from source control mechanisms.

First, version must be removed from the [package] table and declared as dynamic.

[project]
# version = "1.0.2"
dynamic = ["version"]

A minimal template for pyproject.toml becomes:

[build-system]
requires = ["setuptools>=64", "setuptools-scm>=8"]
build-backend = "setuptools.build_meta"

[tool.setuptools_scm]
# presence enables setuptools-scm

Note that another feature of setuptools_scm is automatic Package Data determination. Built packages only include files tracked by the source control mechanism. To disable this feature, you must disable the Package Data functionality entirely.

[tool.setuptools]
include-package-data = false

CategoryRicottone

Python/PyProject