PyProject

The Python package build system is split into frontends and backends according to protocols found in several PEPs; chiefly 517 and PEP 660. In modern usage, this system is orchestrated through the pyproject.toml file.

Legacy

The original packaging system depended on specially-crafted setup.py scripts.

from setuptools import setup
from pathlib import Path

this_directory = Path(__file__).parent
long_description = (this_directory / "README.md").read_text()

setup(
  name='my-project',
  version=1.0.2,
  description='This is the short description',
  long_description=long_description,
  long_description_content_type='text/markdown',
  license='GPL',
  author='John Doe',
  author_email='[email protected]',
  url='example.com/my-project',
  install_requires=[
      'toml>=0.10.2',
  ],
  entry_points={
    'console_scripts': [
      'my-project-cli = my-project:main',
    ],
  },
)

To pull the version from a source control mechanism, snippets like this were used:

import subprocess

ver = subprocess.run(['git', 'describe', '--tags'], stdout=subprocess.PIPE).stdout.decode().strip()

setup(
  ...
  version=ver,
  ...
)

To reduce the size of built packages, source code files to be included were sometimes dynamically selected with snippets like:

from glob import glob

setup(
  ...
  package_data={
    'my-project': [
      'Makefile',
      'README.md',
    ] + [f[5:] for f in glob('static/**', recursive=True)]
  },
  ...
)

PEPs 517, 660, and 518

PEP 518 introduced the pyproject.toml file, intended to declare a package's build backend.

To continue using the legacy build system (i.e., setup.py), the following was sufficient.

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

New packaging tools were encouraged to adopt this file specification and read from separately-named tables. (Note: in the TOML specification, [table-name] declares a table and any lines following it are attributes to that table.)

Setup.cfg

While pyproject.toml was adopted, setuptools did not embrace it. As a half-way step, a declarative configuration language was introduced for the setup.cfg file.

A minimal setup.cfg looked like:

[metadata]
name = my-project
version = 1.0.2
description = This is the short description
long_description = file: path/to/my/long/description
license = GPL
author = John Doe
author_email = [email protected]
url = example.com/my-project

[options]
packages = my-project
python_requires = >= 3.6
install_requires =
    toml >= 0.10.2

[options.entry_points]
console_scripts =
    my-project-cli = my-project:main

[options.package_data]
my-project =
    Makefile
    README.md
    static/*

This enabled the use of minimal shim setup.py files:

import setuptools
if __name__ == "__main__":
    setuptools.setup()

With this pair of files, setup.py install remained a functional build command.

Note that the ability to include globbing patterns under [options.package_data] was added much later. Previously, the recommendation was to not migrate off of setup.py.

PEPs 631 and 621

PEP 631 designed the [project] table for pyproject.toml.

[project]
name = "my-project"
description = "This is the short description"
readme = "path/to/my/long/description"
version = "1.0.2"
authors = [ { name = "John Doe", email = "[email protected]" } ]
urls = { homepage = "example.com/my-project" }
license = { file = "path/to/my/license" }
requires-python = ">=3.6"
dependencies = [
    "toml >= 0.10.1",
]

[project.scripts]
my-project-cli = "my-project:main"

Note the similarities to the setup.cfg syntax. Some notable changes to highlight:

• long_description

  • a long standing practice was to read a project's README file and provide this as a long_description

  • now, just provide a path to the README file

• author, author_email, maintainer, maintainer_email

  • authors and maintainers have replaced these

  • this design greatly simplifies the specification of multiple individuals

• license

  • while a license may continue to be provided as a string, the syntax has changed (license = { text = "GPL" })

  • the current recommendation is to provide a path to the license file
  • future PEPs may build on this design

• entry_points

  • project.scripts exists as a more-or-less perfect mapping

  • an analogous [projects.gui-scripts] exists for scripts that should only be called in a graphical setting

This PEP was ultimately superseded by the specification of PEP 621.

Dependencies

The dependencies attribute can be heavily customized. As an example, this was the pyproject.toml of docker-compose at one point:

[project]
dependencies = [
  'cached-property >= 1.2.0, < 2',
  'distro >= 1.5.0, < 2',
  'docker[ssh] >= 4.2.2, < 5',
  'dockerpty >= 0.4.1, < 1',
  'docopt >= 0.6.1, < 1',
  'jsonschema >= 2.5.1, < 4',
  'PyYAML >= 3.10, < 6',
  'python-dotenv >= 0.13.0, < 1',
  'requests >= 2.20.0, < 3',
  'texttable >= 0.9.0, < 2',
  'websocket-client >= 0.32.0, < 1',

  # Conditional
  'backports.shutil_get_terminal_size == 1.0.0; python_version < "3.3"',
  'backports.ssl_match_hostname >= 3.5, < 4; python_version < "3.5"',
  'colorama >= 0.4, < 1; sys_platform == "win32"',
  'enum34 >= 1.0.4, < 2; python_version < "3.4"',
  'ipaddress >= 1.0.16, < 2; python_version < "3.3"',
  'subprocess32 >= 3.5.4, < 4; python_version < "3.2"',
]

[project.optional-dependencies]
socks = [ 'PySocks >= 1.5.6, != 1.5.7, < 2' ]
tests = [
  'ddt >= 1.2.2, < 2',
  'pytest < 6',
  'mock >= 1.0.1, < 4; python_version < "3.4"',
]

Note that the syntax for version specification is inherited from PEP 440.

Package Data

setuptools continues to support limiting package data. This functionality is moved to a separately-named table.

[tool.setuptools.package-data]
my-project = ["Makefile", "README.md", "static/*"]

Note that globbing is now supported. It is necessary to use Unix-style paths for all glob patterns, even when building on Windows.

Setuptools SCM

setuptools_scm is an extension of setuptools for pulling version information dynamically from source control mechanisms.

First, version must be removed from the [package] table and declared as dynamic.

[project]
# version = "1.0.2"
dynamic = ["version"]

A minimal template for pyproject.toml becomes:

[build-system]
requires = ["setuptools>=64", "setuptools-scm>=8"]
build-backend = "setuptools.build_meta"

[tool.setuptools_scm]
# presence enables setuptools-scm

Note that another feature of setuptools_scm is automatic Package Data determination. Built packages only include files tracked by the source control mechanism. To disable this feature, you must disable the Package Data functionality entirely.

[tool.setuptools]
include-package-data = false

CategoryRicottone