PyProject
The Python package build system is split into frontends and backends according to protocols found in several PEPs; chiefly 517 and PEP 660. In modern usage, this system is orchestrated through the pyproject.toml file.
Contents
Legacy
The original packaging system depended on specially-crafted setup.py scripts.
Almost universally, these required that the third-party setuptools package be installed.
from setuptools import setup from pathlib import Path this_directory = Path(__file__).parent long_description = (this_directory / "README.md").read_text() setup( name='my-project', version=1.0.2, description='This is the short description', long_description=long_description, long_description_content_type='text/markdown', license='GPL', author='John Doe', author_email='[email protected]', url='example.com/my-project', install_requires=[ 'toml>=0.10.2', ], entry_points={ 'console_scripts': [ 'my-project-cli = my-project:main', ], }, )
To pull the version from a source control mechanism, snippets like this were used:
import subprocess ver = subprocess.run(['git', 'describe', '--tags'], stdout=subprocess.PIPE).stdout.decode().strip() setup( ... version=ver, ... )
Such a package is installed by running python setup.py install.
To reduce the size of built packages, source code files to be included were sometimes dynamically selected with snippets like:
from glob import glob
setup(
...
package_data={
'my-project': [
'Makefile',
'README.md',
] + [f[5:] for f in glob('static/**', recursive=True)]
},
...
)
Distributions
There are fundamentally two types of package distributions: source distributions (sdist) and binary distributions (bdist).
The command for creating an sdist is python setup.py sdist. The distribution would be placed in a dist/ subfolder.
There have been several different bdist formats over time. One of the earliest was egg files, which was favored by the easy_install package installer. Eggs files were named like my-package-1.0.2-py3.6.egg and were specially-crafted ZIP archives. The command for creating an wheel was python setup.py bdist_egg. The egg would be placed in a dist/ subfolder, build artifacts would be placed in a build/ subfolder, and parsed information was collected in a subfolder named like my-package.egg-info/.
The modern standard bdist format is wheel files, which is favored by pip. Wheel files are named like my-project-1.0.2-py3-none-any.whl and are also specially-crafted ZIP archives. The command for creating a wheel is python setup.py bdist_wheel, which depended on the wheel package being installed. Otherwise the same subfolders were used, including my-package.egg-info/.
Note that the supported types of distributions came from the standard library's distutils module. This was obscured over time by the popularity of setuptools.
PEPs 517, 660, and 518
PEP 518 introduced the pyproject.toml file, intended to declare a package's build backend.
To continue using the legacy build system (i.e., setup.py), the following was sufficient.
[build-system] requires = ["setuptools"] build-backend = "setuptools.build_meta"
New packaging tools were encouraged to adopt this file specification and read from separately-named tables. (Note: in the TOML specification, [table-name] declares a table and any lines following it are attributes to that table.)
Setup.cfg
While pyproject.toml was adopted, setuptools did not embrace it. As a half-way step, a declarative configuration language was introduced for the setup.cfg file.
A minimal setup.cfg looked like:
[metadata] name = my-project version = 1.0.2 description = This is the short description long_description = file: path/to/my/long/description license = GPL author = John Doe author_email = [email protected] url = example.com/my-project [options] packages = my-project python_requires = >= 3.6 install_requires = toml >= 0.10.2 [options.entry_points] console_scripts = my-project-cli = my-project:main [options.package_data] my-project = Makefile README.md static/*
This enabled the use of minimal shim setup.py files:
import setuptools
if __name__ == "__main__":
setuptools.setup()With this pair of files, setup.py install remained a functional build command.
Note that the ability to include globbing patterns under [options.package_data] was added much later. Previously, the recommendation was to not migrate off of setup.py.
PEPs 631 and 621
PEP 631 designed the [project] table for pyproject.toml.
[project]
name = "my-project"
description = "This is the short description"
readme = "path/to/my/long/description"
version = "1.0.2"
authors = [ { name = "John Doe", email = "[email protected]" } ]
urls = { homepage = "example.com/my-project" }
license = { file = "path/to/my/license" }
requires-python = ">=3.6"
dependencies = [
"toml >= 0.10.1",
]
[project.scripts]
my-project-cli = "my-project:main"Note the similarities to the setup.cfg syntax. Some notable changes to highlight:
long_description
a long standing practice was to read a project's README file and provide this as a long_description
now, just provide a path to the README file
author, author_email, maintainer, maintainer_email
authors and maintainers have replaced these
- this design greatly simplifies the specification of multiple individuals
license
while a license may continue to be provided as a string, the syntax has changed (license = { text = "GPL" })
- the current recommendation is to provide a path to the license file
- future PEPs may build on this design
entry_points
project.scripts exists as a more-or-less perfect mapping
an analogous [projects.gui-scripts] exists for scripts that should only be called in a graphical setting
This PEP was ultimately superseded by the specification of PEP 621.
Dependencies
The dependencies attribute can be heavily customized. As an example, this was the pyproject.toml of docker-compose at one point:
[project] dependencies = [ 'cached-property >= 1.2.0, < 2', 'distro >= 1.5.0, < 2', 'docker[ssh] >= 4.2.2, < 5', 'dockerpty >= 0.4.1, < 1', 'docopt >= 0.6.1, < 1', 'jsonschema >= 2.5.1, < 4', 'PyYAML >= 3.10, < 6', 'python-dotenv >= 0.13.0, < 1', 'requests >= 2.20.0, < 3', 'texttable >= 0.9.0, < 2', 'websocket-client >= 0.32.0, < 1', # Conditional 'backports.shutil_get_terminal_size == 1.0.0; python_version < "3.3"', 'backports.ssl_match_hostname >= 3.5, < 4; python_version < "3.5"', 'colorama >= 0.4, < 1; sys_platform == "win32"', 'enum34 >= 1.0.4, < 2; python_version < "3.4"', 'ipaddress >= 1.0.16, < 2; python_version < "3.3"', 'subprocess32 >= 3.5.4, < 4; python_version < "3.2"', ] [project.optional-dependencies] socks = [ 'PySocks >= 1.5.6, != 1.5.7, < 2' ] tests = [ 'ddt >= 1.2.2, < 2', 'pytest < 6', 'mock >= 1.0.1, < 4; python_version < "3.4"', ]
Note that the syntax for version specification is inherited from PEP 440.
Package Data
setuptools continues to support limiting package data. This functionality is moved to a separately-named table.
[tool.setuptools.package-data] my-project = ["Makefile", "README.md", "static/*"]
Note that globbing is now supported. It is necessary to use Unix-style paths for all glob patterns, even when building on Windows.
Setuptools SCM
setuptools_scm is an extension of setuptools for pulling version information dynamically from source control mechanisms.
First, version must be removed from the [package] table and declared as dynamic.
[project] # version = "1.0.2" dynamic = ["version"]
A minimal template for pyproject.toml becomes:
[build-system] requires = ["setuptools>=64", "setuptools-scm>=8"] build-backend = "setuptools.build_meta" [tool.setuptools_scm] # presence enables setuptools-scm
Note that another feature of setuptools_scm is automatic Package Data determination. Built packages only include files tracked by the source control mechanism. To disable this feature, you must disable the Package Data functionality entirely.
[tool.setuptools] include-package-data = false
Distributions
In the modern build system, distributions have been pared down to sdist and wheel.
Furthermore, distutils is deprecated and will be removed from the standard library. The setuptools project has adopted a large subset of its API as a compatibility module. In other words, building a package distribution will soon require external dependencies.
The new recommended method for building package distributions is installing the build package and running python -m build. This creates both an sdist and a wheel, and specifically builds the wheel from the sdist. (Note that setuptools and wheel are dependencies of build.)