|
Size: 3441
Comment:
|
Size: 9290
Comment: Dists
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 3: | Line 3: |
| The future of Python builds, distribution, and packaging lies in '''pyproject.toml'''. The protocol has been incrementally built through PEPs [[https://www.python.org/dev/peps/pep-0517/|517]], [[https://www.python.org/dev/peps/pep-0518/|518]], [[https://www.python.org/dev/peps/pep-0621/|PEP 621]], [[https://www.python.org/dev/peps/pep-0631/|PEP 631]], and [[https://www.python.org/dev/peps/pep-0660/|PEP 660]]. | The Python package build system is split into frontends and backends according to protocols found in several PEPs; chiefly [[https://www.python.org/dev/peps/pep-0517/|517]] and [[https://www.python.org/dev/peps/pep-0660/|PEP 660]]. In modern usage, this system is orchestrated through the '''`pyproject.toml`''' file. |
| Line 11: | Line 11: |
| == Build System == This is the focus of PEP 517. A minimal example is: |
== Legacy == The original packaging system depended on specially-crafted `setup.py` scripts. {{{ #!/usr/bin/env python3 from setuptools import setup from pathlib import Path this_directory = Path(__file__).parent long_description = (this_directory / "README.md").read_text() setup( name='my-project', version=1.0.2, description='This is the short description', long_description=long_description, long_description_content_type='text/markdown', license='GPL', author='John Doe', author_email='[email protected]', url='example.com/my-project', install_requires=[ 'toml>=0.10.2', ], entry_points={ 'console_scripts': [ 'my-project-cli = my-project:main', ], }, ) }}} To pull the version from a source control mechanism, snippets like this were used: {{{ import subprocess ver = subprocess.run(['git', 'describe', '--tags'], stdout=subprocess.PIPE).stdout.decode().strip() setup( ... version=ver, ... ) }}} Such a package is installed by running `python setup.py install`. To reduce the size of built packages, source code files to be included were sometimes dynamically selected with snippets like: {{{ from glob import glob setup( ... package_data={ 'my-project': [ 'Makefile', 'README.md', ] + [f[5:] for f in glob('static/**', recursive=True)] }, ... ) }}} === Distributions === There are fundamentally two types of package distributions: source distributions ('''sdist''') and binary distributions ('''bdist'''). The command for creating an sdist is `python setup.py sdist`. The distribution would be placed in a `dist/` subfolder. There have been several different bdist formats over time. One of the earliest was '''`egg`''' files, which was favored by the `easy_install` package installer. Eggs files were named like `my-package-1.0.2-py3.6.egg` and were specially-crafted ZIP archives. The command for creating an wheel was `python setup.py bdist_egg`. The egg would be placed in a `dist/` subfolder, build artifacts would be placed in a `build/` subfolder, and parsed information was collected in a subfolder named like `my-package.egg-info/`. The modern standard bdist format is '''`wheel`''' files, which is favored by [[Python/Pip|pip]]. Wheel files are named like `my-project-1.0.2-py3-none-any.whl` and are also specially-crafted ZIP archives. The command for creating a wheel is `python setup.py bdist_wheel`, which depended on the `wheel` package being installed. Otherwise the same subfolders were used, including `my-package.egg-info/`. ---- == PEPs 517, 660, and 518 == [[https://peps.python.org/pep-0518/|PEP 518]] introduced the '''`pyproject.toml`''' file, intended to declare a package's build backend. To continue using the legacy build system (i.e., `setup.py`), the following was sufficient. |
| Line 21: | Line 105: |
| ---- == Project == This is the focus of PEP 621. It is more-or-less a direct mapping of a boilerplate `setup.py` file into the TOML format. {{{ [project] name = "my-project" |
New packaging tools were encouraged to adopt this file specification and read from separately-named tables. (Note: in the TOML specification, `[table-name]` declares a table and any lines following it are attributes to that table.) === Setup.cfg === While `pyproject.toml` was adopted, `setuptools` did not embrace it. As a half-way step, a declarative configuration language was introduced for the '''`setup.cfg`''' file. A minimal `setup.cfg` looked like: {{{ [metadata] name = my-project version = 1.0.2 description = This is the short description long_description = file: path/to/my/long/description license = GPL author = John Doe author_email = [email protected] url = example.com/my-project [options] packages = my-project python_requires = >= 3.6 install_requires = toml >= 0.10.2 [options.entry_points] console_scripts = my-project-cli = my-project:main [options.package_data] my-project = Makefile README.md static/* }}} This enabled the use of minimal shim `setup.py` files: {{{ #!/usr/bin/env python import setuptools if __name__ == "__main__": setuptools.setup() }}} With this pair of files, `setup.py install` remained a functional build command. Note that the ability to include globbing patterns under `[options.package_data]` was added much later. Previously, the recommendation was to ''not'' migrate off of `setup.py`. ---- == PEPs 631 and 621 == [[https://peps.python.org/pep-0631/|PEP 631]] designed the '''`[project]`''' table for `pyproject.toml`. {{{ [project] name = "my-project" |
| Line 33: | Line 168: |
| readme = "path/to/my/long/description" version = "1.0.2" authors = [ { name = "John Doe", email = "[email protected]" } ] urls = { homepage = "example.com/my-project" } license = { file = "path/to/my/license" } requires-python = ">=3.6" dependencies = [ "toml >= 0.10.1", |
readme = "path/to/my/long/description" version = "1.0.2" authors = [ { name = "John Doe", email = "[email protected]" } ] urls = { homepage = "example.com/my-project" } license = { file = "path/to/my/license" } requires-python = ">=3.6" dependencies = [ "toml >= 0.10.1", |
| Line 47: | Line 182: |
| Some notable changes from `setup.py`: |
Note the similarities to the `setup.cfg` syntax. Some notable changes to highlight: |
| Line 63: | Line 197: |
| This PEP was ultimately superseded by the specification of [[https://peps.python.org/pep-0621/|PEP 621]]. |
|
| Line 67: | Line 203: |
| PEP 631 furthered the design of `dependencies` and introduced `optional-dependencies`. This is the `pyproject.toml` of `docker-compose`: | The dependencies attribute can be heavily customized. As an example, this was the `pyproject.toml` of `docker-compose` at one point: |
| Line 102: | Line 238: |
| Note that the syntax for version specification is inherited from [[https://peps.python.org/pep-0440/#version-specifiers|PEP 440]]. === Package Data === `setuptools` continues to support limiting package data. This functionality is moved to a separately-named table. {{{ [tool.setuptools.package-data] my-project = ["Makefile", "README.md", "static/*"] }}} Note that globbing is now supported. It is necessary to use Unix-style paths for all glob patterns, even when building on Windows. === Setuptools SCM === `setuptools_scm` is an extension of `setuptools` for pulling version information dynamically from source control mechanisms. First, `version` '''must''' be removed from the `[package]` table and declared as dynamic. {{{ [project] # version = "1.0.2" dynamic = ["version"] }}} A minimal template for `pyproject.toml` becomes: {{{ [build-system] requires = ["setuptools>=64", "setuptools-scm>=8"] build-backend = "setuptools.build_meta" [tool.setuptools_scm] # presence enables setuptools-scm }}} Note that another feature of `setuptools_scm` is automatic Package Data determination. Built packages only include files tracked by the source control mechanism. To disable this feature, you must disable the Package Data functionality entirely. {{{ [tool.setuptools] include-package-data = false }}} === Distributions === In the modern build system, distributions have been pared down to sdist and wheel. With the gradual deprecation of `setup.py`, the new recommended method for building package distributions is installing the `build` package and running `python -m build`. This creates both distributions, and specifically builds the wheel from the sdist. |
PyProject
The Python package build system is split into frontends and backends according to protocols found in several PEPs; chiefly 517 and PEP 660. In modern usage, this system is orchestrated through the pyproject.toml file.
Contents
Legacy
The original packaging system depended on specially-crafted setup.py scripts.
from setuptools import setup from pathlib import Path this_directory = Path(__file__).parent long_description = (this_directory / "README.md").read_text() setup( name='my-project', version=1.0.2, description='This is the short description', long_description=long_description, long_description_content_type='text/markdown', license='GPL', author='John Doe', author_email='[email protected]', url='example.com/my-project', install_requires=[ 'toml>=0.10.2', ], entry_points={ 'console_scripts': [ 'my-project-cli = my-project:main', ], }, )
To pull the version from a source control mechanism, snippets like this were used:
import subprocess ver = subprocess.run(['git', 'describe', '--tags'], stdout=subprocess.PIPE).stdout.decode().strip() setup( ... version=ver, ... )
Such a package is installed by running python setup.py install.
To reduce the size of built packages, source code files to be included were sometimes dynamically selected with snippets like:
from glob import glob
setup(
...
package_data={
'my-project': [
'Makefile',
'README.md',
] + [f[5:] for f in glob('static/**', recursive=True)]
},
...
)
Distributions
There are fundamentally two types of package distributions: source distributions (sdist) and binary distributions (bdist).
The command for creating an sdist is python setup.py sdist. The distribution would be placed in a dist/ subfolder.
There have been several different bdist formats over time. One of the earliest was egg files, which was favored by the easy_install package installer. Eggs files were named like my-package-1.0.2-py3.6.egg and were specially-crafted ZIP archives. The command for creating an wheel was python setup.py bdist_egg. The egg would be placed in a dist/ subfolder, build artifacts would be placed in a build/ subfolder, and parsed information was collected in a subfolder named like my-package.egg-info/.
The modern standard bdist format is wheel files, which is favored by pip. Wheel files are named like my-project-1.0.2-py3-none-any.whl and are also specially-crafted ZIP archives. The command for creating a wheel is python setup.py bdist_wheel, which depended on the wheel package being installed. Otherwise the same subfolders were used, including my-package.egg-info/.
PEPs 517, 660, and 518
PEP 518 introduced the pyproject.toml file, intended to declare a package's build backend.
To continue using the legacy build system (i.e., setup.py), the following was sufficient.
[build-system] requires = ["setuptools"] build-backend = "setuptools.build_meta"
New packaging tools were encouraged to adopt this file specification and read from separately-named tables. (Note: in the TOML specification, [table-name] declares a table and any lines following it are attributes to that table.)
Setup.cfg
While pyproject.toml was adopted, setuptools did not embrace it. As a half-way step, a declarative configuration language was introduced for the setup.cfg file.
A minimal setup.cfg looked like:
[metadata] name = my-project version = 1.0.2 description = This is the short description long_description = file: path/to/my/long/description license = GPL author = John Doe author_email = [email protected] url = example.com/my-project [options] packages = my-project python_requires = >= 3.6 install_requires = toml >= 0.10.2 [options.entry_points] console_scripts = my-project-cli = my-project:main [options.package_data] my-project = Makefile README.md static/*
This enabled the use of minimal shim setup.py files:
import setuptools
if __name__ == "__main__":
setuptools.setup()With this pair of files, setup.py install remained a functional build command.
Note that the ability to include globbing patterns under [options.package_data] was added much later. Previously, the recommendation was to not migrate off of setup.py.
PEPs 631 and 621
PEP 631 designed the [project] table for pyproject.toml.
[project]
name = "my-project"
description = "This is the short description"
readme = "path/to/my/long/description"
version = "1.0.2"
authors = [ { name = "John Doe", email = "[email protected]" } ]
urls = { homepage = "example.com/my-project" }
license = { file = "path/to/my/license" }
requires-python = ">=3.6"
dependencies = [
"toml >= 0.10.1",
]
[project.scripts]
my-project-cli = "my-project:main"Note the similarities to the setup.cfg syntax. Some notable changes to highlight:
long_description
a long standing practice was to read a project's README file and provide this as a long_description
now, just provide a path to the README file
author, author_email, maintainer, maintainer_email
authors and maintainers have replaced these
- this design greatly simplifies the specification of multiple individuals
license
while a license may continue to be provided as a string, the syntax has changed (license = { text = "GPL" })
- the current recommendation is to provide a path to the license file
- future PEPs may build on this design
entry_points
project.scripts exists as a more-or-less perfect mapping
an analogous [projects.gui-scripts] exists for scripts that should only be called in a graphical setting
This PEP was ultimately superseded by the specification of PEP 621.
Dependencies
The dependencies attribute can be heavily customized. As an example, this was the pyproject.toml of docker-compose at one point:
[project] dependencies = [ 'cached-property >= 1.2.0, < 2', 'distro >= 1.5.0, < 2', 'docker[ssh] >= 4.2.2, < 5', 'dockerpty >= 0.4.1, < 1', 'docopt >= 0.6.1, < 1', 'jsonschema >= 2.5.1, < 4', 'PyYAML >= 3.10, < 6', 'python-dotenv >= 0.13.0, < 1', 'requests >= 2.20.0, < 3', 'texttable >= 0.9.0, < 2', 'websocket-client >= 0.32.0, < 1', # Conditional 'backports.shutil_get_terminal_size == 1.0.0; python_version < "3.3"', 'backports.ssl_match_hostname >= 3.5, < 4; python_version < "3.5"', 'colorama >= 0.4, < 1; sys_platform == "win32"', 'enum34 >= 1.0.4, < 2; python_version < "3.4"', 'ipaddress >= 1.0.16, < 2; python_version < "3.3"', 'subprocess32 >= 3.5.4, < 4; python_version < "3.2"', ] [project.optional-dependencies] socks = [ 'PySocks >= 1.5.6, != 1.5.7, < 2' ] tests = [ 'ddt >= 1.2.2, < 2', 'pytest < 6', 'mock >= 1.0.1, < 4; python_version < "3.4"', ]
Note that the syntax for version specification is inherited from PEP 440.
Package Data
setuptools continues to support limiting package data. This functionality is moved to a separately-named table.
[tool.setuptools.package-data] my-project = ["Makefile", "README.md", "static/*"]
Note that globbing is now supported. It is necessary to use Unix-style paths for all glob patterns, even when building on Windows.
Setuptools SCM
setuptools_scm is an extension of setuptools for pulling version information dynamically from source control mechanisms.
First, version must be removed from the [package] table and declared as dynamic.
[project] # version = "1.0.2" dynamic = ["version"]
A minimal template for pyproject.toml becomes:
[build-system] requires = ["setuptools>=64", "setuptools-scm>=8"] build-backend = "setuptools.build_meta" [tool.setuptools_scm] # presence enables setuptools-scm
Note that another feature of setuptools_scm is automatic Package Data determination. Built packages only include files tracked by the source control mechanism. To disable this feature, you must disable the Package Data functionality entirely.
[tool.setuptools] include-package-data = false
Distributions
In the modern build system, distributions have been pared down to sdist and wheel.
With the gradual deprecation of setup.py, the new recommended method for building package distributions is installing the build package and running python -m build. This creates both distributions, and specifically builds the wheel from the sdist.