Python/Pybuild - Debian Wiki

Translation(s): English

Contents

Links
Introduction
debian/control
debian/rules
1. CUSTOMIZATION
Resources

Links

Main Python page

Introduction

Pybuild is a newer (as in 2013) and more modular build system for Python packages.

Features:

Build Python3, ?PyPy with a one-liner dh command
Support pytest and tox testing

Note: Python2 packages should not be uploaded to unstable.

debian/control

dh-sequence-python3 or dh-python is a mandatory build dependency. dh-sequence-python3 is better, since it saves you having to also write --with python3 in debian/rules.

The pybuild dh integration script (not pybuild itself) uses the Build-Depends to figure out which interpreters it should build for. If the package provides Python 2.X modules it should build-depend on python-all (or python-all-dev in case of Python extensions), if it builds extensions with debug symbols: python-all-dbg; if it provides Python 3 script - python3 needs to be added to Build-Depends, etc. If you want to provide only python3 packages you can build-depend on python3-all (python3-all-dev, python3-all-dbg).

One very important thing to get right is the Build-Depends line in the source package stanza. setuptools/distribute-based packages have the nasty habit of downloading dependencies from PyPI if they are needed at python setup.py build time. If the package is available from the system (as would be the case when Build-Depends is up-to-date), then distribute will not try to download the package, otherwise it will try to download it. This is a huge no-no, and pybuild internally sets the http_proxy and https_proxy environment variables (to http[s]://127.0.0.1:9/) to prevent this from happening.

dh_python3 will correctly fill in the installation dependencies (via ${python3:Depends), but it cannot fill in the build dependencies. Take extra care in getting this right, and double check your build logs for illegal access to pypi.python.org.

debian/rules

Here is a debian/rules file for you to start with. First, I'll show you the whole thing, then I'll explain it line by line.

#export DH_VERBOSE=1
export PYBUILD_NAME=foo

%:
        dh $@ --buildsystem=pybuild

The file starts with the standard #! line. I like adding the (commented out when uploading) DH_VERBOSE line because it can make build problems easier to debug.

The next line is:

export PYBUILD_NAME=foo

pybuild supports a number of variables which can be used to control such things as the destination directory for build artifacts for both the Python 2 and Python 3 builds. The defaults generally do the right thing, but you do need to at least tell pybuild the name of your package. Here, we're telling it that the name is foo. This should match the module name, so for example, in enum34, you'd see:

export PYBUILD_NAME=enum34

even though the binary packages that get produced are python-enum34 and python3-enum34.

The next line is the standard debhelper-based catch-all rule which is used to get the whole build started:

%:
        dh $@ --buildsystem=pybuild

This tells dh to use pybuild as its build system. There's a lot of magic packed into this line, and the pybuild manpage goes into more detail, but essentially what this does is:

Builds the Python 3 package for all supported Python 3 versions.
Automatically detects the command to run the test suite.
Runs the Python 3 test suite.

Once all that's done, it properly installs all the files into binary packages.

You usually won't need debian/python3-foo.install files even if you have multiple binary packages, because again, pybuild does the magic for you (if PYBUILD_NAME is correctly set). You might need these if there are some non-standard installation tricks you need to implement.

Declaring dh-sequence-python3 as a build-dependency, or using dh $@ --with python3 with dh-python, adds the dh_python3 helper to your packaging process, which helps to make sure the resulting binary packages have properly-shaped metadata and contents. See its manual page for more detail.

CUSTOMIZATION

Pybuild can be configured via command line - by exporting environment variables in debian/rules that look like this:

PYBUILD_OPTION
PYBUILD_OPTION_$interpreter
PYBUILD_OPTION_$interpreter$version

For example:

If you don't want to use debian/*.install files you can set DESTDIR for each interpreter (note that you can use "python3" for all python3.X ones, this doesn't include -dbg interpreters, though):

export PYBUILD_DESTDIR_python3=debian/python3-foo/
export PYBUILD_DESTDIR_python3-dbg=debian/python3-foo-dbg/
export PYBUILD_DESTDIR_python3.1-dbg=debian/spam-pkg-with-python3.1-only/
export PYBUILD_DESTDIR_pypy=debian/pypy-foo/

If you want to add some custom arguments to Python 3's install command:

export PYBUILD_INSTALL_ARGS_python3=--install-scripts=/usr/share/foo/

You can disable (almost) any action via --disable / DISABLE env. var:

export PYBUILD_DISABLE=configure
export PYBUILD_DISABLE_python3.1=test

If you want to disable Build-Depends detection:

export PYBUILD_INTERPRETERS=python{version} python{version}-dbg
export PYBUILD_VERSIONS=2.7 3.3

If auto detection didn't detect the right build system:

export PYBUILD_SYSTEM=distutils

If you want to add a custom command before or after each iteration:

export PYBUILD_BEFORE_BUILD=echo {version} >> '{dir}/enabled'
export PYBUILD_AFTER_INSTALL=rm -rf '{destdir}/{install_dir}/foo/tests'

If your build system is not yet supported, you can use custom build system:

export PYBUILD_SYSTEM=custom
export PYBUILD_CLEAN_ARGS=rm -rf {build_dir}/python{version}/
export PYBUILD_CONFIGURE_ARGS=./configure --python={version}
export PYBUILD_BUILD_ARGS=make build --dir={build_dir} --python={version}
export PYBUILD_INSTALL_ARGS=make --destdir={destdir} --python={version} --install-dir={install_dir}
export PYBUILD_TEST_ARGS_python3=cd {build_dir}; python{version} -m discover

To disable a specific test (pytest), note the '-k $exp'

export PYBUILD_TEST_ARGS=-k 'not test_that_needs_internet'

To drop more than one test add additional tests for exclusion by using and not test_foo to the variable.

export PYBUILD_TEST_ARGS=-k 'not test_1 and not test_2'

The '-k $exp' command skips running the test, but the tests are still included in the collection. This will cause failures if the tests have includes that cannot be satisfied. In that case, the --ignore command can be used instead, which skips adding the test to the collection.

export PYBUILD_TEST_ARGS=--ignore path/to/test.py

the --ignore command may be called multiple times if needed.

export PYBUILD_TEST_ARGS=--ignore path/to/test1.py --ignore path/to/test2.py

To package a program instead of a module:

export PYBUILD_INSTALL_ARGS=--install-lib=/usr/share/packagename/ --install-scripts=/usr/share/packagename/

Resources

Pybuild manpage
Pybuild announcement email

Examples of packages using Pybuild on codesearch: search

introduction to pybuild and Python packaging talk at DebConf14: slides, video

PythonPybuild