Technical Notes¶
How It Works¶
The first thing the build.py scripts do is bootstrap an environment
for building Python. Linux builds use Docker images based on deterministic
Debian snapshots selected for each target:
x86-64 targets use Debian Jessie and a prebuilt Clang toolchain.
aarch64 targets use Debian Stretch and a prebuilt Clang toolchain.
Other cross-compiled targets generally use Debian Stretch and Debian-provided GCC cross-compilers.
riscv64 targets use Debian Buster and a Debian-provided GCC cross-compiler.
The selected toolchain is used to build Python’s dependencies (OpenSSL, ncurses, libedit, SQLite, etc.). Finally, Python itself is built.
Python is built in such a way that extensions are statically linked
against their dependencies. For example, instead of the sqlite3 Python
extension having a run-time dependency against libsqlite3.so, the
SQLite symbols are statically inlined into the Python extension object
file. Extension modules are statically linked into python rather than
provided as shared extensions in the lib-dynload directory.
_dbm and _tkinter are handled differently. On dynamically linked
Linux builds, _dbm is provided as a shared extension with Berkeley DB
statically linked. This allows the module to be easily removed when the
Berkeley DB license is undesired (see the DBM section below). On macOS,
_dbm instead uses the system NDBM implementation. _tkinter is
also provided as a shared extension, with Tcl and Tk dynamically linked,
on builds that support shared extensions.
From the built Python, we produce an archive containing the raw Python
distribution (as if you had run make install) as well as other files
useful for downstream consumers.
Setup.local Hackery¶
Starting with Python 3.12, C extension modules are configured and built
using configure, Modules/Setup.stdlib, and Makefile. A
generated Modules/Setup.local file disables selected modules and
overrides whether others are linked statically or built as shared
libraries.
Prior to 3.12, many extensions were configured and built using setup.py
scripts. These scripts do not provide much flexibility and rely on default
behaviors in distutils, as well as other inline code in setup.py.
This default behavior is often undesirable for our desired outcome of
producing a standalone Python distribution.
Because of this, when building Python prior to 3.12, a custom Setup.local
file is generated that builds all C extensions in a specific manner.
The undesirable behavior of setup.py is bypassed and the Python C
extensions are compiled just the way we want.
Dependency Notes¶
DBM¶
Python has the option of building its _dbm extension against NDBM,
GDBM, and Berkeley DB. GDBM and its NDBM compatibility libraries are
licensed under GNU GPL Version 3. Modern versions of Berkeley DB are
licensed under GNU AGPL v3. Versions 6.0.19 and older are licensed under
the more permissive Sleepycat License.
On Linux, we build the _dbm extension against Berkeley DB 6.0.19. On
macOS, _dbm uses the NDBM implementation provided by the system
libSystem library instead.
We explicitly disable the _gdbm extension on all targets to avoid
the GPL dependency.
readline / libedit / ncurses¶
Python has the option of building its readline extension against
either libreadline or libedit. libreadline is licensed under
GNU GPL Version 3, and libedit has a more permissive license.
libedit/libreadline link against a curses library, most likely
ncurses. And ncurses has tie-ins with a terminal database. This
is a thorny situation, as terminal databases can be difficult to
distribute because end-users often want software to respect their
terminal databases. But for that to work, ncurses needs to be compiled
in a way that respects the user’s environment.
On macOS, we use the system libedit and libncurses, which is
typically provided in /usr/lib.
On Linux, we build libedit and ncurses from source and statically
link against their respective libraries. Project releases before 2023 linked
against readline on Linux.
gettext / locale Module¶
The locale Python module exposes some functionality from the gettext
software (specifically libintl). (Technically, this functionality is exposed
from the _locale C extension module and locale re-exports symbols.)
gettext is GPL version 3 or later licensed. And having it statically linked
in the Python distribution via the _locale module can have licensing
implications.
Python’s configure script probes for the ability to compile/link with
-lintl. If it works, Python is linked against libintl. If it doesn’t,
libintl is omitted. (Search configure for ac_cv_lib_intl_textdomain
and -lintl references.)
With the container based build environment on Linux, presence of gettext
and libintl is deterministic. However, on macOS where there is no
sandboxing of the build environment, Python’s configure script can find and
use a gettext/libintl installed outside the system default (e.g. via
Homebrew or MacPorts). This can result in the built Python referencing a shared
library not reliably present on every macOS machine. So our build system
disables the configure check.
This means that the gettext/libintl features in the Python distribution
are not available.
libnsl / nis Module¶
The nis Python extension module has a dependency on libnsl.
libnsl has historically been in base Linux distribution installations.
But it is being phased away, with it being an optional install in modern
versions of Fedora and RHEL.
Because the nis extension is perceived to be likely unused functionality,
we’ve decided to not build it instead of adding complexity to deal with
the libnsl dependency. See further discussion in
https://github.com/astral-sh/python-build-standalone/issues/51.
The nis module was deprecated in Python 3.11 and removed in 3.13.
Upgrading CPython¶
This section documents some of the work that needs to be performed when upgrading CPython major versions.
Review Release Notes¶
CPython’s release notes often have a section on build system changes. e.g. https://docs.python.org/3/whatsnew/3.13.html#build-changes. These are a must review.
Modules/Setup¶
The Modules/Setup file defines the default extension build settings.
We need to audit it for differences such as added/removed extensions, changes to compile settings, etc just in case we have special code handling an extension defined in this file.
See code in cpython.py dealing with this file.