The Problem

Packaging and application distribution is a hard problem on multiple dimensions. For Python, large aspects of this problem space are more or less solved if you are distributing open source Python libraries and your target audience is developers (use pip and PyPI). But if you are distributing Python applications - standalone executables that use Python - your world can be much more complicated.

One of the primary reasons why distributing Python applications is difficult is because of the complex and often sensitive relationship between a Python application and the environment it runs in.

For starters we have the Python interpreter itself. If your application doesn't distribute the Python interpreter, you are at the whims of the Python interpreter provided by the host machine. You may want to target Python 3.7 only. But because Python 3.5 or 3.6 is the most recent version installed by many Linux distros, you are forced to support older Python versions and all their quirks and lack of features.

Going down the rabbit hole, even the presence of a supposedly compatible version of the Python interpreter isn't a guarantee for success! For example, the Python interpreter could have a built-in extension that links against an old version of a library. Just last week I was encountering weird SQlite bugs in Firefox's automation because Python was using an old version of SQLite with known bugs. Installing a modern SQLite fixed the problems. Or the interpreter could have modifications or extra installed packages interfering with the operation of your application. There are never-ending corner cases. And I can tell you from my experience with having to support the Firefox build system (which uses Python heavily) that you will encounter these corner cases given a broad enough user base.

And even if the Python interpreter on the target machine is fully compatible, getting your code to run on that interpreter could be difficult! Several Python applications leverage compiled extensions linking against Python's C API. Distributing the precompiled form of the extension can be challenging, especially when your code needs to link against 3rd party libraries, which may conflict with something on the target system. And, the precompiled extensions need to be built in a very delicate manner to ensure they can run on as many target machines as possible. But not distributing pre-built binaries requires the end-user be able to compile Python extensions. Not every user has such an environment and forcing this requirement on them is not user friendly.

From an application developer's point of view, distributing a copy of the Python interpreter along with your application is the only reliable way of guaranteeing a more uniform end-user experience. Yes, you will still have variability because every machine is different. But you've eliminated the the Python interpreter from the set of unknowns and that is a huge win. (Unfortunately, distributing a Python interpreter comes with a host of other problems such as size bloat, security/patching concerns, poking the OS packaging bears, etc. But those problems are for another post.)

Existing Solutions

There are tons of existing tools for solving the Python application distribution problem.

The approach that tools like Shiv and PEX take is to leverage Python's built-in support for running zip files. Essentially, if there is a zip file containing a __main__.py file and you execute python file.zip (or have a zip file with a #!/usr/bin/env python shebang), Python can load modules in that zip file and execute an application within. Pretty cool!

This approach works great if your execution environment supports shebangs (Windows doesn't) and the Python interpreter is suitable. But if you need to support Windows or don't have control over the execution environment and can't guarantee the Python interpreter is good, this approach isn't suitable.

As stated above, we want to distribute the Python interpreter with our application to minimize variability. Let's talk about tools that do that.

XAR is a pretty cool offering from Facebook. XAR files are executables that contain SquashFS filesystems. Upon running the executable, SquashFS filesystems are created. For Python applications, the XAR contains a copy of the Python interpreter and all your Python modules. At run-time, these files are extracted to SquashFS filesystems and the Python interpreter is executed. If you squint hard enough, it is kind of like a pre-packaged, executable virtualenv which also contains the Python interpreter.

XARs are pretty cool (and aren't limited to Python). However, because XARs rely on SquashFS, they have a run-time requirement on the target machine. This is great if you only need to support Linux and macOS and your target machines support FUSE and SquashFS. But if you need to support Windows or a general user population without SquashFS support, XARs won't help you.

Zip files and XARs are great for enterprises that have tightly controlled environments. But for a general end-user population, we need something more robust against variance among target machines.

There are a handful of tools for packaging Python applications along with the Python interpreter in more resilient manners.

Nuitka converts Python source to C code then compiles and links that C code against libpython. You can perform a static link and compile everything down to a single executable. If you do the compiling properly, that executable should just work on pretty much every target machine. That's pretty cool and is exactly the kind of solution application distributors are looking for: you can't get much simpler than a self-contained executable! While I'd love to vouch for Nuitka and recommend using it, I haven't used it so can't. And I'll be honest, the prospect of compiling Python source to C code kind of terrifies me. That effectively makes Nuitka a new Python implementation and I'm not sure I can (yet) place the level of trust in Nuitka that I have for e.g. CPython and PyPy.

And that leads us to our final category of tools: freezing your code. There are a handful of tools like PyInstaller which automate the process of building your Python application (often via standard setup.py mechanisms), assembling all the requisite bits of the Python interpreter, and producing an artifact that can be distributed to end users. There are even tools that produce Windows installers, RPMs, DEBs, etc that you can sign and distribute.

These freezing tools are arguably the state of the art for Python application distribution to general user populations. On first glance it seems like all the needed tools are available here. But there are cracks below the surface.

Issues with Freezing

A common problem with freezing is it often relies on the Python interpreter used to build the frozen application. For example, when building a frozen application on Linux, it will bundle the system's Python interpreter with the frozen application. And that interpreter may link against libraries or libc symbol versions not available on all target machines. So, the build environment has to be just right in order for the binaries to run on as many target systems as possible. This isn't an insurmountable problem. But it adds overhead and complexity to application maintainers.

Another limitation is how these frozen applications handle importing Python modules.

Multiple tools take the approach of embedding an archive (usually a zip file) in the executable containing the Python standard library bits not part of libpython. This includes C extensions (compiled to .so or .pyd files) and Python source (.py) or bytecode (.pyc) files. There is typically a step - either at application start time or at module import time - where a file is extracted to the filesystem such that Python's filesystem-based importer can load it from there.

For example, PyInstaller extracts the standard library to a temporary directory at application start time (at least when running in single file mode). This can add significant overhead to the startup time of applications - more than enough to blow through people's ability to perceive something as instantaneous. This is acceptable for long-running applications. But for applications (like CLI tools or support tools for build systems), the overhead can be a non-starter. And, the mere fact that you are doing filesystem write I/O establishes a requirement that the application have write access to the filesystem and that write I/O can perform reasonably well lest application performance suffer. These can be difficult pills to swallow!

Another limitation is that these tools often assume the executable being produced is only a Python application. Sometimes Python is part of a larger application. It would be useful to produce a library that can easily be embedded within a larger application.

Improving the State of the Art

Existing Python application distribution mechanisms don't tick all the requirements boxes for me. We have tools that are suitable for internal distribution in well-defined enterprise environments. And we have tools that target general user populations, albeit with a burden on application maintainers and often come with a performance hit and/or limited flexibility.

I want something that allows me to produce a standalone, single file executable containing a Python interpreter, the Python standard library (or a subset of it), and all the custom code and resources my application needs. That executable should not require any additional library dependencies beyond what is already available on most target machines (e.g. libc). That executable should not require any special filesystem providers (e.g. FUSE/SquashFS) nor should it require filesystem write access nor perform filesystem write I/O at run-time. I should be able to embed a Python interpreter within a larger application, without the overhead of starting the Python interpreter if it isn't needed.

No existing solution ticks all of these boxes.

So I set out to build one.

One problem is producing a Python interpreter that is portable and fully-featured. You can't punt on this problem because if the core Python interpreter isn't produced in just the right way, your application will depend on libraries or symbol versions not available in all environments.

I've created the python-build-standalone project for automating the process of building Python interpreters suitable for use with standalone, distributable Python applications. The project produces (and has available for download) binary artifacts including a pre-compiled Python interpreter and object files used for compiling that interpreter. The Python interpreter is compiled with PGO/LTO using a modern Clang, helping to ensure that Python code runs as fast as it can. All of Python's dependencies are compiled from source with the modern toolchain and everything is aggressively statically linked to avoid external dependencies. The toolchain and pre-built distribution are available for downstream consumers to compile Python extensions with/against.

It's worth noting that use of a modern Clang toolchain is likely sufficiently different from what you use today. When producing manylinux wheels, it is recommended to use the pypa/manylinux Docker images. These Docker images are based on CentOS 5 (for maximum libc and other system library compatibility). While they do install a custom toolchain, Python and any extensions compiled in that environment are compiled with GCC 4.8.2 (as of this writing). That's a GCC from 2013. A lot has changed in compilers since 2013 and building Python and extensions with a compiler released in 2018 should result in various benefits (faster code, better warnings, etc).

If producing custom CPython builds for standalone distribution interests you, you should take a look at how I coerced CPython to statically link all extensions. Spoiler: it involves producing a custom-tailored Modules/Setup.local file that bypasses setup.py, along with some Makefile hacks. Because the build environment is deterministic and isolated in a container, we can get away with some ugly hacks.

A statically linked libpython from which you can produce a standalone binary embedding Python is only the first layer in the onion. The next layer is how to handle the Python standard library.

libpython only contains the code needed to run the core bits of the Python interpreter. If we attempt to run a statically linked python executable without the standard library in the filesystem, things fail pretty fast:

$ rm -rf lib
$ bin/python
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: initfsencoding: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007fe9a3432740 (most recent call first):
Aborted (core dumped)

I'll spare you the details for the moment, but initializing the CPython interpreter (via Py_Initialize() requires that parts of the Python standard library be available). This means that in order to fulfill our dream of a single file executable, we will need custom code that teaches the embedded Python interpreter to load the standard library from within the binary... somehow.

As far as I know, efficient embedded standard library handling without run-time requirements does not exist in the current Python packaging/distribution ecosystem. So, I had to devise something new.

Enter PyOxidizer. PyOxidizer is a collection of Rust crates that facilitate building an embeddable Python library, which can easily be added to an executable. We need native code to interface with the Python C APIs in order to influence Python interpreter startup. It is 2018 and Rust is a better C/C++, so I chose Rust for this driver functionality instead of C. Plus, Rust's integrated build system makes it easier to automate the integration of the custom Python interpreter files into binaries.

The role of PyOxidizer is to take the pre-built Python interpreter files from python-build-standalone, combine those files with any other Python files needed to run an application, and marry them to a Rust crate. This Rust crate can trivially be turned into a self-contained executable containing a Python application. Or, it can be combined with another Rust project. Or it can be emitted as a library and integrated with a non-Rust application. There's a lot of flexibility by design.

The mechanism I used for embedding the Python standard library into a single file executable without incurring explicit filesystem access at run-time is (I believe) new, novel, and somewhat crazy. Let me explain how it works.

First, there are no .so/.pyd shared library compiled Python extensions to worry about. This is because all compiled extensions are statically linked into the Python interpreter. To the interpreter, they exist as built-in modules. Typically, a CPython build will have some modules like _abc, _io, and sys provided by built-in modules. Modules like _json exist as standalone shared libraries that are loaded on demand. python-build-standalone's modifications to CPython's build system converts all these would-be standalone shared libraries into built-in modules. (Because we distribute the object files that compose the eventual libpython, it is possible to filter out unwanted modules to cut down on binary size if you don't want to ship a fully-featured Python interpreter.) Because there are no standalone shared libraries providing Python modules, we don't have the problem of needing to load a shared library to load a module, which would undermine our goal of no filesystem access to import modules. And that's a good thing, too, because dlopen() requires a path: you can't load a shared library from a memory address. (Fun fact: there are hacks like dlopen_with_offset() that provide an API to load a library from memory, but they require a custom libc. Google uses this approach for their internal single-file Python application solution.)

From the python-build-standalone artifacts, PyOxidizer collects all files belonging to the Python standard library (notably .py and .pyc files). It also collects other source, bytecode, and resource files needed to run a custom application.

The relevant files are assembled and serialized into data structures which contain the names of the resources and their raw content. These data structures are made available to Rust as &'static [u8] variables (essentially a static void* if you don't speak Rust).

Using the rust-cpython crate, PyOxidizer defines a custom Python extension module implemented purely in Rust. When loaded, the module parses the data structures containing available Python resource names and data into HashMap<&str, &[u8]> instances. In other words, it builds a native mapping from resource name to a pointer to its raw data. The Rust-implemented module exports to Python an API for accessing that data. From the Python side, you do the equivalent of MODULES.get_code('foo') to request the bytecode for a named Python module. When called, the Rust code will perform the lookup and return a memoryview instance pointing to the raw data. (The use of &[u8] and memoryview means that embedded resource data is loaded from its static, read-only memory location instead of copied into a data structure managed by Python. This zero copy approach translates to less overhead for importing modules. Although, the memory needs to be paged in by the operating system. So on slow filesystems, reducing I/O and e.g. compressing module data might be a worthwhile optimization. This can be a future feature.)

Making data embedded within a binary available to a Python module is relatively easy. I'm definitely not the first person to come up with this idea. What is hard - and what I might be the first person to actually do - is how you make the Python module importing mechanism load all standard library modules via such a mechanism.

With a custom extension module built-in to the binary exposing module data, it should just be a matter of registering a custom sys.meta_path importer that knows how to load modules from that custom location. This problem turns out to be quite hard!

The initialization of a CPython interpreter is - as I've learned - a bit complex. A CPython interpreter must be initialized via Py_Initialize() before any Python code can run. That means in order to modify sys.meta_path, Py_Initialize() must finish.

A lot of activity occurs under the hood during initialization. Applications embedding Python have very little control over what happens during Py_Initialize(). You can change some superficial things like what filesystem paths to use to bootstrap sys.path and what encodings to use for stdio descriptors. But you can't really influence the core actions that are being performed. And there's no mechanism to directly influence sys.meta_path before an import is performed. (Perhaps there should be?)

During Py_Initialize(), the interpreter needs to configure the encodings for the filesystem and the stdio descriptors. Encodings are loaded from Python modules provided by the standard library. So, during the course of Py_Initialize(), the interpreter needs to import some modules originally backed by .py files. This creates a dilemma: if Py_Initialize() needs to import modules in the standard library, the standard library is backed by memory and isn't available to known importing mechanisms, and there's no opportunity to configure a custom sys.meta_path importer before Py_Initialize() runs, how do you teach the interpreter about your custom module importer and the location of the standard library modules needed by Py_Initialize()?

This is an extremely gnarly problem and it took me some hours and many false leads to come up with a solution.

My first attempt involved the esoteric frozen modules feature. (This work predated the use of a custom data structure and module containing modules data.) The Python interpreter has a const struct _frozen* PyImport_FrozenModules data structure defining an array of frozen modules. A frozen module is defined by its module name and precompiled bytecode data (roughly equivalent to .pyc file content). Partway through Py_Initialize(), the Python interpreter is able to import modules. And one of the built-in importers that is automatically registered knows how to load modules if they are in PyImport_FrozenModules!

I attempted to audit Python interpreter startup and find all modules that were imported during Py_Initialize(). I then defined a custom PyImport_FrozenModules containing these modules. In theory, the import of these modules during Py_Initialize() would be handled by the FrozenImporter and everything would just work: if I were able to get Py_Initialize() to complete, I'd be able to register a custom sys.meta_path importer immediately afterwards and we'd be set.

Things did not go as planned.

FrozenImporter doesn't fully conform to the PEP 451 requirements for setting specific attributes on modules. Without these attributes, the from . import aliases statement in encodings/__init__.py fails because the importer is unable to resolve the relative module name. Derp. One would think CPython's built-in importers would comply with PEP 451 and that all of Python's standard library could be imported as frozen modules. But this is not the case! I was able to hack around this particular failure by using an absolute import. But I hit another failure and did not want to excavate that rabbit hole. Once I realized that FrozenImporter was lacking mandated module attributes, I concluded that attempting to use frozen modules as a general import-from-memory mechanism was not viable. Furthermore, the C code backing FrozenImporter walks the PyImport_FrozenModules array and does a string compare on the module name to find matches. While I didn't benchmark, I was concerned that un-indexed scanning at import time would add considerable overhead when hundreds of modules were in play. (The C code backing BuiltinImporter uses the same approach and I do worry CPython's imports of built-in extension modules is causing measurable overhead.)

With frozen modules off the table, I needed to find another way to inject a custom module importer that was usable during Py_Initialize(). Because we control the source Python interpreter, modifications to the source code or even link-time modifications or run-time hacks like trampolines weren't off the table. But I really wanted things to work out of the box because I don't want to be in the business of maintaining patches to Python interpreters.

My foray into frozen modules enlightened me to the craziness that is the bootstrapping of Python's importing mechanism.

I remember hearing that the Python module importing mechanism used to be written in C and was rewritten in Python. And I knew that the importlib package defined interfaces allowing you to implement your own importers, which could be registered on sys.meta_path. But I didn't know how all of this worked at the interpreter level.

The internal initimport() C function is responsible for initializing the module importing mechanism. It does the equivalent of import _frozen_importlib, but using the PyImport_ImportFrozenModule() API. It then manipulates some symbols and calls _frozen_importlib.install() with references to the sys and imp built-in modules. Later (in initexternalimport()), a _frozen_importlib_external module is imported and has code within it executed.

I was initially very confused by this because - while there are references to _frozen_importlib and _frozen_importlib_external all over the CPython code base, I couldn't figure out where the code for those modules actually lived! Some sleuthing of the build directory eventually revealed that the files Lib/importlib/_bootstrap.py and Lib/importlib/_bootstrap_external.py were frozen to the module names _frozen_importlib and _frozen_importlib_external, respectively.

Essentially what is happening is the bulk of Python's import machinery is implemented in Python (rather than C). But there's a chicken-and-egg problem where you can't run just any Python code (including any import statement) until the interpreter is partially or fully initialized.

When building CPython, the Python source code for importlib._bootstrap and importlib._bootstrap_external are compiled to bytecode. This bytecode is emitted to .h files, where it is exposed as a static char *. This bytecode is eventually referenced by the default PyImport_FrozenModules array, allowing the modules to be imported via the frozen importer's C API, which bypasses the higher-level importing mechanism, allowing it to work before the full importing mechanism is initialized.

initimport() and initexternalimport() both call Python functions in the frozen modules. And we can clearly look at the source of the corresponding modules and see the Python code do things like register the default importers on sys.meta_path.

Whew, that was a long journey into the bowels of CPython's internals. How does all this help with single file Python executables?

Well, the predicament that led us down this rabbit hole was there was no way to register a custom module importer before Py_Initialize() completes and before an import is attempted during said Py_Initialize().

It took me a while, but I finally realized the frozen importlib._bootstrap_external module provided the window I needed! importlib._bootstrap_external/_frozen_importlib_external is always executed during Py_Initialize(). So if you can modify this module's code, you can run arbitrary code during Py_Initialize() and influence Python interpreter configuration. And since _frozen_importlib_external is a frozen module and the PyImport_FrozenModules array is writable and can be modified before Py_Initialize() is called, all one needs to do is replace the _frozen_importlib / _frozen_importlib_external bytecode in PyImport_FrozenModules and you can run arbitrary code during Python interpreter startup, before Py_Initialize() completes and before any standard library imports are performed!

My solution to this problem is to concatenate some custom Python code to importlib/_bootstrap_external.py. This custom code defines a sys.meta_path importer that knows how to use our Rust-backed built-in extension module to find and load module data. It redefines the _install() function so that this custom importer is registered on sys.meta_path when the function is called during Py_Initialize(). The new Python source is compiled to bytecode and the PyImport_FrozenModules array is modified at run-time to point to the modified _frozen_importlib_external implementation. When Py_Initialize() executes its first standard library import, module data is provided by the custom sys.meta_path importer, which grabs it from a Rust extension module, which reads it from a read-only data structure in the executable binary, which is converted to a Python memoryview instance and sent back to Python for processing.

There's a bit of magic happening behind the scenes to make all of this work. PyOxidizer attempts to hide as much of the gory details as possible. From the perspective of an application maintainer, you just need to define a minimal config file and it handles most of the low-level details. And there's even a higher-level Rust API for configuring the embedded Python interpreter, should you need it.

python-build-standalone and PyOxidizer are still in their infancy. They are very much alpha quality. I consider them technology previews more than usable software at this point. But I think enough is there to demonstrate the viability of using Rust as the build system and run-time glue to build and distribute standalone applications embedding Python.

Time will tell if my utopian vision of zero-copy, no explicit filesystem I/O for Python module imports will pan out. Others who have ventured into this space have warned me that lots of Python modules rely on __file__ to derive paths to other resources, which are later stat()d and open()d. __file__ for in-memory modules doesn't exactly make sense and can't be operated on like normal paths/files. I'm not sure what the inevitable struggles to support these modules will lead to. Maybe we'll have to extract things to temporary directories like other standalone Python applications. Maybe PyOxidizer will take off and people will start using the ResourceReader API, which is apparently the proper way to do these things these days. (Caveat: PyOxidizer doesn't yet implement this API but support is planned.) Time will tell. I'm not opposed to gross hacks or writing more code as needed.

Producing highly distributable and performant Python applications has been far too difficult for far too long. My primary goal for PyOxidizer is to lower these barriers. By leveraging Rust, I also hope to bring Python and Rust closer together. I want to enable applications and libraries to effortlessly harness the powers of both of these fantastic programming languages.

Again, PyOxidizer is still in its infancy. I anticipate a significant amount of hacking over the holidays and hope to share updates in the weeks ahead. Until then, please leave comments, watch the project on GitHub, file issues for bugs and feature requests, etc and we'll see where things lead.