Controlling Python From Rust Code

PyOxidizer can be used to embed Python in a Rust application.

This page documents what that looks like from a Rust code perspective.

Interacting with the pyembed Crate

When writing Rust code to interact with a Python interpreter, your primary area of contact will be with the pyembed crate.

The pyembed crate is a standalone crate maintained as part of the PyOxidizer project. This crate provides the core run-time functionality for PyOxidizer, such as the implementation of PyOxidizer’s custom importer. It also exposes a high-level API for initializing a Python interpreter and running code in it.

See The pyembed Rust Crate for full documentation on the pyembed crate. Controlling Python from Rust Code in particular describes how to interface with the embedded Python interpreter.

The following documentation will be unique to PyOxidizer’s use of the pyembed crate.

Using the Default OxidizedPythonInterpreterConfig

When using a PyOxidizer-generated Rust project and that project is configured to use PyOxidizer to build (the default), that project/crate’s build script will call into PyOxidizer to emit various build artifacts. This will process the PyOxidizer configuration file and write some files somewhere.

One of the files generated is a Rust source file containing a fn default_python_config() -> pyembed::OxidizedPythonInterpreterConfig which emits a pyembed::OxidizedPythonInterpreterConfig using the configuration from the PyOxidizer configuration file. This configuration is based off the PythonInterpreterConfig defined in the PyOxidizer Starlark configuration file.

The crate’s build script will set the DEFAULT_PYTHON_CONFIG_RS environment variable to the path to this file, exposing it to Rust code.

This all means that to use the auto-generated pyembed::OxidizedPythonInterpreterConfig instance with your Rust application, you simply need to do something like the following:

include!(env!("DEFAULT_PYTHON_CONFIG_RS"));

fn create_interpreter() -> Result<pyembed::MainPythonInterpreter> {
    // Calls function from include!()'d file.
    let config: pyembed::OxidizedPythonInterpreterConfig = default_python_config();

    pyembed::MainPythonInterpreter::new(config)
}

Using a Custom OxidizedPythonInterpreterConfig

If you don’t want to use the default pyembed::OxidizedPythonInterpreterConfig instance, that’s fine too! However, this will be slightly more complicated.

First, if you use an explicit OxidizedPythonInterpreterConfig, the PythonInterpreterConfig Starlark type defined in your PyOxidizer configuration file doesn’t matter that much. The primary purpose of this Starlark type is to derive the default OxidizedPythonInterpreterConfig Rust struct. And if you are using your own custom OxidizedPythonInterpreterConfig instance, you can ignore most of the arguments when creating the PythonInterpreterConfig instance.

An exception to this is the raw_allocator argument/field. If you are using a custom allocator (like jemalloc, mimalloc, or snmalloc), you will need to enable a Cargo feature when building the pyembed crate or else you will get a run-time error that the specified allocator is not available.

pyembed::OxidizedPythonInterpreterConfig::default() can be used to construct a new instance, pre-populated with default values for each field. The defaults should match what the PythonInterpreterConfig Starlark type would yield.

The main catch to constructing the instance manually is that the custom meta path importer won’t be able to service Python import requests unless you populate a few fields. In fact, if you just use the defaults, things will blow up pretty hard at run-time:

$ myapp
Fatal Python error: initfsencoding: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007fa0e2cbe9c0 (most recent call first):
Aborted (core dumped)

What’s happening here is that Python interpreter initialization hits a fatal error because it can’t import encodings (because it can’t locate the Python standard library) and Python’s C code is exiting the process. Rust doesn’t even get the chance to handle the error, which is why we’re seeing a segfault.

The reason we can’t import encodings is twofold:

  1. The default filesystem importer is disabled by default.

  2. No Python resources are being registered with the OxidizedPythonInterpreterConfig instance.

This error can be addressed by working around either.

To enable the default filesystem importer:

let mut config = pyembed::OxidizedPythonInterpreterConfig::default();
config.filesystem_importer = true;
config.sys_paths.push("/path/to/python/standard/library");

As long as the default filesystem importer is enabled and sys.path can find the Python standard library, you should be able to start a Python interpreter.

Hint

The sys_paths field will expand the special token $ORIGIN to the directory of the running executable. So if the Python standard library is in e.g. the lib directory next to the executable, you can do something like config.sys_paths.push("$ORIGIN/lib").

If you want to use the custom PyOxidizer Importer to import Python resources, you will need to update a handful of fields:

let mut config = pyembed::OxidizedPythonInterpreterConfig::default();
config.packed_resources = ...;
config.oxidized_importer = true;

The packed_resources field defines a reference to packed resources data (a PackedResourcesSource enum. This is a custom serialization format for expressing resources to make available to a Python interpreter. See Python Packed Resources for more. The easiest way to obtain this data blob is by using PyOxidizer and consuming the packed-resources build artifact/file, likely though include_bytes!. OxidizedFinder Meta Path Finder can also be used to produce these data structures.

Finally, setting oxidized_importer = true is necessary to enable oxidized_importer.OxidizedFinder.