Generic Python Embedding in Rust Applications

PyOxidizer can be used to produce artifacts facilitating the embedding of Python in a Rust application. This enables Rust developers to leverage PyOxidizer’s technology for linking an embedded Python and managing the Python interpreter at run-time without a build-time dependency on PyOxidizer. This can greatly simplify development workflows at the cost of not being able to utilize the full power of PyOxidizer during builds. If you would like to use PyOxidizer as a build dependency, see PyOxidizer Rust Projects instead.

Producing Embedding Artifacts

The pyoxidizer generate-python-embedding-artifacts command can be used to write Python embedding artifacts into an output directory. e.g.:

$ pyoxidizer generate-python-embedding-artifacts artifacts
$ ls artifacts
default_python_config.rs  libpython3.a  packed-resources  pyo3-build-config-file.txt stdlib tcl

This command essentially runs pyoxidizer run-build-script with a default configuration file that produces artifacts suitable for generic Python embedding scenarios.

The Written Artifacts

pyoxidizer generate-python-embedding-artifacts will write the following files.

A Linkable Python Library

On UNIX platforms, this will likely be named libpython3.a. On Windows, python3.dll and a pythonXY.dll (where XY is the major-minor Python version, e.g. 39).

The library can be linked to provide an embedded Python interpreter.

A Rust Source File Containing a Python Interpreter Config

The default_python_config.rs file contains the definition of a pyembed::OxidizedPythonInterpreterConfig Rust struct for defining an embedded Python interpreter. The config should just work with the other files produced.

You can include!(...) this file in your Rust program if you want. Or you can ignore it and write your own configuration.

Packed Resources for the Standard Library

A file containing the Python Packed Resources for the Python standard library will be written. This file can be used by oxidized_importer Python Extension to import the standard library efficiently.

PyO3 Build Configuration

A pyo3-build-config-file.txt file will be written defining a configuration for the pyo3-build-config crate which will link the libpython produced by this command.

To use this configuration, set the PYO3_CONFIG_FILE environment variable to its absolute path and Python should get linked the way PyOxidizer would link it.

Python Standard Library

The stdlib directory will contain a copy of the Python standard library as it existed in the source distribution.

Note

.pyc files are often not present and PyOxidizer doesn’t yet provide a turnkey way to produce these files.

Tcl/tk Support Files

The tcl directory will contain tcl/tk support files to support the tkinter Python module.

Exporting Python Symbols

The binary embedding libpython must export libpython’s public symbols in order to support loading external extension modules / shared libraries (which need to be able to resolve libpython’s public symbols).

If libpython is loaded as a library, its symbols should be exported automatically. However, if libpython is embedded in an executable (non-library) binary, the default linker behavior is likely to not export symbols.

To ensure a binary embedding libpython is exporting the proper symbols, you may need to define custom linker arguments.

On Linux, you typically pass -export-dynamic to the linker, often via -Wl,-export-dynamic.

On macOS, you typically pass -rdynamic.

From a Cargo build script, you can add these extra arguments by printing a cargo:rustc-link-arg={} line. e.g.

let target_os = std::env::var("CARGO_CFG_TARGET_OS").expect("CARGO_CFG_TARGET_OS not defined");

match target_os.as_str() {
    "linux" => {
        println!("cargo:rustc-link-arg=-Wl,-export-dynamic");
    }
    "macos" => {
        println!("cargo:rustc-link-arg=-rdynamic");
    }
    _ => {}
}

Example Workflows

Embed Python With pyo3

In this example, we will produce a Rust executable that uses the pyo3 crate for interfacing with an embedded Python interpreter. We will not use PyOxidizer’s pyembed crate or the oxidized_importer extension module for enhancing functionality of Python.

First, create a new Rust project:

$ cargo init --bin pyapp

Then edit its Cargo.toml to add the pyo3 dependency. e.g.

[package]
name = "pyapp"
version = "0.1.0"
edition = "2021"

[dependencies]
pyo3 = "0.14"

And define a src/main.rs:

use pyo3::prelude::*;

fn main() -> PyResult<()> {
    unsafe {
        pyo3::with_embedded_python_interpreter(|py| {
            py.run("print('hello, world')", None, None)
        })
    }
}

Now use pyoxidizer to generate the Python embedding artifacts:

$ pyoxidizer generate-python-embedding-artifacts pyembedded

And finally build the Rust project using the PyO3 configuration file to tell PyO3 how to link the Python library we just generated:

$ PYO3_CONFIG_FILE=$(pwd)/pyembedded/pyo3-build-config-file.txt cargo run

If you are doing this on a UNIX-like platform like Linux or macOS, chances are this fails with an error similar to the following:

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/usr/bin/python3'
  sys.base_prefix = '/install'
  sys.base_exec_prefix = '/install'
  sys.platlibdir = 'lib'
  sys.executable = '/usr/bin/python3'
  sys.prefix = '/install'
  sys.exec_prefix = '/install'
  sys.path = [
    '/install/lib/python39.zip',
    '/install/lib/python3.9',
    '/install/lib/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007ffa5abd9c80 (most recent call first):
<no Python frame>

This is because the embedded Python library doesn’t know how to locate the Python standard library. Essentially, the compiled Python library has some hard-coded defaults for where the Python standard library is located and its default logic is to search in those paths. The references to /install are referring to the build environment for the Python distributions.

The quick fix for this is to define the PYTHONPATH environment variable to the location of the Python standard library. e.g.:

$ PYO3_CONFIG_FILE=$(pwd)/pyembedded/pyo3-build-config-file.txt PYTHONPATH=pyembedded/stdlib cargo run
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
hello, world

We still get some warnings. But our embedded Python interpreter does work!

To make these config changes more permanent and to silence the remaining warnings, you’ll need to customize the initialization of the Python interpreter using C APIs like the Python Initialization Configuration APIs. This requires a fair bit of unsafe code.

Abstracting away the complexities of initializing the embedded Python interpreter is one of the reasons the pyembed Rust crate exists. So if you want a simpler approach, consider using pyembed for controlling the Python interpreter.

Embed Python with pyembed

In this example we’ll use the pyembed crate (part of the PyOxidizer project) for managing the embedded Python interpreter.

First, create a new Rust project:

$ cargo init --bin pyapp

Then edit its Cargo.toml to add the pyembed dependency. e.g.

[package]
name = "pyapp"
version = "0.1.0"
edition = "2021"

[dependencies]
# Check for the latest version in case these docs are out of date.
pyembed = "0.18"

And define a src/main.rs:

include!("../pyembedded/default_python_config.rs");

fn main() {
    // Get config from default_python_config.rs.
    let config = default_python_config();

    let interp = pyembed::MainPythonInterpreter::new(config).unwrap();

    // `py` is a `pyo3::Python` instance.
    interp.with_gil(|py| {
        py.run("print('hello, world')", None, None).unwrap();
    });

}

Now use pyoxidizer to generate the Python embedding artifacts:

$ pyoxidizer generate-python-embedding-artifacts pyembedded

And finally build the Rust project using the PyO3 configuration file to tell PyO3 how to link the Python library we just generated:

$ PYO3_CONFIG_FILE=$(pwd)/pyembedded/pyo3-build-config-file.txt cargo run
...
 Finished dev [unoptimized + debuginfo] target(s) in 3.87s
  Running `target/debug/pyapp`
hello, world

If all goes as expected, this should just work!