Porting a Python Application to Rust¶
PyOxidizer can be used to gradually port a Python application to Rust. What we mean by this is that Python code in an application would slowly be rewritten in Rust.
Overview¶
When porting a Python application to Rust, the goal is to port Python code - and possibly Python C extension code - to Rust. Parts of the Rust code will presumably need to call into Python code and vice-versa.
When porting code to Rust, there are essentially two flavors of Rust code that will be written and executed:
Vanilla Rust code
Python-flavored Rust code
Vanilla Rust code is standard Rust code. It is what you would write if authoring a Rust-only project.
Python-flavored Rust code is Rust code that interacts with the Python C
API. It is regular Rust code, of course, but it is littered with references
to PyObject and function calls into the Python C API (although these
function calls may be abstracted so you don’t have to use unsafe
).
These different flavors of Rust code dictate different approaches to porting. Both flavors/approaches can be used simultaneously when porting an application to Rust.
Vanilla Rust code will supplement the boilerplate Rust code that PyOxidizer uses to define and build a standalone executable embedded Python. See Extending Rust Projects for more.
Python-flavored Rust code typically involves writing Python extension modules in Rust. In this approach, you create a Python extension modules implemented in Rust and then make them available to the Python interpreter, which is managed by a Rust project.
Extending Rust Projects¶
When building an application from a standalone pyoxidizer.bzl
file,
PyOxidizer creates and builds a temporary, boilerplate Rust project behind
the scenes. This Rust project has just enough code to initialize and run an
embedded Python interpreter. That’s the extent of the Rust code.
PyOxidizer also supports persistent Rust projects. In this mode, you have full control over the Rust project and can add custom Rust code to it as you desire. In this mode, you can run Rust code independent of the Python interpreter.
Supplementing the Rust code contained in your executable gives you the power to run arbitrary Rust code however you see fit. Here are some common scenarios this can enable:
Implementing argument parsing in Rust instead of Python. This could allow you to parse out the sub-command being invoked and dispatch to pure Rust code paths if possible, falling back to running Python code only if necessary.
Running a forking server, which doesn’t start a Python interpreter until an event occurs.
Starting a thread with a high-performance application component implemented in Rust. For example, you could run a thread servicing a high-performance logging subsystem or HTTP server implemented in Rust and have that thread interact with a Python interpreter via a pipe or some other handle.
Getting Started¶
To extend a Rust project with custom Rust code, you’ll first want to materialize the boilerplate Rust project used by PyOxidizer:
$ pyoxidizer init-rust-project myapp
See PyOxidizer Rust Projects for details on the files materialized by this command.
If you are using version control, now would be a good time to add the created files to version control. e.g.:
$ git add myapp
$ git commit -m 'create boilerplate PyOxidizer project'
From here, your next steps are to modify the Rust project to do something new and different.
The auto-generated src/main.rs
file contains the main()
function used
as the entrypoint for the Rust executable. The default file will simply
instantiate a Python interpreter from a configuration, run that interpreter,
then exit the process.
To extend your application with custom Rust code, simply add custom code to
main()
. e.g.
fn main() {
println!("hello from Rust!")
// Code auto-generated by ``pyoxidizer init-rust-project`` goes here.
// ...
}
That is literally all there is to it!
To build your custom Rust project, pyoxidizer build
is the most robust way
to do that. But it is also possible to use cargo build
.
What Can Go Wrong¶
pyoxidizer
Not Found or Rust Code Version Mismatch¶
When using cargo build
, the pyoxidizer
executable will be invoked behind
the scenes. This requires that executable to be on PATH
and for the version
to be compatible with the Rust code you are trying to build. (The Rust APIs do
change from time to time.)
If the pyoxidizer
executable is not on PATH
or its version doesn’t
match the Rust code, you can forcefully tell the Rust build system which
pyoxidizer
executable to use:
$ PYOXIDIZER_EXE=/path/to/pyoxidizer cargo build
thread 'main' panicked at 'jemalloc is not available in this build configuration'
¶
If you see this error, the problem is that the Python interpreter configuration
says to use jemalloc as the memory allocator but the Rust project was built
without jemalloc support. This is likely because the default Rust project
features in Cargo.toml
don’t include jemalloc by default.
You can resolve this issue by either disabling jemalloc in the Python configuration or by enabling jemalloc in Rust.
To disable jemalloc, open your pyoxidizer.bzl
file and find the
definition of allocator_backend
. You can set it to raw_allocator="default"
so Python uses the system memory allocator instead of jemalloc.
To enable jemalloc, you have a few options.
First, you could build the Rust project with jemalloc support:
$ cargo build --features allocator-jemalloc
Or, you modify Cargo.toml
so the allocator-jemalloc feature is enabled by
default:
.. code-block:: toml
[features] default = [“build-mode-pyoxidizer-exe”, “allocator-jemalloc”]
jemalloc is typically a faster allocator than the system allocator. So if you care about performance, you may want to use it.
Implementing Python Extension Modules in Rust¶
If you want to port a Python application to Rust, chances are that you will need to have Rust and Python code interact with each other. A common way to do this is to implement Python extensions in Rust so that Rust code will be invoked as a Python interpreter is running.
There are two ways Rust-implemented Python extension modules can be consumed by PyOxidizer:
Define them via Python packaging tools (e.g. via a
setup.py
file for your Python package).Define them in Rust code and register them as a built-in extension module.
Python Built Rust Extension Modules¶
If you’ve defined a Rust Python extension module via a Python package
build tool (e.g. inside a setup.py
), PyOxidizer should automatically
detect said extension module as part of packaging the corresponding Python
package: there is no need to take special action to tell PyOxidizer it is
a Rust extension, as this is all handled by Python packaging tools invoked
as part of processing your pyoxidizer.bzl
file.
See Packaging User Guide for more.
The topic of authoring Python extension modules implemented in Rust is
arguably outside the scope of this documentation. A search engine search
for Rust Python extension
should set you on the right track.
Built-in Rust Extension Modules¶
A Python extension module is defined as a PyInit__<name>
function which
is called to initialize an extension module. Typically, Python extension
modules are compiled as standalone shared libraries, which are then loaded
into a process, after which their PyInit__<name>
function is called.
But Python has an additional mechanism for defining extension modules:
built-ins. A built-in extension module is simply an extension module
whose PyInit__<name>
function is already present in the process address
space. Typically, these are extensions that are part of the Python distribution
itself and are compiled directly into libpython
.
When you instantiate a Python interpreter, you give it a list of the
available built-in Python extension modules. And PyOxidizer’s pyembed
crate allows you to supplement the default list with custom extensions.
To use built-in extension modules implemented in Rust, you’ll need to implement said extension module in Rust, either as part of your application’s Rust crate or as part of a different crate. Either way, you’ll need to extend the boilerplate Rust project code (see Extending Rust Projects) and tell it about additional built-in extension modules. See Adding Extension Modules At Run-Time for instructions on how to do this.
The tricky part here is implementing your Rust extension module.
You probably want to use the cpython
or PyO3 Rust crates for interfacing with the
CPython API, as these provide an interface that is more ergonomic and doesn’t
require use of unsafe { }
. Use of these crates is beyond the scope of the
PyOxidizer documentation.
If you attempt to use the cpython
or PyO3
macros for defining a
Python extension module, you’ll likely run into problems because these assume
that extension modules are standalone shared libraries, which isn’t the case for
built-in extension modules!
If you attempt to use a separate Rust crate to define your extension module,
you may run into Python symbol issues at link time because the build system
for the cpython
and PyO3
crates will use their own logic for locating
a Python interpreter and that interpreter may not have a configuration that
is compatible with the one embedded in your PyOxidizer binary!
At the end of the day, all you need to register a built-in extension module
with PyOxidizer is an extern "C" fn () -> *mut python3_sys::PyObject
. Here
is the boilerplate for defining a Python extension module in Rust (this uses
the cpython
crate).
use python3_sys as pyffi;
use cpython::{PyErr, PyModule, PyObject};
static mut MODULE_DEF: pyffi::PyModuleDef = pyffi::PyModuleDef {
m_base: pyffi::PyModuleDef_HEAD_INIT,
m_name: std::ptr::null(),
m_doc: std::ptr::null(),
m_size: std::mem::size_of::<ModuleState>() as isize,
m_methods: 0 as *mut _,
m_slots: 0 as *mut _,
m_traverse: None,
m_clear: None,
m_free: None,
};
#[allow(non_snake_case)]
pub extern "C" fn PyInit_my_module() -> *mut pyffi::PyObject {
let py = unsafe { cpython::Python::assume_gil_acquired() };
unsafe {
if MODULE_DEF.m_name.is_null() {
MODULE_DEF.m_name = "my_module".as_ptr() as *const _;
MODULE_DEF.m_doc = "usage docs".as_ptr() as *const _;
}
}
let module = unsafe { pyffi::PyModule_Create(&mut MODULE_DEF) };
if module.is_null() {
return module;
}
let module = match unsafe { pyffi::from_owned_ptr(py, module).cast_into::<PyModule>(py) } {
Ok(m) => m,
Err(e) => {
PyErr::from(e).restore(py);
return std::ptr::null_mut();
}
};
match module_init(py, &module) {
Ok(()) => module.into_object().steal_ptr(),
Err(e) => {
e.restore(py);
std::ptr::null_mut()
}
}
}
If you want a concrete example of what this looks like and how to do things like
define Python types and have Python functions implemented in Rust, do a search for
PyInit_oxidized_importer
in the source code of the pyembed
crate (which
is part of the PyOxidizer repository) and go from there.
The documentation for authoring Python extension modules and using the Python C API is well beyond the scope of this document. A good place to start is the official documentation.