I am very excited to announce the 0.8 release of PyOxidizer, a modern Python application packaging tool. You can find the full changelog in the docs. First time user? See the Getting Started documentation.

Foremost, I apologize that this release took so long to publish (0.7 was released on 2020-04-09). I fervently believe that frequent releases are a healthy software development practice. And 6 months between PyOxidizer releases was way too long. Part of the delay was due to world events (it has proven difficult to focus on... anything given a global pandemic, social unrest, and wildfires further undermining any resemblance of lifestyle normalcy in California). Another contributing factor was I was waiting on a few 3rd party Rust crates to have new versions published to crates.io (you can't release a crate to crates.io unless all your dependencies are also published there).

Release delay and general life hardships aside, the 0.8 release is here and it is full of notable improvements!

Python 3.8 and 3.9 Support

PyOxidizer 0.8 now targets Python 3.8 by default and support for Python 3.9 is available by tweaking configuration files. Previously, we only supported Python 3.7 and this release drops support for Python 3.7. I feel a bit bad for dropping compatibility. But Python 3.8 introduced a new C API for initializing Python interpreters (thank you Victor Stinner!) and this makes PyOxidizer's run-time code for interfacing with Python interpreters vastly simpler. I decided that given the beta nature of PyOxidizer, it wasn't worth maintaining complexity to continue to support Python 3.7. I'm optimistic that I'll be able to support Python 3.8 as a baseline for a while.

Better Default Packaging Settings

PyOxidizer started as a science experiment of sorts to see if I could achieve the elusive goal of producing a single file executable providing a Python application. I was successful in proving this hypothesis. But the cost to achieving this outcome was rather high in terms of end-user experience: in order to produce single file executables, you had to break a lot of assumptions about how Python typically works and this in turn broke a lot of Python code and packages in the wild.

In other words, PyOxidizer's opinionated defaults of producing a single file executable were externalizing hardship on end-users and preventing them from using PyOxidizer.

PyOxidizer 0.8 contains a handful of changes to defaults that should hopefully lessen the friction.

On Windows, the default Python distribution now has a more traditional build configuration (using .pyd extension modules and a pythonXY.dll file). This means that PyOxidizer can consume pre-built extension modules without having to recompile them from source. If you publish a Windows binary wheel on PyPI, in many cases it will just work with PyOxidizer 0.8! (There are some notable exceptions to this, such as numpy, which is doing wonky things with the location of shared libraries in wheels - but I aim to fix this soon.)

Also on Windows, we no longer attempt to embed Python extension modules (.pyd files) and their shared library dependencies in the produced binary and load them from memory by default. This is because PyOxidizer's from-memory library loader didn't work in all cases. For example, some OpenSSL functionality used by the _ssl module in the standard library didn't work, preventing Python from establishing TLS connections. The old mode enabling you to produce a single file executable on Windows is still available. But you have to opt in to it (at the likely cost of more packaging and compatibility pain).

Starlark Configuration Overhaul

PyOxidizer 0.8 contains a ton of changes to its Starlark configuration files. There are so many changes that you may find it easier to port to PyOxidizer 0.8 by creating a new configuration file rather than attempting to port an existing one.

I apologize for this churn and recognize it will be disruptive. However, this churn needed to happen for various reasons.

Much of the old Starlark configuration semantics was rooted in the days when configuration files were static TOML files. Now that configuration files provide the power of a (Python-inspired) programming language, we are free to expose much more flexibility. But that flexibility requires refactoring things so the experience feels more native.

Many changes to Starlark were rooted in necessity. For example, the methods for invoking setup.py or pip install used to live on a Python distribution type and have been moved to a type representing executables. This is because the binary we are targeting influences how packaging actions behave. For example, if the binary only supports loading resources from memory (as opposed to standalone files), we need to know that when invoking the packaging tool so we can produce files (notably Python extension modules) compatible with the destination.

A major change to Starlark in 0.8 is around resource location handling. Before, you could define a static string denoting the resources policy for where things should be placed. And there were 10+ methods for adding different resource types (source, bytecode, extensions, package data) to different load locations (memory, filesystem). This mechanism is vastly simplified and more powerful in PyOxidizer 0.8!

In PyOxidizer 0.8, there is a single add_python_resource() method for adding a resource to a binary and the Starlark objects you add can denote where they should be added by defining attributes on those objects.

Furthermore, you can define a Starlark function that is called when resource objects are created to apply custom packaging rules using custom Starlark code defined in your PyOxidizer config file. So rather than having everyone try to abide by a few pre-canned policies for packaging resources, you can define a proper function in your config file that can be as complex as you want/need it to be! I feel this is vastly simpler and more powerful than implementing a custom DSL in static configuration files (like TOML, JSON, YAML, etc).

While the ability to implement your own arbitrarily complex packaging policies is useful, there is a new PythonPackagingPolicy Starlark type with enough flexibility to suit most needs.

Shipping oxidized_importer

During the development of PyOxidizer 0.8, I broke out the custom Rust-based Python meta-path importer used by PyOxidizer's run-time code into a standalone Python package. This sub-project is called oxidized_importer and I previously blogged about it.

PyOxidizer 0.8 ships oxidized_importer and all of its useful APIs available to Python. Read more in the official docs. The new Python APIs should make debugging issues with PyOxidizer-packaged applications vastly simpler: I found them invaluable when tracking down user-reported bugs!

Tons of New Tests and Refactored Code

PyOxidizer was my first non-toy Rust project. And the quality of the Rust code I produced in early versions of PyOxidizer clearly showed it. And when I was in the rapid-prototyping phase of PyOxidizer, I eschewed writing tests in favor of short-term progress.

PyOxidizer 0.8 pays down a ton of technical debt in the code base. Lots of Rust code has been refactored and is using somewhat reasonable practices. I'm not yet a Rust guru. But I'm at the point where I cringe when I look at some of the early code I wrote, which is a good sign. I do have to say that Rust has been a dream to work with during this transition. Despite being a low-level language, my early misuse of Rust did not result in crashes like you would see in languages like C/C++. And Rust's seemingly omniscient compiler and IDE tools facilitating refactoring have ensured that code changes aren't accompanied by subtle random bugs that would occur in dynamic programming languages. I really need to write a dedicated post espousing the virtues of Rust...

There are a ton of new tests in PyOxidizer 0.8 and I now feel somewhat confident that the main branch of PyOxidizer should be considered production-ready at any time assuming the tests pass. This will hopefully lead to more rapid releases in the future.

There are now tests for the pyembed Rust crate, which provides the run-time code for PyOxidizer-built binaries. We even have Python-based unit tests for validating the Python-exposed APIs behave as expected. These tests have been invaluable for ensuring that the run-time code works as expected. So now when someone files a bug I can easily write a test to capture it and keep the code working as intended through various refactors.

The packaging-time Rust code has also gained its fair share of tests. We now have fairly comprehensive test coverage around how resources are added/packaged. Python extension modules have proved to be highly nuanced in how they are handled. Tremendously helping testing of extension modules is that we're able to run tests for platform non-native extensions! While not yet exposed/supported by Starlark configuration files, I've taught PyOxidizer's core Rust code to be cross-compiling aware so that we can e.g. test Windows or macOS behavior from Linux. Before, I'd have to test Windows wheel handling on Windows. But after writing a wheel parser in Rust and teaching PyOxidizer to use a different Python distribution for the host architecture from the target architecture, I'm now able to write tests for platform-specific functionality that run on any platform that PyOxidizer can run on. This may eventually lead to proper cross-compiling support (at least in some configuration). Time will tell. But the foundation is definitely there!

New Rust Crates

As part of the aforementioned refactoring of PyOxidizer's Rust code, I've been extracting some useful/generic functionality built as part of developing PyOxidizer to their own Rust crates.

As part of this release, I'm publishing the initial 0.1 release of the python-packaging crate (docs). This crate provides pure Rust code for various Python packaging related functionality. This includes:

• Rust types representing Python resource types (source modules, bytecode modules, extension modules, package resources, etc).
• Scanning the filesystem for Python resource files .
• Configuring an embedded Python interpreter.
• Parsing PKG-INFO and related files.
• Parsing wheel files.
• Collecting Python resources and serializing them to a data structure.

The crate is somewhat PyOxidizer centric. But if others are interested in improving its utility, I'll happily accept pull requests!

PyOxidizer's crates footprint now includes:

• python-packed-resources
• python-packaging
• pyembed
• pyoxidizer

Major Documentation Updates

I strongly believe that software should be documented thoroughly and I strive for PyOxidizer's documentation to be useful and comprehensive.

There have been a lot of changes to PyOxidizer's documentation since the 0.7 release.

All configuration file documentation has been consolidated.

Likewise, I've attempted to consolidate a lot of the paved road documentation for how to use PyOxidizer in the Packaging User Guide section of the docs.

I'll be honest, since I have so much of PyOxidizer's workings internalized, it can be difficult for me to empathize with PyOxidizer's users. So if you have difficult with the readability of the documentation, please file an issue and report what is confusing so the documentation can be improved!

Mercurial Shipping With PyOxidizer 0.8

PyOxidizer is arguably an epic yak shave of mine to help the Mercurial version control tool transition to Python 3 and Rust.

I'm pleased to report that Mercurial is now shipping PyOxidizer-built distributions on Windows as of the 5.2.2 release a few days ago! If a complex Python application like Mercurial can be configured to work with PyOxidizer, chances are your Python application will work as well.

Whats Next

I view PyOxidizer 0.8 as a pivotal release where PyOxidizer is turning the corner from a prototyping science experiment to something more generally usable. The investments in test coverage and refactoring of the Rust internals are paving the way towards future features and bug fixes.

In upcoming releases, I'd like to close remaining known compatibility gaps with popular Python packages (such as numpy and other packages in the scientific/data space). I have a general idea of what work needs to be done and I've been laying the ground work via various refactorings to execute here.

I want a general theme of future releases to be eliminating reasons why people can't use PyOxidizer. PyOxidizer's historical origin was as a science experiment to see if single file Python applications were possible. It is clear that achieving this is fundamentally incompatible with compatibility with tons of Python packages in the wild. I'd like to find a way where PyOxidizer can achieve 99% package compatibility by default so new users don't get discouraged when using PyOxidizer. And for the subset of users who want single file executables, they can spend the magnitude of additional effort to achieve that.

At some point, I also want to make a pivot towards focusing on producing distributable artifacts (Debian/RPM packages, MSI installers, macOS DMG files, etc). I'm slightly bummed that I haven't made much progress here. But I have a vision in my mind of where I want to go (I'll be making a standalone Rust crate + Starlark dialect to facilitate producing distributable artifacts for any application) and I'm anticipating starting this work in the next few months. In the mean time, PyOxidizer 0.8 should be able to give people a directory tree that they can coerce into distributable artifacts using existing packaging tooling. That's not as turnkey as I would like it to be. But the technical problems around building a distributable Python application binary still needs some work and I view that as the most pressing need for the Python ecosystem. So I'll continue to focus there so there is a solid foundation to build upon.

In conclusion, I hope you enjoy the new release! Please report any issues or feedback in the GitHub issue tracker.