Project History

Work on PyOxidizer started in November 2018 by Gregory Szorc.

Blog Posts

Version History

0.6.0

Released February 12, 2020.

Backwards Compatibility Notes

  • The default_python_distribution() Starlark function now accepts a flavor argument denoting the distribution flavor.

  • The pyembed crate no longer includes the auto-generated default configuration file. Instead, it is consumed by the application that instantiates a Python interpreter.

  • Rust projects for the main executable now utilize and require a Cargo build script so metadata can be passed from pyembed to the project that is consuming it.

  • The pyembed crate is no longer added to created Rust projects. Instead, the generated Cargo.toml will reference a version of the pyembed crate identical to the PyOxidizer version currently running. Or if pyoxidizer is running from a Git checkout of the canonical PyOxidizer Git repository, a local filesystem path will be used.

  • The fields of EmbeddedPythonConfig and pyembed::PythonConfig have been renamed and reordered to align with Python 3.8’s config API naming. This was done for the Starlark type in version 0.5. We have made similar changes to 0.6 so naming is consistent across the various types.

Bug Fixes

  • Module names without a . are now properly recognized when scanning the filesystem for Python resources and a package allow list is used (#223). Previously, if filtering scanned resources through an explicit list of allowed packages, the top-level module/package without a dot in its full name would not be passed through the filter.

New Features

  • The PythonDistribution() Starlark function now accepts a flavor argument to denote the distribution type. This allows construction of alternate distribution types.

  • The default_python_distribution() Starlark function now accepts a flavor argument which can be set to windows_embeddable to return a distribution based on the zip file distributions published by the official CPython project.

  • The pyembed crate and generated Rust projects now have various build-mode-* feature flags to control how build artifacts are built. See Rust Projects for more.

  • The pyembed crate can now be built standalone, without being bound to a specific PyOxidizer configuration.

  • The register_target() Starlark function now accepts an optional default_build_script argument to define the default target when evaluating in Rust build script mode.

  • The pyembed crate now builds against published cpython and python3-sys crates instead of a a specific Git commit.

  • Embedded Python interpreters can now be configured to run a file specified by a filename. See the run_file argument of PythonInterpreterConfig(...)`.

Other Relevant Changes

  • Rust internals have been overhauled to use traits to represent various types, namely Python distributions. The goal is to allow different Python distribution flavors to implement different logic for building binaries.

  • The pyembed crate’s build.rs has been tweaked so it can support calling out to pyoxidizer. It also no longer has a build dependency on pyoxidizer.

0.5.1

Released January 26, 2020.

Bug Fixes

  • Fixed bad Starlark example for building black in docs.

  • Remove resources attached to packages that don’t exist. (This was a regression in 0.5.)

  • Warn on failure to annotate a package. (This was a regression in 0.5.)

  • Building embedded Python resources now emits warnings when __file__ is seen. (This was a regression in 0.5.)

  • Missing parent packages are now automatically added when constructing embedded resources. (This was a regression in 0.5.)

0.5.0

Released January 26, 2020.

General Notes

This release of PyOxidizer is significant rewrite of the previous version. The impetus for the rewrite is to transition from TOML to Starlark configuration files. The new configuration file format should allow vastly greater flexibility for building applications and will unlock a world of new possibilities.

The transition to Starlark configuration files represented a shift from static configuration to something more dynamic. This required refactoring a ton of code.

As part of refactoring code, we took the opportunity to shore up lots of the code base. PyOxidizer was the project author’s first real Rust project and a lot of bad practices (such as use of .unwrap()/panics) were prevalent. The code mostly now has proper error handling. Another new addition to the code is unit tests. While coverage still isn’t great, we now have tests performing meaningful packaging activities. So regressions should hopefully be less common going forward.

Because of the scale of the rewritten code in this release, it is expected that there are tons of bugs of regressions. This will likely be a transitional release with a more robust release to follow.

Backwards Compatibility Notes

  • Support for building distributions/installers has been temporarily dropped.

  • Support for installing license files has been temporarily dropped.

  • Python interpreter configuration setting names have been changed to reflect names from Python 3.8’s interpreter initialization API.

  • .egg-info directories are now ignored when scanning for Python resources on the filesystem (matching the behavior for .dist-info directories).

  • The pyoxidizer init sub-command has been renamed to init-rust-project.

  • The pyoxidizer app-path sub-command has been removed.

  • Support for building distributions has been removed.

  • The minimum Rust version to build has been increased from 1.31 to 1.36. This is mainly due to requirements from the starlark crate. We could potentially reduce the minimum version requirements again with minimal changes to 3rd party crates.

  • PyOxidizer configuration files are now Starlark instead of TOML files. The default file name is pyoxidizer.bzl instead of pyoxidizer.toml. All existing configuration files will need to be ported to the new format.

Bug Fixes

  • The repl run mode now properly exits with a non-zero exit code if an error occurs.

  • Compiled C extensions now properly honor the ext_package argument passed to setup(), resulting in extensions which properly have the package name in their extension name (#26).

New Features

  • A glob(include, exclude=None, strip_prefix=None) function has been added to config files to allow referencing existing files on the filesystem.

  • The in-memory MetaPathFinder now implements find_module().

  • A pyoxidizer init-config-file command has been implemented to create just a pyoxidizer.bzl configuration file.

  • A pyoxidizer python-distribution-info command has been implemented to print information about a Python distribution archive.

  • The EmbeddedPythonConfig() config function now accepts a legacy_windows_stdio argument to control the value of Py_LegacyWindowsStdioFlag (#190).

  • The EmbeddedPythonConfig() config function now accepts a legacy_windows_fs_encoding argument to control the value of Py_LegacyWindowsFSEncodingFlag.

  • The EmbeddedPythonConfig() config function now accepts an isolated argument to control the value of Py_IsolatedFlag.

  • The EmbeddedPythonConfig() config function now accepts a use_hash_seed argument to control the value of Py_HashRandomizationFlag.

  • The EmbeddedPythonConfig() config function now accepts an inspect argument to control the value of Py_InspectFlag.

  • The EmbeddedPythonConfig() config function now accepts an interactive argument to control the value of Py_InteractiveFlag.

  • The EmbeddedPythonConfig() config function now accepts a quiet argument to control the value of Py_QuietFlag.

  • The EmbeddedPythonConfig() config function now accepts a verbose argument to control the value of Py_VerboseFlag.

  • The EmbeddedPythonConfig() config function now accepts a parser_debug argument to control the value of Py_DebugFlag.

  • The EmbeddedPythonConfig() config function now accepts a bytes_warning argument to control the value of Py_BytesWarningFlag.

  • The Stdlib() packaging rule now now accepts an optional excludes list of modules to ignore. This is useful for removing unnecessary Python packages such as distutils, pip, and ensurepip.

  • The PipRequirementsFile() and PipInstallSimple() packaging rules now accept an optional extra_env dict of extra environment variables to set when invoking pip install.

  • The PipRequirementsFile() packaging rule now accepts an optional extra_args list of extra command line arguments to pass to pip install.

Other Relevant Changes

  • PyOxidizer no longer requires a forked version of the rust-cpython project (the python3-sys and cpython crates. All changes required by PyOxidizer are now present in the official project.

0.4.0

Released October 27, 2019.

Backwards Compatibility Notes

  • The setup-py-install packaging rule now has its package_path evaluated relative to the PyOxidizer config file path rather than the current working directory.

Bug Fixes

  • Windows now explicitly requires dynamic linking against msvcrt. Previously, this wasn’t explicit. And sometimes linking the final executable would result in unresolved symbol errors because the Windows Python distributions used external linkage of CRT symbols and for some reason Cargo wasn’t dynamically linking the CRT.

  • Read-only files in Python distributions are now made writable to avoid future permissions errors (#123).

  • In-memory InspectLoader.get_source() implementation no longer errors due to passing a memoryview to a function that can’t handle it (#134).

  • In-memory ResourceReader now properly handles multiple resources (#128).

New Features

  • Added an app-path command that prints the path to a packaged application. This command can be useful for tools calling PyOxidizer, as it will emit the path containing the packaged files without forcing the caller to parse command output.

  • The setup-py-install packaging rule now has an excludes option that allows ignoring specific packages or modules.

  • .py files installed into app-relative locations now have corresponding .pyc bytecode files written.

  • The setup-py-install packaging rule now has an extra_global_arguments option to allow passing additional command line arguments to the setup.py invocation.

  • Packaging rules that invoke pip or setup.py will now set a PYOXIDIZER=1 environment variable so Python code knows at packaging time whether it is running in the context of PyOxidizer.

  • The setup-py-install packaging rule now has an extra_env option to allow passing additional environment variables to setup.py invocations.

  • [[embedded_python_config]] now supports a sys_frozen flag to control setting sys.frozen = True.

  • [[embedded_python_config]] now supports a sys_meipass flag to control setting sys._MEIPASS = <exe directory>.

  • Default Python distribution upgraded to 3.7.5 (from 3.7.4). Various dependency packages also upgraded to latest versions.

All Other Relevant Changes

  • Built extension modules marked as app-relative are now embedded in the finaly binary rather than being ignored.

0.3.0

Released on August 16, 2019.

Backwards Compatibility Notes

  • The pyembed::PythonConfig struct now has an additional extra_extension_modules field.

  • The default musl Python distribution now uses LibreSSL instead of OpenSSL. This should hopefully be an invisible change.

  • Default Python distributions now use CPython 3.7.4 instead of 3.7.3.

  • Applications are now built into directories named apps/<app_name>/<target>/<build_type> rather than apps/<app_name>/<build_type>. This enables builds for multiple targets to coexist in an application’s output directory.

  • The program_name field from the [[embedded_python_config]] config section has been removed. At run-time, the current executable’s path is always used when calling Py_SetProgramName().

  • The format of embedded Python module data has changed. The pyembed crate and pyoxidizer versions must match exactly or else the pyembed crate will likely crash at run-time when parsing module data.

Bug Fixes

  • The libedit extension variant for the readline extension should now link on Linux. Before, attempting to link a binary using this extension variant would result in missing symbol errors.

  • The setup-py-install [[packaging_rule]] now performs actions to appease setuptools, thus allowing installation of packages using setuptools to (hopefully) work without issue (#70).

  • The virtualenv [[packaging_rule]] now properly finds the site-packages directory on Windows (#83).

  • The filter-include [[packaging_rule]] no longer requires both files and glob_files be defined (#88).

  • import ctypes now works on Windows (#61).

  • The in-memory module importer now implements get_resource_reader() instead of get_resource_loader(). (The CPython documentation steered us in the wrong direction - https://bugs.python.org/issue37459.)

  • The in-memory module importer now correctly populates __package__ in more cases than it did previously. Before, whether a module was a package was derived from the presence of a foo.bar module. Now, a module will be identified as a package if the file providing it is named __init__. This more closely matches the behavior of Python’s filesystem based importer. (#53)

New Features

  • The default Python distributions have been updated. Archives are generally about half the size from before. Tcl/tk is included in the Linux and macOS distributions (but PyOxidizer doesn’t yet package the Tcl files).

  • Extra extension modules can now be registered with PythonConfig instances. This can be useful for having the application embedding Python provide its own extension modules without having to go through Python build mechanisms to integrate those extension modules into the Python executable parts.

  • Built applications now have the ability to detect and use terminfo databases on the execution machine. This allows applications to interact with terminals properly. (e.g. the backspace key will now work in interactive pdb sessions). By default, applications on non-Windows platforms will look for terminfo databases at well-known locations and attempt to load them.

  • Default Python distributions now use CPython 3.7.4 instead of 3.7.3.

  • A warning is now emitted when a Python source file contains __file__. This should help trace down modules using __file__.

  • Added 32-bit Windows distribution.

  • New pyoxidizer distribution command for producing distributable artifacts of applications. Currently supports building tar archives and .msi and .exe installers using the WiX Toolset.

  • Libraries required by C extensions are now passed into the linker as library dependencies. This should allow C extensions linked against libraries to be embedded into produced executables.

  • pyoxidizer --verbose will now pass verbose to invoked pip and setup.py scripts. This can help debug what Python packaging tools are doing.

All Other Relevant Changes

  • The list of modules being added by the Python standard library is no longer printed during rule execution unless --verbose is used. The output was excessive and usually not very informative.

0.2.0

Released on June 30, 2019.

Backwards Compatibility Notes

  • Applications are now built into an apps/<appname>/(debug|release) directory instead of apps/<appname>. This allows debug and release builds to exist side-by-side.

Bug Fixes

  • Extracted .egg directories in Python package directories should now have their resources detected properly and not as Python packages with the name *.egg.

  • site-packages directories are now recognized as Python resource package roots and no longer have their contents packaged under a site-packages Python package.

New Features

  • Support for building and embedding C extensions on Windows, Linux, and macOS in many circumstances. See Native Extension Modules for support status.

  • pyoxidizer init now accepts a --pip-install option to pre-configure generated pyoxidizer.toml files with packages to install via pip. Combined with the --python-code option, it is now possible to create pyoxidizer.toml files for a ready-to-use Python application!

  • pyoxidizer now accepts a --verbose flag to make operations more verbose. Various low-level output is no longer printed by default and requires --verbose to see.

All Other Relevant Changes

  • Packaging now automatically creates empty modules for missing parent packages. This prevents a module from being packaged without its parent. This could occur with namespace packages, for example.

  • pip-install-simple rule now passes --no-binary :all: to pip.

  • Cargo packages updated to latest versions.

0.1.3

Released on June 29, 2019.

Bug Fixes

  • Fix Python refcounting bug involving call to PyImport_AddModule() when mode = module evaluation mode is used. The bug would likely lead to a segfault when destroying the Python interpreter. (#31)

  • Various functionality will no longer fail when running pyoxidizer from a Git repository that isn’t the canonical PyOxidizer repository. (#34)

New Features

  • pyoxidizer init now accepts a --python-code option to control which Python code is evaluated in the produced executable. This can be used to create applications that do not run a Python REPL by default.

  • pip-install-simple packaging rule now supports excludes for excluding resources from packaging. (#21)

  • pip-install-simple packaging rule now supports extra_args for adding parameters to the pip install command. (#42)

All Relevant Changes

  • Minimum Rust version decreased to 1.31 (the first Rust 2018 release). (#24)

  • Added CI powered by Azure Pipelines. (#45)

  • Comments in auto-generated pyoxidizer.toml have been tweaked to improve understanding. (#29)

0.1.2

Released on June 25, 2019.

Bug Fixes

  • Honor HTTP_PROXY and HTTPS_PROXY environment variables when downloading Python distributions. (#15)

  • Handle BOM when compiling Python source files to bytecode. (#13)

All Relevant Changes

  • pyoxidizer now verifies the minimum Rust version meets requirements before building.

0.1.1

Released on June 24, 2019.

Bug Fixes

  • pyoxidizer binaries built from crates should now properly refer to an appropriate commit/tag in PyOxidizer’s canonical Git repository in auto-generated Cargo.toml files. (#11)

0.1

Released on June 24, 2019. This is the initial formal release of PyOxidizer. The first pyoxidizer crate was published to crates.io.

New Features

  • Support for building standalone, single file executables embedding Python for 64-bit Windows, macOS, and Linux.

  • Support for importing Python modules from memory using zero-copy.

  • Basic Python packaging support.

  • Support for jemalloc as Python’s memory allocator.

  • pyoxidizer CLI command with basic support for managing project lifecycle.