Project History¶
Work on PyOxidizer started in November 2018 by Gregory Szorc.
Blog Posts¶
Announcing the 0.9 Release of PyOxidizer (2020-10-18)
Announcing the 0.8 Release of PyOxidizer (2020-10-12)
Using Rust to Power Python Importing with oxidized_importer (2020-05-10)
PyOxidizer 0.7 (2020-04-09)
C Extension Support in PyOxidizer (2019-06-30)
Building Standalone Python Applications with PyOxidizer (2019-06-24)
PyOxidizer Support for Windows (2019-01-06)
Faster In-Memory Python Module Importing (2018-12-28)
Distributing Standalone Python Applications (2018-12-18)
Version History¶
0.24.0¶
Released 2022-12-30
Backwards Compatibility Notes¶
The minimum Rust version has been changed from 1.61 to 1.65.0.
Bug Fixes¶
Fixed a bad regression in 0.23.0 that was causing Python symbols to not get exported from built binaries, which was causing extension modules to not load successfully. (Extension module imports would fail with
undefined symbol
errors, or similar.) (#663).
Changes¶
Default CPython distributions upgraded. CPython 3.8.15 -> 3.8.16; 3.9.15 -> 3.9.16; 3.10.8 -> 3.10.9. See https://github.com/indygreg/python-build-standalone/releases/tag/20221220 for additional changes.
Other Relevant Changes¶
Managed Rust toolchain upgraded from 1.61.0 to 1.66.0.
Package versions in autogenerated
Cargo.toml
are now defined to the patch level.SPDX license list upgraded from 1.18 to 1.19.
0.23.0¶
Released November 7, 2022.
Backwards Compatibility Notes¶
The minimum Rust version has been changed from 1.60 to 1.61 to facilitate use of features required by some Rust crates.
Bug Fixes¶
Default macOS Python distributions should no longer crash when running tkinter. This fixes a regression introduced in the 0.20 release.
The paths to Python modules when using Python 3.10 is now properly resolved. Before, buggy version string parsing caused various code to look for Python modules in a
3.1
directory instead of3.10
. (#569, #600).Changed interpreter initialization logic around
sys.meta_path
handling to be more resistant to additional entries. Ifsite
is imported during interpreter initialization, a.pth
file insite-packages
could have the side-effect of registering additional entries onsys.meta_path
. Before, this could confuse our interpreter initialization code and result inOxidizedFinder not found on sys.meta_path[0] (this should never happen)
errors. Our overly strict code is now more tolerant of unknown entries onsys.meta_path
and this error should no longer occur. (#602)
Changes¶
Default CPython distributions upgraded. CPython 3.8.13 -> 3.8.15; 3.9.13 -> 3.9.15; 3.10.4 -> 3.10.8. See https://github.com/indygreg/python-build-standalone/releases/tag/20220630, https://github.com/indygreg/python-build-standalone/releases/tag/20220802, https://github.com/indygreg/python-build-standalone/releases/tag/20221002, and https://github.com/indygreg/python-build-standalone/releases/tag/20221106 for additional changes.
PyO3 Rust crate(s) upgraded from 0.16.5 to 0.17.1.
The mechanism for passing custom arguments to the linker for exporting symbols from built executables has transitioned from using
.cargo/config
files to emittingcargo:rustc-link-arg=FLAG
lines from Cargo build scripts. There should be no change in behavior. Generated Rust projects with a.cargo/config
overridingrustflags
to pass linker arguments can safely delete these lines, as they are no longer necessary.SETUPTOOLS_USE_DISTUTILS=stdlib
is now unconditionally set when invoking Python packaging tools. Previously, it was only set when statically linking libpython. The new behavior ensures that we consistently use the distutils from the Python stdlib and not the distutils in setuptools. The reason for this change was to work around a change in behavior in modern setuptools/distutils resulting in extension module building failing due to new failures resolving the path toPython.h
.
0.22.0¶
Released June 5, 2022.
Bug Fixes¶
macOS binaries no longer dynamically link
liblzma.5.dylib
.
0.21.0¶
Released June 4, 2022.
Backwards Compatibility Notes¶
The minimum Rust version has been changed from 1.56 to 1.60 to facilitate use of features required by some Rust crates.
The default Python version is 3.10 (instead of 3.9).
Bug Fixes¶
Fixed regression in the behavior of various
pyoxidizer
commands which prevented them from working without specifying any arguments. This regression was introduced in 0.20 with the upgrade of theclap
crate to version 3.1. (#523)PyO3 Rust crates upgraded from 0.16.4 to 0.16.5. The upgrade fixes compatibility issues with Python 3.10 that could lead to runtime crashes or incorrect behavior in many configurations.
Fixed a runtime panic when incorrect attribute assignments were attempted on the
PythonExtensionModule
,PythonPackageDistributionResource
, andPythonPackageResource
Starlark types. (#561)We no longer panic when encountering invalid UTF-8 when reading process output of various
python
invocations. Previously, invocations ofpip
,setup.py
, and other processes could result in a panic if invalid UTF-8 was emitted. (#579)
New Features¶
Default CPython distributions upgraded from 3.8.12, 3.9.10, and 3.10.2 to 3.8.13, 3.9.13, and 3.10.4, respectively. See additional changes in these distributions at https://github.com/indygreg/python-build-standalone/releases/tag/20220318, https://github.com/indygreg/python-build-standalone/releases/tag/20220501, and https://github.com/indygreg/python-build-standalone/releases/tag/20220528.
The default Python version is now 3.10 (instead of 3.9).
The mechanism for handling software licenses has been overhauled.
The formatting of licenses during building has changed significantly.
Rust licensing information is now dynamically derived at build time rather than derived from a static list. The Rust components with annotated licensing should be more accurate as a result.
PythonExecutable
Starlark types now write out a file containing licensing information for software components within the binary. This restores a feature that was dropped in version 0.5. The name of the file (or disabling of the feature) can be controlled via thePythonExecutable.licenses_filename
attribute.A new
pyoxidizer rust-project-licensing
command for printing licensing information for Rust projects. This can be used to help debug Rust licensing issues or to generate licensing content for any Rust project.A
PythonExecutable.add_cargo_manifest_licensing()
Starlark method for registering the licensing information for aCargo.toml
Rust project. This can be used by Rust projects wishing to have their licensing information captured.
Initial support for
aarch64-unknown-linux-gnu
Python distributions. The distributions are now defined and PyOxidizer should use them when appropriate. However, the distributions aren’t yet well tested. So feedback on their operation via GitHub issues would be appreciated!aarch64-apple-darwin
(Apple M1) now has a default Python 3.8 distribution. This distribution is not very well tested and use of a newer distribution is strongly preferred.
Other Relevant Changes¶
Managed Rust toolchain upgraded from 1.56.1 to 1.61.0.
Starlark
PythonInterpreterConfig
documentation has been changed to refer to Python Interpreter Configuration Data Structures. The latter is automatically derived from the canonical Rust source code. So the change effectively results in a single, consistent set of documentation for interpreter configuration.The mechanism for locating the Apple SDK now uses the apple-sdk Rust crate. The new crate work similarly to how our custom logic was working before. But there may be subtle changes in behavior. If you see new build errors related to Apple SDKs in this release, don’t hesitate to create an issue. One notable change is that we will now look for SDKs in all
/Applications/Xcode*.app
directories. In environments like GitHub Actions, this will result in finding and using the newest installed SDK.The jemalloc allocator in built binaries has been upgraded to version 5.3.
The auto-generated Rust project created during binary building is now explicitly licensed to the public domain.
Derivation of a custom
libpython
library archive now sometimes uses pure Rust code instead of calling external processes. There should be no meaningful change in behavior except for build output being more concise.Auto-generated Rust projects now contain an empty
[workspace]
table in theirCargo.toml
. This enables auto-generated projects to be nested under an existing workspace directory without Cargo complaining. This approach is more robust in the common case where the Rust project isn’t part of a larger workspace.
0.20.0¶
Released March 6, 2022.
Bug Fixes¶
The
pyembed
crate will now properly callmultiprocessing.spawn.spawn_main()
when themultiprocessing
auto dispatch function as configured byPythonInterpreterConfig.multiprocessing_start_method
is set tospawn
. This resolves aTypeError: spawn_main() missing 1 required positional argument: 'pipe_handle'
run-time error that would occur in this configuration. (#483)The
pyembed
crate better handles errors during interpreter initialization. This fixes a regression to the error handling introduced by the port to PyO3 in version 0.18.0. (#481)The
pyembed::MainPythonInterpreter
type is now more resilient against calling into a finalized Python interpreter. Before, callingpy_runmain()
(possibly viarun()
) could result in a segfault in the type’sDrop
implementation.oxidized_importer.OxidizedFinder.find_distributions()
now properly normalizes names when performing comparisons. Previously, the specifiedname
was properly normalized but it was compared against un-normalized strings. Both the search and candidate names are now normalized when performing a comparison. This should fix cases where case and other special character differences could result in a distribution not being found. (#488)A potential crash when importing extension modules from memory on Windows was fixed. The crash could occur due to discrepancy in Python reference counting when multi-phase initialization was used. (#490)
Our patched
distutils
only setsPy_BUILD_CORE_BUILTIN
on Windows. This fixes errors building at least thegrpcio
package outside of Windows.When using a modified
distutils
to install Python packages, theSETUPTOOLS_USE_DISTUTILS=stdlib
environment variable is now set. This preventssetuptools
from using its vendored copy ofdistutils
and ignoring our modifications. Before this change, packages with extension modules may not have built correctly, resulting in build or run-time errors.
Backwards Compatibility Notes¶
The
pyembed::MainPythonInterpreter
Rust API for controlling embedded Python interpreters has been refactored. Various methods now take&self
instead of&mut self
.acquire_gil()
andrelease_gil()
have been removed (usewith_gil()
instead).MainPythonInterpreter
instances now release the GIL after initialization. Before, the GIL would be perpetually held by the instance. Consumers that were callingPyEval_SaveThread()
to release the GIL to work around this should delete calls to that function, as the GIL is now released automatically. APIs onMainPythonInterpreter
will acquire the GIL as necessary. (#500)
New Features¶
Support for Python 3.10 on all previously supported platforms. Python 3.9 is still the default Python version. Target Python 3.10 by passing
python_version = "3.10"
to thedefault_python_distribution()
Starlark function.Default Python distributions upgraded from 3.9.7 to 3.9.10. Various library dependencies have also been upgraded. See https://github.com/indygreg/python-build-standalone/releases/tag/20211017 and https://github.com/indygreg/python-build-standalone/releases/tag/20220222 for the full list of changes.
The
pyembed::MainPythonInterpreter
Rust struct has gained awith_gil()
function for executing a function with the Python GIL held.
Other Relevant Changes¶
PyO3 Rust crate upgraded from version
0.14
to0.16
.Managed Rust toolchain upgraded from 1.56.0 to 1.56.1.
0.19.0¶
Released October 28, 2021.
Changes¶
p12
Rust crate updated to avoid dependency on version yanked from crates.io (version 0.18.0 could not be installed viacargo
in some configurations because of this).PyOxidizer’s documentation is now more isolated from the rest of the projects in the same repository.
0.18.0¶
Released October 24, 2021.
Bug Fixes¶
The
unable to identify deployment target environment variable for macosx (please report this bug)
error message seen when attempting to use a too-old macOS SDK has been replaced to automatically assume the use ofMACOSX_DEPLOYMENT_TARGET
. A warning message that this will possibly lead to build failures is printed. (#414)Invocation of
signtool.exe
on Windows now always passes/fd SHA256
by default. Previously, we did not specify/fd
unless a signing algorithm was explicitly requested. Newer versions ofsigntool.exe
appear to insist that/fd
be specified.Default Python distributions now properly advertise system library dependencies on Linux and macOS. The older distributions failed to annotate some library dependencies, which could lead to missing symbol errors in some build environments.
Linux default Python distributions no longer utilize the
pthread_yield()
function, enabling them to be linked against glibc 2.34+, which deprecated the symbol. (#463)Python
.whl
resources parsing now ignores directories. Previously, directories may have been emitted as 0-sized resources.In some
pyoxidizer.bzl
configurations, an error would occur due to attempting to write a built executable to a directory that doesn’t exist. This should no longer occur. (#447)
Backwards Compatibility Notes¶
The minimum Rust version has been changed from 1.52 to 1.56 to facilitate use of the newest versions of some Rust crates, Rust 2021 edition, and some Cargo features to enhance linker control.
The run-time Rust code for interfacing with the Python interpreter now uses the PyO3 crate instead of cpython. The code port was quite extensive and while we believe all important run-time functionality is backwards compatible, there are possibly subtle differences in behavior. Please file GitHub issues to report any undesired changes in behavior.
Development workflows relying on specifying the
PYTHON_SYS_EXECUTABLE
environment variable have changed to usePYO3_PYTHON
, as the environment variable has changed between thecpython
andpyo3
crates.The
pyembed
crate no longer hascpython-link-unresolved-static
andcpython-link-default
Cargo features. Autogenerated Rust projects also no longer havecpython-link-unresolved-static
andcpython-link-default
features (which existed as proxies to these features in thepyembed
crate).The
pyoxidizer add
command has been removed because it didn’t work as advertised.The
pyembed
crate no longer hasbuild-mode-*
features and its build script no longer attempts to integrate with PyOxidizer or its build artifacts.The
pyembed
crate no longer annotates alinks
entry.The mechanism by which auto-generated Rust projects integrate with the
pyembed
crate has changed substantially. If you had created a standalone Rust project viapyoxidizer init-rust-project
, you may wish to create a fresh project and reconcile differences in the auto-generated files to ensure things now build as expected.Default Python distributions on macOS aarch64 are now built with macOS SDK 11.3. macOS x86_64 are now built with macOS SDK 11.1.
New Features¶
Default Python distributions upgraded from 3.8.11 and 3.9.6 to 3.8.12 and 3.9.7. Various library dependencies have also been upgraded. See https://github.com/indygreg/python-build-standalone/releases/tag/20211012 and https://github.com/indygreg/python-build-standalone/releases/tag/20211017 for the full list of changes.
When in verbose mode, messages will be printed displaying the actual result of every request to add a resource. Before, the Starlark code would emit a message like
adding extension module foo
before requesting the addition and the operation could no-op. This behavior was misleading and hard to debug since it often implied a resource was added when in fact it wasn’t! The new behavior is for the resource collector to tell its callers exactly what actions it took and for the actual results to be displayed to the end-user. This should hopefully make it easier to debug issues with how resources are added to binaries.A new
pyoxidizer generate-python-embedding-artifacts
command that writes out file artifacts useful for embedding Python in another project. The command essentially enables other projects to use PyOxidizer’s embeddable Python distributions without using PyOxidizer to build them. See Generic Python Embedding in Rust Applications for documentation.
Other Relevant Changes¶
When Apple SDKs are found via the
SDKROOT
environment variables, a hard error now occurs if that SDK does not support the target platform or deployment target. Previously, we would allow the use of the SDK, only to likely encounter a hard-to-debug compile error. If the SDKs version does not meet the desired minimum version, a warning message is printed but the build proceeds. (#431)The
pyembed
crate (which built binaries use to interface with an embedded Python interpreter) now uses thepyo3
crate instead ofcpython
to interface with Python APIs.Nightly Cargo features are no longer required on Windows. (Courtesy of PyO3 giving us complete control over how Python is linked.)
The mechanism by which built binaries link against
libpython
has been significantly refactored. Before, thecpython
crate would link against a partiallibpython
in many configurations and thepyembed
crate would complete the linking with a library defined by PyOxidizer. With the PyO3 crate supporting a configuration file to configure all attributes of linking, the PyO3 crate now fully links againstlibpython
andpyembed
doesn’t care about linkinglibpython
at all.The
pyembed
crate is now generic and no longer attempts to integrate withpyoxidizer
or its build artifacts. The crate can now be used by any Rust application wishing to embed a Python interpreter.The
oxidized_importer
Python extension has been extracted from thepyembed
crate and is now defined in thepython-oxidized-importer
crate. Thepyembed
crate now depends on this crate to provide the custom importer functionality.Previous versions of PyOxidizer would not build on Rust 1.56+ due to incompatibilities with an older version of the
starlark
crate. The crate was upgraded to version 0.3.2 to fix this issue.Managed Rust toolchain upgraded from 1.54.0 to 1.56.0.
0.17.0¶
Released August 8, 2021.
Backwards Compatibility Notes¶
The minimum Rust version has been changed from 1.46 to 1.52 to facilitate use of const generics and some other stabilized APIs.
starlark_tugger.PythonWheelBuilder.write_to_directory()
now interprets relative paths as relative to the currently configured build path, not relative to the process’s current working directory.Various Starlark types now ensure that they cannot get out of sync when cloned. Previously, various Starlark types would clone their underlying Rust struct when the Starlark value was cloned. This could cause Starlark value instances to become out of sync with each other if one value was mutated. Now, all mutable Starlark types should hold a reference to a shared resource, ensuring that cloned Starlark values all refer to the same instance. This change could result in Starlark configuration files behaving differently. For example, before you could mutate a value in a function call and that mutation wouldn’t be reflected in the caller’s Starlark value. Now, it would be.
oxidized_importer.OxidizedFinder
will now automatically callmultiprocessing.set_start_method()
when it imports themultiprocessing
module. Applications that explicitly callmultiprocessing.set_start_method()
may fail withRuntimeError("context has already been set")
as a result of this change. See Using the multiprocessing Python Module for workarounds.PythonInterpreterConfig.sys_frozen
now defaults toTrue
instead ofFalse
.starlark_tugger.WiXInstaller
,starlark_tugger.WiXMSIBuilder
, andstarlark_tugger.WiXBundleBuilder
instances now always default to building an installer for thex64
WiX architecture. Previously, the default architecture would be derived from the architecture of the running binary.starlark_tugger.WiXMSIBuilder
instances no longer have atarget_triple
attribute.
Bug Fixes¶
The default target triple is now derived from the target triple of the running binary, not the environment the running binary was built in. In many cases these would be identical. However, they would diverge if the binary was cross-compiled.
The default Python packaging policy now disables bytecode generation for various modules in the Python standard library in the
lib2to3.tests
andtest
packages that contain invalid Python 3 source code and would fail to compile to bytecode. This should enable Python resources to compile without error when settingPythonPackagingPolicy.include_test
toTrue
, without requiring a custom resource handling callback to disable bytecode generation. (#147)Applications with hyphens (
-
) in their name now build properly on Windows. Previously, there would be a cryptic build failure when runningrc.exe
. (#402)The ELF (read: Linux) binaries in the default Python distributions have changed how they perform dynamic library loading so they should always pick up the libpython from the distribution. Before,
LD_LIBRARY_PATH
environment variables could result in the wrong libpython being loaded and errors likeModuleNotFoundError: No module named '_posixsubprocess'
being encountered. (#406)
New Features¶
Default Python distributions upgraded from 3.8.10 and 3.9.5 to 3.8.11 and 3.9.6. Various library dependencies have also been upgraded. See https://github.com/indygreg/python-build-standalone/releases/tag/20210724 for the full list of changes.
oxidized_importer.OxidizedFinder
now callsmultiprocessing.set_start_method()
when themultiprocessing
module is imported. The behavior of this feature can be controlled via the newPythonInterpreterConfig.multiprocessing_start_method
attribute. On macOS, the default start method is effectively switched fromspawn
tofork
, as PyOxidizer supports this mode. The main execution routine of built executables also now recognizes the signatures of processes spawned formultiprocessing
use and will automatically function accordingly. This behavior can be disabled viaPythonInterpreterConfig.multiprocessing_auto_dispatch
. These changes mean thatmultiprocessing
should just work when default settings are used. See Using the multiprocessing Python Module for full documentation ofmultiprocessing
interactions with PyOxidizer.The
oxidized_importer.OxidizedFinder.pkg_resources_import_auto_register
now exposes whether theoxidized_importer.OxidizedFinder
instance will automatically register itself withpkg_resources
.starlark_tugger.AppleUniversalBinary
has gained thestarlark_tugger.AppleUniversalBinary.write_to_directory()
method.starlark_tugger.FileContent
has gained thestarlark_tugger.FileContent.write_to_directory()
method.starlark_tugger.MacOsApplicationBundleBuilder
has gained thestarlark_tugger.MacOsApplicationBundleBuilder.write_to_directory()
method.starlark_tugger.WiXInstaller
has gained thestarlark_tugger.WiXInstaller.to_file_content()
andstarlark_tugger.WiXInstaller.write_to_directory()
methods.starlark_tugger.WiXMSIBuilder
has gained thestarlark_tugger.WiXMSIBuilder.to_file_content()
andstarlark_tugger.WiXMSIBuilder.write_to_directory()
methods.starlark_tugger.WiXBundleBuilder
has gained thestarlark_tugger.WiXBundleBuilder.to_file_content()
andstarlark_tugger.WiXBundleBuilder.write_to_directory()
methods.starlark_tugger.WiXInstaller
has gained thestarlark_tugger.WiXInstaller.arch
attribute to retrieve and modify the architecture of the WiX installer being built.The constructors for
starlark_tugger.WiXInstaller
,starlark_tugger.WiXMSIBuilder
, andstarlark_tugger.WiXBundleBuilder
now accept anarch
argument to control the WiX architecture of the installer.starlark_tugger.WiXMSIBuilder
has gained thestarlark_tugger.WiXMSIBuilder.arch
attribute to define the architecture of the WiX installer being built.
Other Relevant Changes¶
Managed Rust toolchain upgraded from 1.52.0 to 1.54.0.
Visual C++ Redistributable installers upgraded from version 14.28.29910 to 14.29.30040.
0.16.0¶
Released May 9, 2021.
Bug Fixes¶
The Rust build environment now always sets
RUSTC
to the path to the Rust compiler that we’ve detected. This should hopefully preventcould not execute process `rustc...
errors in environments where Rust is not otherwise installed.Pre-release
pyoxidizer
binaries built in CI should now generateCargo.lock
files in Rust projects that work withcargo build --frozen
.Managed Rust toolchains now properly install the Rust stdlib for cross-compiles. Previously, the logs said it was installing them but didn’t actually, leading to build failures due to an incomplete Rust toolchain.
The file modified times in files extracted from Python distributions are now set to the current time. Previously, we preserved the mtime in the tar archive and the Windows archives had an mtime of the UNIX epoch. This could lead to runtime errors in
pip
due to pip attempting to create a zip file of itself and Python’s zip file code not supporting times older than 1980. If you see aValueError: ZIP does not support timestamps before 1980
error when runningpip
as part of running PyOxidizer, you are hitting this bug. You will need modernize the mtimes in the extracted Python distributions. The easiest way to do this is to clear PyOxidizer’s Python distribution cache viapyoxidizer cache-clear
.MSI installers built with
starlark_tugger.WiXMSIBuilder
should now properly update thePATH
environment variable if that installation option is active. This affects PyOxidizer’s own MSI installers.
New Features¶
The new
starlark_tugger.PythonWheelBuilder
type can be used to create Python wheel (.whl
) files. It is currently rather low-level and doesn’t have any integrations with other Starlark Python types. But it does allow you to create Python wheels from file content. PyOxidizer uses the type for building its own wheels (previously it was usingmaturin
).
Other Relevant Changes¶
When building for Apple platforms, we now check for a compatible Apple SDK earlier during binary building (when compiling a custom
config.c
for a customlibpython
). This should surface missing dependencies sooner in the build and potentially replace cryptic compiler error messages with an actionable one about the Apple SDK. Related to this, we now target a specific Apple SDK when compiling the aforementioned source file to ensure that the same, validated SDK is consistently used.
0.15.0¶
Released May 6, 2021.
Backwards Compatibility Notes¶
The order of the
content
andpath
arguments tostarlark_tugger.MacOsApplicationBundleBuilder.add_macos_file()
andstarlark_tugger.MacOsApplicationBundleBuilder.add_resources_file()
has been reversed andpath
now defaults toNone
. While technically a backwards incompatible change, the old methods weren’t usable in prior versions of PyOxidizer because thestarlark_tugger.FileContent
Starlark type couldn’t be instantiated!starlark_tugger.FileManifest
now performs path normalization and checking on every insertion. Before, there were a few code paths that may have skipped this step, causing bad paths to be inserted.Tracked paths in
starlark_tugger.FileManifest
should now have Windows-style directory separators (\
) normalized to UNIX style (/
).
Bug Fixes¶
Apple code signatures using a time-stamp server now validate Apple’s code signature checks. Previously, they failed validation due the time-stamped data being incorrect.
The WiX XML IDs and GUIDs in autogenerated
.wxs
files corresponding to install files were sometimes internally inconsistent or duplicated, leading to malformed.wxs
files being generated. Autogenerated.wxs
files should now hopefully be well-formed.Release artifacts should now reference the
pyembed
crate from the package registry instead of a Git URL. Previously, auto-generated Rust projects might insist thepyembed
crate was available at a Git URL. This would disagree with the auto-generatedCargo.lock
file and result in a build failure due to building withcargo build --frozen
.
New Features¶
Default Python distributions upgraded from 3.8.9 and 3.9.4 to 3.8.10 and 3.9.5.
PyOxidizer releases are now published as pre-built binary wheels to PyPI and can be installed via
pip install pyoxidizer
.Apple code signatures now include a time-stamp token issued by Apple’s time-stamp server by default. Presence of the time-stamp token in code signatures is a requirement to notarize applications.
It is now possible to add code signatures to Mach-O binaries that don’t have an existing signature. Previously, it was only possible to sign binaries that had an existing signature.
The
starlark_tugger.FileContent
Starlark type can now be constructed from filesystem paths or string content viastarlark_tugger.FileContent.__init__()
. The type also exposes mutable attributesstarlark_tugger.FileContent.executable
andstarlark_tugger.FileContent.filename
to view and change instance state.The new
starlark_tugger.FileManifest.add_file()
method can be used to add astarlark_tugger.FileContent
to astarlark_tugger.FileManifest
. The method allows controlling the destination path within thestarlark_tugger.FileManifest
. Combined with the introduction ofstarlark_tugger.FileContent.__init__()
, it is now possible to add arbitrary file-based or string-based files to astarlark_tugger.FileManifest
.The new
starlark_tugger.FileManifest.paths()
method can be used to retrieve the paths currently tracked by astarlark_tugger.FileManifest
.The new
starlark_tugger.FileManifest.get_file()
method can be used to retrieve astarlark_tugger.FileContent
from a path instarlark_tugger.FileManifest
. The newstarlark_tugger.FileManifest.remove()
method can be used to remove a tracked path from astarlark_tugger.FileManifest
. The new methods unlock the ability to mutate the contents ofstarlark_tugger.FileManifest
instances.Starlark now has a
starlark_tugger.AppleUniversalBinary
type that can be used to construct universal/fat/multi-architecture Mach-O binaries, the binary executable format used by Apple operating systems. Starlark primitives likePythonExecutable
can today only yield a single architecture binary. However, with the new type, it is possible to take multiple source binaries and combine them into a universal binary, all from Starlark.The
starlark_tugger.WiXInstaller
Starlark type now exposes mutable attributesstarlark_tugger.WiXInstaller.install_files_root_directory_id
andstarlark_tugger.WiXInstaller.install_files_wxs_path
to control the autogenerated.wxs
file containing fragment for install files. See the type’s documentation for more.
Other Relevant Changes¶
starlark_tugger.WiXInstaller.build()
now automatically materializes and builds a.wxs
file containing fragments for files registered for installation. Before, this Starlark type was not very usable without this file, as WiX wouldn’t pick up files that had been registered for install.Rust 1.52.0 is now used as the default Rust toolchain (from version 1.51.0).
The musl libc linked default Python distributions no longer use the
reallocarray()
symbol, which was introduced in musl libc 1.2.2. This should enable musl libc builds to work with musl 1.2.1 and possibly older versions.
0.14.1¶
Released April 30, 2021.
Bug Fixes¶
Fixed a bug in the 0.14.0 release where newly created projects won’t build due to
Cargo.lock
issues.
0.14.0¶
Released April 30, 2021.
Backwards Compatibility Notes¶
PyOxidizer no longer uses the system’s installed Rust toolchain when building projects. By default, it will download and use a specific version of the Rust toolchain. See Managed Rust Toolchain for instructions on disabling this behavior.
The
pyembed
crate now always canonicalizes the path to the current executable. Previously, ifOxidizedPythonInterpreterConfig.exe
were set, it would not be canonicalized. It is possible this could break use cases where the current executable is deleted after the executable starts. In this case, the Python interpreter will fail to initialize. If this functionality is important to you, file a feature request.The
pyembed
crate will now remove entries fromsys.path_hooks
related to filesystem importers if filesystem importing is disabled. Previously, onlysys.meta_path
would have its filesystem importers removed.The
pyembed
crate now always registers theoxidized_importer.OxidizedFinder
path hook onsys.path_hooks
when an instance is being installed onsys.meta_path
. This ensures that consumers ofsys.path_hooks
outside the module importing mechanism (such aspkgutil
andpkg_resources
) can use the path hook.The
pyembed
crate now registers theoxidized_importer.OxidizedFinder
path hook as the 1st entry onsys.path_hooks
, not the last.The
oxidized_importer.OxidizedFinder
path hook is now more strict about the path values it will respond to. Previously, it would acceptstr
,bytes
,pathlib.Path
, or any other path-like type. Now, it only responds tostr
values. Furthermore, it will only respond to values that exactly matchoxidized_importer.OxidizedFinder.path_hook_base_str
or a well-formed virtual sub-directory thereof. Previously, it would attempt to canonicalize path strings, taking into account the current working directory, filesystem links, and other factors affecting path normalization. The new implementation is simpler and by being stricter should be less brittle at run-time. See Paths Hooks Compatibility for documentation on the path hooks behavior.The
pyembed
crate has prefixed all its allocator features (jemalloc
,mimalloc
, andsnmalloc
) withallocator-
. This makes the names consistent with the features in auto-generated Rust projects.
Bug Fixes¶
Rust projects created with
pyoxidizer init-rust-project
no longer fail to build due to a crypticwriting packed resources
error.When materializing Python package distribution resources (i.e. files in
.dist-info
and.egg-info
directories) to the filesystem, package names are now normalized to lowercase with hyphens replaced with underscores. The new behavior matches expectations of official Python resource handling APIs likeimportlib.metadata
. Before, APIs likeimportlib.metadata
would fail to find files materialized by PyOxidizer for package names containing a hyphen or capital latter. (#394)
New Features¶
PyOxidizer now automatically downloads and uses a Rust toolchain at run time. This means there is no longer an install requirement of having Rust already available on your system (unless you install PyOxidizer from source). See Managed Rust Toolchain for details of the new feature, including directions on how to disable the feature and have PyOxidizer use an already installed Rust.
oxidized_importer.OxidizedFinder
now supportspkg_resources
integration. Most of thepkg_resources
APIs are implemented, enabling mostpkg_resources
functionality to work.pkg_resources
integration is automatically enabled upon import of thepkg_resources
module, sopkg_resources
integration should just work for many applications. See Support for pkg_resources for the full documentation, including which features aren’t implemented.oxidized_importer.OxidizedFinder
now exposes the propertiesoxidized_importer.OxidizedFinder.path_hook_base_str
andoxidized_importer.OxidizedFinder.origin
.Starlark configuration files can now produce macOS Application Bundles. See :py:class`starlark_tugger.MacOsApplicationBundleBuilder` for the API documentation.
pyoxidizer
commands that evaluate Starlark files now accept the arguments--var
and--var-env
to define extra variables to define in the evaluated Starlark file. This enables Starlark files to be parameterized based on explicit strings provided via--var
or through the content of environment variables via--var-env
.PyOxidizer can now automatically add cryptographic code signatures when running. This feature is extensively documented at Code Signing. From a high-level, you instantiate and activate a
starlark_tugger.CodeSigner
in your Starlark configuration to define your code signing certificate. As files are processed as part of evaluating your Starlark configuration file, they are examined for the ability to be signed and code signing is automatically attempted. We support signing Windows files using Microsoft’s officialsigntool.exe
application and Apple Mach-O and bundle files using a pure Rust reimplementation of Apple’s code signing functionality. This functionality is still in its early stages of development and is lacking some power user features to exert low-level control over code signing. Please file feature requests as you encounter limitations with the functionality!The new Starlark functions
starlark_tugger.prompt_confirm()
,starlark_tugger.prompt_input()
,starlark_tugger.prompt_password()
, andstarlark_tugger.can_prompt()
can be used to allow configuration files to perform interaction with the user via the terminal. The functions all allow a default value to be provided, enabling them to be used in scenarios when stdin isn’t connected to a TTY and can’t be prompted.
Other Relevant Changes¶
The Python API for the
oxidized_importer
Python extension module providing our custom importer logic is now centrally documented instead of spread out over multiple documentation pages. See API Reference for the new docs. Various type references throughout the generated documentation should now link to the new API docs.The Starlark dialect is now documented as native Python classes and functions using Sphinx’s support for doing so. The documentation should now look more familiar to Python developers familiar with Sphinx for Python API documentation.
PyOxidizer now stores persistent artifacts (like Rust toolchains) and downloaded Python distributions) in a per-user cache directory. See Cache Directory for more.
The
pyoxidizer
CLI now accepts--verbose
as a sub-command argument. Previously, it was only accepted as an argument before the sub-command name.Generated Rust projects (which can be temporary as part of building binaries) now contain a
Cargo.lock
file and are built withcargo build --locked
. The template of theCargo.lock
is static and under version control. The presence of theCargo.lock
coupled withcargo build --locked
should ensure that Rust crate versions used by Rust projects exactly match those used by the build of PyOxidizer that produced the project. This should result in more deterministic builds and higher reliability of build success.
0.13.2¶
Released April 15, 2021.
Bug Fixes¶
Fixes a build failure on Windows.
0.13.1¶
Released April 15, 2021.
Bug Fixes¶
The 0.13.0 release contained improper crate paths in
Cargo.toml
files due to a bug in our automated release mechanism. This release should fix those issues.
0.13.0¶
Released April 15, 2021.
Bug Fixes¶
WiXSimpleMsiBuilder
now properly writes XML when a license file is provided.WixBundleInstallerBuilder
now handles the already installed exit code from the VC++ Redistributable installer as a success condition. Previously, installs would abort.WixBundleInstallerBuilder
no longer errors on a missing build directory when attempting to download the Visual C++ Redistributable runtime files.
New Features¶
Per-platform Windows MSI and multi-platform Windows exe installers for PyOxidizer are now available. The installers are built with PyOxidizer, using its built-in support for producing Windows installers.
Other Relevant Changes¶
Default CPython distributions upgraded from 3.9.3 to 3.9.4.
Default Python distributions upgraded setuptools from 54.2.0 to 56.0.0.
0.12.0¶
Released April 14, 2021.
Danger
The 0.12.0 release uses CPython 3.9.3, which inadvertently shipped an ABI incompatible change, causing some extension modules to not work or crash. Please avoid this release if you use pre-built Python extension modules.
Backwards Compatibility Notes¶
The minimum Rust version has been changed from 1.45 to 1.46 to facilitate use of const fn.
On Apple platforms, PyOxidizer now validates that the Apple SDK being used is compatible with the Python distribution being used and will abort the build if not. Previously, PyOxidizer would blindly use whatever SDK was the default and this could lead to cryptic error messages when building (likely undefined symbol errors when linking). The current default Python distributions impose a requirement of the macosx10.15+ SDK for Python 3.8 and macosx11.0+ for Python 3.9. See issue #373 for a comprehensive discussion of this topic.
On Apple platforms, binaries built with PyOxidizer now automatically target the OS version that the Python distribution was built to target. Previously, binaries would likely target the OS version of the building machine unless explicit action was taken. The practical effect of this change is binaries targeting x86_64 should now work on macOS 10.9 without any end-user action required.
Documentation URLs for PyOxidizer now all consistently begin with
pyoxidizer_
. Many old documentation URLs no longer work.
Bug Fixes¶
The autogenerated
pyoxidizer.bzl
correctly references theno-copyleft
extension module filter instead of the oldno-gpl
value.Linux binaries using the
libedit
variant of thereadline
Python extension (occurs when using theno-copyleft
extension module filter) no longer encounter an undefined symbol error when linking. (#376)The ctypes extension was previously compiled incorrectly, leading to run-time errors on various platforms. These issues should be fixed.
New Features¶
On Apple platforms, PyOxidizer now automatically locates, validates, and uses an appropriate SDK given the settings of the Python distribution being used. PyOxidizer will reject building with an SDK older than the one used to produce the Python distribution. PyOxidizer will automatically use the newest installed SDK compatible with the target configuration. The SDK and targeting information is printed during builds. See Build Machine Requirements for details on how to override default behavior.
OxidizedFinder
now implementspath_hook()
and a path hook is automatically registered onsys.path_hooks
during interpreter initialization when anOxidizedFinder
is being used. Feature contributed by William Schwartz in #343.
Other Relevant Changes¶
The
snmalloc
allocator now uses the C API directly and avoids going through an allocation tracking layer, improving the performance of this allocator. Improvement contributed by Ryan Clanton.Python distributions updated to latest versions. Changes include: macOS Python 3.8 is now built against the 10.15 SDK instead of 11.1; musl libc upgraded to 1.2.2; setuptools upgraded to 54.2.0; LibreSSL upgraded to 3.2.5; OpenSSL upgraded to 1.1.1k; SQLite upgraded to 3.35.4.
0.11.0¶
Released March 4, 2021.
Backwards Compatibility Notes¶
The default Python distribution is now CPython 3.9 instead of 3.8. To use 3.8, pass the
python_version="3.8"
argument todefault_python_distribution()
in your configuration file. We don’t anticipate dropping support for 3.8 any time soon. However, this may be necessary in order to more easily support new Python features.The Python 3.8 distributions no longer support Windows 7 and require Windows 8, Windows 2012, or newer. The Python 3.9 distributions already required these Windows versions.
The minimum Rust version has been changed from 1.41 to 1.45 to facilitate the use of procedural macros.
The
pyembed::MainPythonInterpreter::run_as_main()
method has been renamed topy_runmain()
to reflect that it always callsPy_RunMain()
.The
py-module-names
file is no longer written as part of the files comprising an embedded Python interpreter.OxidizedFinder.__init__()
no longer acceptsresources_data
andresources_file
argument to specify the resources to load. Instead, call one of the newindex_*
methods on constructed instances.OxidizedFinder.__init__()
no longer automatically indexes builtin extension modules and frozen modules. Instead, you must now call one of theindex_*
methods to index these resources.The
pyembed::OxidizedPythonInterpreterConfig.packed_resources
field is now aVec<pyembed::PackedResourcesSource>
instead ofVec<&[u8]>
. The new enum allows specifying files as alternative resources sources.The
no-gpl
value ofPythonPackagingPolicy.extension_module_filter
has been changed tono-copyleft
and it operates on the SPDX license annotations instead of a list we maintained.show_alloc_count
has been removed from types representing Python interpreter configuration because support for this feature was removed in Python 3.9.pyembed::MainPythonInterpreter.acquire_gil()
’s signature has changed, now returning aPython
value directly without wrapping it in aResult
.pyembed::OxidizedPythonInterpreterConfig
had its memory allocator fields refactored to support new features and to help prevent bad configs (like defining multiple custom memory allocators).The Starlark
PythonInterpreterConfig.raw_allocator
field has been renamed toallocator_backend
. Thesystem
value has been renamed todefault
.The
pyembed
crate now canonicalizes the current executable’s path and uses this canonicalized path when resolving values with$ORIGIN
in them. Previously, the path passed into the program was used without resolving symlinks, etc. If that path were a symlink or hardlink, unexpected results could ensue.OxidizedFinder.find_distributions()
now returns an iterator ofOxidizedDistribution
instead of alist
. Code in the standard library of older versions of CPython expected an iterator to be returned and the new behavior is more compatible. This change enablesimportlib.metadata.metadata()
to work withOxidizedFinder
.
Bug Fixes¶
Escaping of string and path values when emitting Rust code for the embedded Python interpreter configuration should now be more robust. Previously, special characters (like
\
) were not escaped properly. (#321)The
load()
Starlark function should now work. (#328)pyembed::OxidizedPythonInterpreterConfig.argv
is now always used when set, even ifself.interpreter_config.argv
is also set.OxidizedFinder
now normalizes trailing.__init__
in module names to be equivalent to the parent package to partially emulate CPython’s behavior. See Support for __init__ in Module Names for more. (#317)The lifetime of
pyembed::MainPythonInterpreter.acquire_gil()
’s return value has been adjusted so the Rust compiler will refuse to compile code that could crash due to attempting to use a finalized interpreter. (#345)pyembed::MainPythonInterpreter.py_runmain()
’s signature has changed, now consuming ownership of the receiver. Subsequent borrows ofself
now fail to compile rather than causing runtime errors.The optional
rust
memory allocator is now thread-safe. Previously, it wasn’t and releasing of the GIL could lead to memory corruption and crashes.OxidizedResourceCollector.oxidize()
should now properly clean up the temporary directory it uses during execution. Before, premature Python interpreter termination (such as during failing tests) could cause the temporary directory to not be removed. Closes #346. Fix contributed by William Schwartz in #347.OxidizedFinder.find_distributions()
now properly handles the default/emptyContext
instance (specifically instances where.name = None
). Previously,name = None
would filter as if.name = "None"
. This means that all distributions should now be returned with the default/emptyContext
instance.OxidizedFinder.find_distributions()
now properly filters when the passedContext
’sname
attribute is set to a string. Previously, thename
andpath
attributes had their order swapped in a function call, leading to incorrect filtering.The Windows
standalone_static
distributions should now work again. They had been broken for a few releases and likely never worked with Python 3.9. Test coverage of this build configuration has been added to help prevent future regressions. (#360)
New Features¶
Support added for
aarch64-apple-darwin
(Apple M1 machines). Only Python 3.9 is supported on this architecture. Because we do not have CI coverage for this architecture (due to GitHub Actions not yet having M1 machines), support is considered beta quality at this time.The
FileManifest
Starlark type now exposes anadd_path()
to add a single file to the manifest.The
PythonExecutable
Starlark type now exposes ato_file_manifest()
to convert the instance to aFileManifest
.The
PythonExecutable
Starlark type now exposes ato_wix_msi_builder()
method to obtain aWiXMSIBuilder
, which can be used to generate an MSI installer for the application.The
PythonExecutable
Starlark type now exposes ato_wix_bundle_builder()
method to obtain aWiXBundleBuilder
, which can be used to generate an.exe
installer for the application.The
pyembed
crate andOxidizedFinder
importer now support indexing multiple resources sources. You can have multiple in-memory data blobs, multiple file-based resources, or a mix of all of the above.The
OxidizedFinder
Python type now exposed variousindex_*
methods to facilitate loading/indexing of resource data in arbitrary byte buffers or files. You can call these methods multiple times to chain multiple resources blobs together.The
PythonExecutable
Starlark type now exposes apacked_resources_load_mode
attribute allowing control over where packed resources data is written and how it is loaded at run-time. This attribute facilitates disabling the embedding of packed resources data completely (enabling you to produce an executable that behaves very similarly topython
) and allows writing and loading resources data to a standalone file installed next to the binary (enabling multiple binaries to share the same resources file). See Managing Packed Resources Data for more on this feature.PyOxidizer now scans for licenses of Python packages processed during building and prints a report about what it finds when writing build artifacts. This feature is best effort and relies on packages properly advertising their license metadata.
Support for configuring Python’s memory allocators has been expanded. The Starlark
PythonInterpreterConfig.allocator_debug
field has been added and allows enabling Python memory allocator debug hooks. The StarlarkPythonInterpreterConfig.allocator_mem
,PythonInterpreterConfig.allocator_obj
, andPythonInterpreterConfig.allocator_pymalloc_arena
fields have been added to control whether to install a custom allocator for the mem and obj domains as well aspymalloc
’s arena allocator.The mimalloc and snmalloc memory allocators can now be used as Python’s memory allocators. See documentation for
PythonInterpreterConfig.allocator_backend
. Code contributed by Ryan Clanton in #358.The mimalloc and snmalloc memory allocators will now automatically be used as Rust’s global allocator when configured to be used by Python.
The
@classmethod
OxidizedDistribution.find_name()
andOxidizedDistribution.discover()
are now implemented, filling in a feature gap inimportlib.metadata
functionality.There is a new
PythonExecutable.windows_runtime_dlls_mode
attribute to control how required Windows runtime DLL files should be materialized during application building. By default, if a built binary requires the Visual C++ Redistributable Runtime (e.g.vcruntime140.dll
), PyOxidizer will attempt to locate and copy those files next to the built binary. See Managing the Visual C++ Redistributable Requirement for more.Documentation around portability of binaries produced with PyOxidizer has been reorganized and overhauled. See Portability of Binaries Built with PyOxidizer for the new documentation.
Other Relevant Changes¶
Python distributions upgraded to CPython 3.8.8 and 3.9.2 (from 3.8.6 and 3.9.0). See https://github.com/indygreg/python-build-standalone/releases/tag/20210103 and https://github.com/indygreg/python-build-standalone/releases/tag/20210227 for a full list of changes in these distributions.
CI has been moved from Azure Pipelines to GitHub Actions.
Low level code in the
pyembed
crate for loading and indexing resources has been significantly refactored. This code has historically been a bit brittle, as it needs to do unsafe things. We think the new code is much more robust. But there’s a chance that crashes could occur.When using the
no-copyleft
(formerlyno-gpl
) extension module filter, some system library dependencies are now allowed, enabling various extension modules to be present in this mode.The
pyembed
andoxidized-importer
crates had their SPDX license expression changed fromPython-2.0 AND MPL-2.0
toPython-2.0 OR MPL-2.0
. The author misunderstood whatAND
did and after realizing his mistake, corrected it toOR
so the crates can one license or the other.When using dynamically linked Python distributions on Windows, the
python3.dll
file is automatically installed if it is present. (#336)libclang_rt.osx.a
is now linked into Python binaries on macOS. This was necessary to avoid undefined symbols errors from symbols which Python 3.9.1+ relies on.The
oxidized_importer
Python module now exports theOxidizedDistribution
symbol, which is the customimportlib.metadata
distribution type used byOxidizedFinder
.When building with Windows
standalone_static
distributions,pyoxidizer
now setsRUSTFLAGS=-C target-feature=+crt-static -C link-args=/FORCE:MULTIPLE
to force static CRT linkage and ignore duplicate symbol errors. Previously, the Python distribution would be using static CRT linkage and the Rust application would use dynamic/DLL CRT linkage. Furthermore, manystandalone_static
distributions have build configurations that lead to duplicate symbols and this would lead to a linker error. Suppressing the duplicate symbol error is not ideal, but it restores building withstandalone_static
until a more appropriate workaround can be devised.
0.10.3¶
Released November 10, 2020.
Bug Fixes¶
The
run_as_main()
function on embedded Python interpreters now always callsPy_RunMain()
. This fixes a regression in previous 0.10 releases that prevented a REPL from running when no explicitrun_*
attribute was set on the Python interpreter configuration.
0.10.2¶
Released November 10, 2020.
Bug Fixes¶
Fixes a version mismatch between the
pyoxidizer
andpyembed
crates that could cause builds to fail.
0.10.1¶
Released November 9, 2020.
Danger
The 0.10.1 release has a serious bug where the version of the pyembed
crate needed to build binaries may not be correct, preventing the build from
working. Please use a newer release.
Bug Fixes¶
0.10.0¶
Released November 8, 2020.
Danger
The 0.10.0 release has a serious Starlark bug preventing PyOxidizer from working correctly in many scenarios. Please use a newer release.
Backwards Compatibility Notes¶
A lot of unused Rust functions for running Python code have been removed from the
pyembed
crate. The deleted code has not been used since thePyConfig
data structure was adopted for running code during interpreter initialization. The deleted code was reimplementing functionality in CPython and much of it was of questionable quality.The built-in Python distributions have been updated to use version
6
of the standalone distribution format. PyOxidizer only recognizes version6
distributions.The
pyembed::OxidizedPythonInterpreterConfig
Rust struct now contains atcl_library
field to control the value of the TCL_LIBRARY environment variable.The
pyembed::OxidizedPythonInterpreterConfig
Rust struct no longer has arun_mode
field.The
PythoninterpreterConfig
Starlark type no longer has arun_mode
attribute. To define what code to run at interpreter startup, populate arun_*
attribute or leave allNone
with.parse_argv = True
(the default forprofile = "python"
) to start a REPL.Minimum Rust version changed from 1.40 to 1.41 to facilitate using a new crate which requires 1.41.
The default Cargo features of the
pyembed
crate now use the default Python interpreter detection and linking configuration as determined by thecpython
crate. This enables thecargo build
orcargo test
to just work without having to explicitly specify features.The
python-distributions-extract
command now receives the path to an existing distribution archive via the--archive-path
argument instead of an unnamed argument.
Bug Fixes¶
Fixed a broken documentation example for
glob()
. (#300)Fixed a bug where generated Rust code for Option<PathBuf> interpreter configuration fields was not being generated correctly.
Fixed serialization of string config options to Rust code that was preventing the following attributes of the
PythonInterpreterConfig
Starlark type from working:filesystem_encoding
,filesystem_errors
,python_path_env
,run_command
,run_module
,stdio_encoding
,stdio_errors
,warn_options
, andx_options
. (#309)
New Features¶
The
PythonExecutable
Starlark type now exposes awindows_subsystem
attribute to control the value of Rust’s#![windows_subsystem = "..."]
attribute. Setting this towindows
prevents Windows executables from opening a console window when run. (#216)The
PythonExecutable
Starlark type now exposes atcl_files_path
attribute to define a directory to install tcl/tk support files into. Setting this attribute enables the use of thetkinter
Python module with compatible Python distributions. (#25)The
python-distribution-extract
CLI command now accepts a--download-default
flag to download the default distribution for the current platform.
Other Relevant Changes¶
The Starlark types with special build or run behavior are now explicitly documented.
The list of glibc and GCC versions used by popular Linux distributions has been updated.
The built-in Linux and macOS Python distributions are now compiled with LLVM/Clang 11 (as opposed to 10).
The built-in Python distributions now use pip 20.2.4 and setuptools 50.3.2.
The Starlark primitives for defining build system targets have been extracted into a new
starlark-dialect-build-targets
crate.The code for resolving how to reference PyOxidizer’s Git repository has been rewritten. The resolution is now performed at build time in the pyoxidizer crate’s
build.rs
. There now exist environment variables that can be specified at crate build time that influence how PyOxidizer constructs these references.
0.9.0¶
Released October 18, 2020.
Backwards Compatibility Notes¶
The
pyembed::OxidizedPythonInterpreterConfig
Rust struct now contains anargv
field that can be used to control the population ofsys.argv
.The
pyembed::OxidizedPythonInterpreterConfig
Rust struct now contains aset_missing_path_configuration
field that can be used to control the automatic run-time population of missing path configuration fields.The
configure_locale
interpreter configuration setting is enabled by default. (#294)The
pyembed::OxidizedPythonInterpreterConfig
Rust struct now contains anexe
field holding the path of the currently running executable.At run-time, the
program_name
andhome
fields of the embedded Python interpreter’s path configuration are now always set to the currently running executable and its directory, respectively, unless explicit values have been provided.The packed resource data version has changed from 2 to 3 in order to support storing arbitrary file data. Support for reading and writing version 2 has been removed. Packed resources blobs will need to be regenerated in order to be compatible with new versions of PyOxidizer.
The
pyembed::OxidizedPythonInterpreterConfig
Rust struct had itspacked_resources
field changed fromOption<&'a [u8]>
toVec<&'a [u8]>
so multiple resource inputs can be specified.The
PythonDistribution
Starlark type no longer hasextension_modules()
,package_resources()
andsource_modules()
methods. UsePythonDistribution.python_resources()
instead.
New Features¶
A
print(*args)
function is now exposed to Starlark. This function is documented as a Starlark built-in but isn’t provided by the Rust Starlark implementation by default. So we’ve implemented it ourselves. (#292)The new
pyoxidizer find-resources
command can be used to invoke PyOxidizer’s code for scanning files for resources. This command can be used to debug and triage bugs related to PyOxidizer’s custom code for finding and handling resources.Executables built on Windows now embed an application manifest that enables long paths support. (#197)
The Starlark
PythonPackagingPolicy
type now exposes anallow_files
attribute controlling whether files can be added as resources.The Starlark
PythonPackagingPolicy
type now exposesfile_scanner_classify_files
andfile_scanner_emit_files
attributes controlling whether file scanning attempts to classify files and whether generic file instances are emitted, respectively.The Starlark
PythonPackagingPolicy
type now exposesinclude_classified_resources
andinclude_file_resources
attributes to control whether certain classes of resources have theiradd_include
attribute set by default.The Starlark
PythonPackagingPolicy
type now has aset_resources_handling_mode()
method to quickly apply a mode for resource handling.The Starlark
PythonDistribution
type now has apython_resources()
method for obtaining all Python resources associated with the distribution.Starlark
File
instances can now be added to resource collections viaPythonExecutable.add_python_resource()
andPythonExecutable.add_python_resources()
.
Bug Fixes¶
Fix some documentation references to outdated Starlark configuration syntax (#291).
Emit only the
PythonExtensionModule
built with our patched distutils instead of emitting 2PythonExtensionModule
for the same named module. This should result in compiled Python extension modules being usable as built-in extensions instead of being recognized as only shared libraries.Fix typo preventing the Starlark method
PythonExecutable.read_virtualenv()
from being defined. (#297)The default value of the Starlark
PythonInterpreterConfig.configure_locale
field isTrue
instead ofNone
(effectivelyFalse
since the default.profile
value isisolated
). This results in Python’s encodings being more reasonable by default, which helps ensure non-ASCII arguments are interpreted properly. (#294)Properly serialize
module_search_paths
to Rust code. Before, attempting to setPythonInterpreterConfig.module_search_paths
in Starlark would result in malformed Rust code being generated. (#298)
Other Relevant Changes¶
The
pyembed
Rust crate now callsPyConfig_SetBytesArgv
orPyConfig_SetArgv()
to initialize argv instead ofPySys_SetObject()
. The encoding of string values should also behave more similarly to whatpython
does.The
pyembed
tests exercising Python interpreters now run in separate processes. Before, Rust would instantiate multiple interpreters in the same process. However, CPython uses global variables and APIs (likesetlocale()
) that also make use of globals and process reuse resulted in tests not having pristine execution environments. All tests now run in isolated processes and should be much more resilient.When PyOxidizer invokes a subprocess and logs its output, stderr is now redirected to stdout and logged as a unified stream. Previously, stdout was logged and stderr went to the parent process stderr.
There now exists documentation on how to create an executable that behaves like
python
.The documentation on binary portability has been overhauled to go in much greater detail.
The list of standard library test packages is now obtained from the Python distribution metadata instead of a hardcoded list in PyOxidizer’s source code.
0.8.0¶
Released October 12, 2020.
Backwards Compatibility Notes¶
The default Python distributions have been upgraded to CPython 3.8.6 (from 3.7.7) and support for Python 3.7 has been removed.
On Windows, the
default_python_distribution()
Starlark function now defaults to returning astandalone_dynamic
distribution variant, meaning that it picks a distribution that can load standalone.pyd
Python extension modules by default.The standalone Python distributions are now validated to be at least version 5 of the distribution format. If you are using the default Python distributions, this change should not affect you.
Support for packaging the official Windows embeddable Python distributions has been removed. This support was experimental. The official Windows embeddable distributions are missing critical support files that make them difficult to integrate with PyOxidizer.
The
pyembed
crate now defines a newOxidizedPythonInterpreterConfig
type to configure Python interpreters. The legacyPythonConfig
type has been removed.Various
run_*
functions onpyembed::MainPythonInterpreter
have been moved to standalone functions in thepyembed
crate. Therun_as_main()
function (which is called by the default Rust program that is generated) will always callPy_RunMain()
and finalize the interpreter. See the extensive crate docs for move.Python resources data in the
pyembed
crate is no longer annotated with the'static
lifetime. Instances ofPythonConfig
andOxidizedPythonInterpreterConfig
must now be annotated with a lifetime for the resources data they hold such that Rust lifetimes can be enforced.The type of the custom Python importer has been renamed from
PyOxidizerFinder
toOxidizedFinder
.The name of the module providing our custom importer has been renamed from
_pyoxidizer_importer
tooxidized_importer
.Minimum Rust version changed from 1.36 to 1.40 to allow for upgrading various dependencies to modern versions.
Windows static extension building is possibly broken due to changes to
distutils
. However, since we changed the default configuration to not use this build mode, we’ve deemed this potential regression acceptable for the 0.8 release. If it exists, it will hopefully be fixed in the 0.9 release.The
pip_install()
,read_package_root()
,read_virtualenv()
andsetup_py_install()
methods of thePythonDistribution
Starlark type have been moved to thePythonExecutable
type. Existing Starlark config files will need to change references accordingly (often by replacingdist.
withexe.
).The
PythonDistribution.extension_modules()
Starlark function no longer accepts argumentsfilter
andpreferred_variants
. The function now returns every extension in the distribution. The reasons for this change were to make code simpler and the justification for removing it was rather weak. Please file an issue if this feature loss affects you.The
PythonInterpreterConfig
Starlark type now internally has most of its fields defined toNone
by default instead of their default values.The following Starlark methods have been renamed:
PythonExecutable.add_module_source()
->PythonExecutable.add_python_module_source()
;PythonExecutable.add_module_bytecode()
->PythonExecutable.add_python_module_bytecode()
;PythonExecutable.add_package_resource()
->PythonExecutable.add_python_package_resource()
;PythonExecutable.add_package_distribution_resource()
->PythonExecutable.add_python_package_distribution_resource()
;PythonExecutable.add_extension_module()
->PythonExecutable.add_python_extension_module()
.The location-specific Starlark methods for adding Python resources have been removed. The functionality can be duplicated by modifying the
add_location
andadd_location_fallback
attributes on Python resource types. The following methods were removed:PythonExecutable.add_in_memory_module_source()
;PythonExecutable.add_filesystem_relative_module_source()
,PythonExecutable.add_in_memory_module_bytecode()
;PythonExecutable.add_filesystem_relative_module_bytecode()
;PythonExecutable.add_in_memory_package_resource()
;PythonExecutable.add_filesystem_relative_package_resource()
;PythonExecutable.add_in_memory_package_distribution_resource()
PythonExecutable.add_filesystem_relative_package_distribution_resource()
;PythonExecutable.add_in_memory_extension_module()
;PythonExecutable.add_filesystem_relative_extension_module()
;PythonExecutable.add_in_memory_python_resource()
;PythonExecutable.add_filesystem_relative_python_resource()
;PythonExecutable.add_in_memory_python_resources()
;PythonExecutable.add_filesystem_relative_python_resources()
.The Starlark
PythonDistribution.to_python_executable()
method no longer accepts the argumentsextension_module_filter
,preferred_extension_module_variants
,include_sources
,include_resources
, andinclude_test
. All of this functionality has been replaced by the optionalpackaging_policy
, which accepts aPythonPackagingPolicy
instance. The new type represents all settings influencing executable building and control over resources added to the executable.The Starlark type
PythonBytecodeModule
has been removed. Previously, this type was internally a request to convert Python module source into bytecode. The introduction ofPythonPackagingPolicy
and underlying abilities to derive bytecode from a Python source module instance when adding that resource type rendered this Starlark type redundant. There may still be the need for a Starlark type to represent actual Python module bytecode (not derived from source code at build/packaging time). However, this functionality did not exist before so the loss of this type is not a loss in functionality.The Starlark methods
PythonExecutable.add_python_resource()
andPythonExecutable.add_python_resources()
no longer accept the argumentsadd_source_module
,add_bytecode_module
, andoptimize_level
. Instead, set variousadd_*
attributes on resource instances being passed into the methods to influence what happens when they are added.The Starlark methods
PythonExecutable.add_python_module_source()
,PythonExecutable.add_python_module_bytecode()
,PythonExecutable.add_python_package_resource()
,PythonExecutable.add_python_package_distribution_resource()
, andPythonExecutable.add_python_extension_module()
have been removed. The remainingPythonExecutable.add_python_resource()
andPythonExecutable.add_python_resources()
methods are capable of handling all resource types and should be used. Previous functionality available via argument passing on these methods can be accomplished by settingadd_*
attributes on individual Python resource objects.The Starlark type
PythonSourceModule
has been renamed toPythonModuleSource
.Serialized Python resources no longer rely on the
flavor
field to influence how they are loaded at run-time. Instead, the newis_*
fields expressing individual type affinity are used. Theflavor
attributes from theOxidizedResource
Python type has been removed since it does nothing.The packed resources data format version has been changed from 1 to 2. The parser has dropped support for reading version 1 files. Packed resources blobs will need to be written and read by the same version of the Rust crate to be compatible.
The autogenerated Rust file containing the Python interpreter configuration now emits a
pyembed::OxidizedPythonInterpreterConfig
instance instead ofpyembed::PythonConfig
. The new type is more powerful and what is actually used to initialize an embedded Python interpreter.The concept of a resources policy in Starlark has now largely been replaced by attributes denoting valid locations for resources.
oxidized_importer.OxidizedResourceCollector.__init__()
nowaccepts an
allowed_locations
argument instead ofpolicy
.
The
PythonInterpreterConfig()
constructor has been removed. Instances of this Starlark type are now created viaPythonDistribution.make_python_interpreter_config()
. In addition, instances are mutated by setting attributes rather than passing perhaps dozens of arguments to a constructor function.The default build configuration for Windows no longer forces extension modules to be loaded from memory and materializes some extension modules as standalone files. This was done because some some extension modules weren’t working when loaded from memory and the configuration caused lots of problems in the wild. The new default should be much more user friendly. To use the old settings, construct a custom
PythonPackagingPolicy
and setallow_in_memory_shared_library_loading = True
andresources_location_fallback = None
.
New Features¶
Python distributions upgraded to CPython 3.8.6.
CPython 3.9 distributions are now supported by passing
python_version="3.9"
to thedefault_python_distribution()
Starlark function. CPython 3.8 is the default distribution version.Embedded Python interpreters are now managed via the new apis defined by PEP-587. This gives us much more control over the configuration of interpreters.
A
FileManifest
Starlark instance will now have its defaultpyoxidizer run
executable set to the last added Python executable. Previously, it would only have a run target if there was a single executable file in theFileManifest
. If there were multiple executables or executable files (such as Python extension modules) a run target would not be available andpyoxidizer run
would do nothing.Default Python distributions upgraded to version 5 of the standalone distribution format. This new format advertises much more metadata about the distribution, enabling PyOxidizer to take fewer guesses about how the distribution works and will help enable more features over time.
The
pyembed
crate now exposes a newOxidizedPythonInterpreterConfig
type (and associated types) allowing configuration of every field supported by Python’s interpreter configuration API.Resources data loaded by the
pyembed
crate can now have a non-'static
lifetime. This means that resources data can be more dynamically obtained (e.g. by reading a file). PyOxidizer does not yet support such mechanisms, however.OxidizedFinder
instances can now beconstructed from Python code
. This means that a Python application can instantiate and install its own oxidized module importer.The resources indexed by
OxidizedFinder
instances are now representable to Python code asOxidizedResource
instances. These types can be created, queried, and mutated by Python code. See OxidizedResource for the API.OxidizedFinder
instances can now have customOxidizedResource
instances registered against them. This means Python code can collect its own Python modules and register them with the importer. Seeoxidized_importer.OxidizedFinder.add_resource()
for more.OxidizedFinder
instances can now serialize indexed resources out to abytes
. The serialized data can be loaded into a separateOxidizedFinder
instance, perhaps in a different process. This facility enables the creation and reuse of packed resources data structures without having to usepyoxidizer
to collect Python resources data.The types returned by
OxidizedFinder.find_distributions()
now implemententry_points
, allowing entry points to be discovered.The types returned by
OxidizedFinder.find_distributions()
now implementrequires
, allowing package requirements to be discovered.OxidizedFinder
is now able to load Python modules when only source code is provided. Previously, it required that bytecode be available.OxidizedFinder
now implementsiter_modules()
. This enablespkgutil.iter_modules()
to return modules serviced byOxidizedFinder
.The
PythonModuleSource
Starlark type now exposes module source code via thesource
attribute.The
PythonExecutable
Starlark type now has amake_python_module_source()
method to allow construction ofPythonModuleSource
instances.The
PythonModuleSource
Starlark type now has attributesadd_include
,add_location
,add_location_fallback
,add_source
,add_bytecode_optimization_level_zero
,add_bytecode_optimization_level_one
, andadd_bytecode_optimization_level_two
to influence what happens when instances are added to to binaries.The Starlark methods for adding Python resources now accept an optional
location
argument for controlling the load location of the resource. This functionality replaces the prior functionality provided by location-specific APIs such asPythonExecutable.add_in_memory_python_resource()
. The following methods gained this argument:PythonExecutable.add_python_module_source()
;PythonExecutable.add_python_module_bytecode()
;PythonExecutable.add_python_package_resource()
;PythonExecutable.add_python_package_distribution_resource()
;PythonExecutable.add_python_extension_module()
;PythonExecutable.add_python_resource()
;PythonExecutable.add_python_resources()
.Starlark now has a
PythonPackagingPolicy
type to represent the collection of settings influencing how Python resources are packaged into binaries.The
PythonDistribution
Starlark type has gained amake_packaging_policy()
method for obtaining the defaultPythonPackagingPolicy
for that distribution.The
PythonPackagingPolicy.register_resource_callback()
method can be used to register a Starlark function that will be called whenever resources are created. The callback allows a single function to inspect and manipulate resources as they are created.Starlark types representing Python resources now expose an
is_stdlib
attribute denoting whether they came from the Python distribution.The new
PythonExecutable.pip_download()
method will runpip download
to obtain Python wheels for the requested package(s). Those wheels will then be parsed for Python resources, which can be added to the executable.The Starlark function
default_python_distribution()
now accepts apython_version
argument to control the X.Y version of Python to use.The
PythonPackagingPolicy
Starlark type now exposes a flag to control whether shared libraries can be loaded from memory.The
PythonDistribution
Starlark type now has amake_python_interpreter_config()
method to obtain instances ofPythonInterpreterConfig
that are appropriate for that distribution.PythonInterpreterConfig
Starlark types now expose attributes to query and mutate state. Nearly every setting exposed by Python’s initialization API can be set.
Bug Fixes¶
Fixed potential process crash due to illegal memory access when loading Python bytecode modules from the filesystem.
Detection of Python bytecode files based on registered suffixes and cache tags is now more robust. Before, it was possible for modules to get picked up having the cache tag (e.g.
cpython-38
) in the module name.In the custom Python importer,
read_text()
of distributions returned fromfind_distributions()
now returnsNone
on unknown file instead of raisingIOError
. This matches the behavior ofimportlib.metadata
.The
pyembed
Rust project build script now reruns when the source Starlark file changes.Some Python resource types were improperly installed in the wrong relative directory. The buggy behavior has been fixed.
Python extension modules and their shared library dependencies loaded from the filesystem should no longer have the library file suffix stripped when materialized on the filesystem.
On Windows, the
sqlite
module can now be imported. Before, the system for serializing resources thought thatsqlite
was a shared library and not a Python module.The build script of the pyoxidizer crate now uses the
git2
crate to try to resolve the Git commit instead of relying on thegit
command. This should result in fewer cases where the commit was being identified asunknown
.$ORIGIN
is properly expanded insys.path
. (This was a regression during the development of version 0.8 and is not a regression from the 0.7 release.)
Other Relevant Changes¶
The registration of the custom Python importer during interpreter initialization no longer relies on running custom frozen bytecode for the
importlib._bootstrap_external
Python module. This simplifies packaging and interpreter configuration a bit.Packaging documentation now gives more examples on how to use available Starlark packaging methods.
The modified
distutils
files used when building statically linked extensions have been upgraded to those based on Python 3.8.3.The default
pyoxidizer.bzl
now has comments for thepackaging_policy
argument toPythonDistribution.to_python_executable()
.The default
pyoxidizer.bzl
now usesadd_python_resources()
instead ofadd_in_memory_python_resources()
.The Rust Starlark crate was upgraded from version 0.2 to 0.3. There were numerous changes as part of this upgrade. While we think behavior should be mostly backwards compatible, there may be some slight changes in behavior. Please file issues if any odd behavior or regressions are observed.
The configuration documentation was reorganized. The unified document for the complete API document (which was the largest single document) has been split into multiple documents.
The serialized data structure for representing Python resources metadata and its data now allows resources to identify as multiple types. For example, a single resource can contain both Python module source/bytecode and a shared library.
pyoxidizer --version
now prints verbose information about where PyOxidizer was installed, what Git commit was used, and how thepyembed
crate will be referenced. This should make it easier to help debug installation issues.The autogenerated/default Starlark configuration file now uses the
install
target as the default build/run target. This allows extra files required by generated binaries to be available and for built binaries to be usable.
0.7.0¶
Released April 9, 2020.
Backwards Compatibility Notes¶
Packages imported from memory using PyOxidizer now set
__path__
with a value formed by joining the current executable’s path with the package name. This mimics the behavior ofzipimport
.Resolved Python resource names have changed behavior. See the note in the bug fixes section below.
The
PythonDistribution.to_python_executable()
Starlark method has added apackaging_policy
named argument as its 2nd argument / 1st named argument. If you were affected by this, you should add argument names to all arguments passed to this method.The default Rust project for built executables now builds executables such that dynamic symbols are exported from the executable. This change is necessary in order to support executables loading Python extension modules, which are shared libraries which need access to Python symbols defined in executables.
The
PythonResourceData
Starlark type has been renamed toPythonPackageResource
.The
PythonDistribution.resources_data()
Starlark method has been renamed toPythonDistribution.package_resources()
.The
PythonExecutable.to_embedded_data()
Starlark method has been renamed toPythonExecutable.to_embedded_resources()
.The
PythonEmbeddedData
Starlark type has been renamed toPythonEmbeddedResources
.The format of Python resource data embedded in binaries has been completely rewritten. The separate modules and resource data structures have been merged into a single data structure. Embedded resources data can now express more primitives such as package distribution metadata and different bytecode optimization levels.
The pyembed crate now has a dev dependency on the pyoxidizer crate in order to run tests.
Bug Fixes¶
PyOxidizer’s importer now always sets
__path__
on imported packages in accordance with Python’s stated behavior (#51).The mechanism for resolving Python resource files from the filesystem has been rewritten. Before, it was possible for files like
package/resources/foo.txt
to be normalized to a (package, resource_name) tuple of (package, resources.foo.txt), which was weird and not compatible with Python’s resource loading mechanism. Resources in sub-directories should no longer encounter munging of directory separators to.
. In the above example, the resource path will now be expressed as(package, resources/foo.txt)
.Certain packaging actions are only performed once during building instead of twice. The user-visible impact of this change is that some duplicate log messages no longer appear.
Added a missing ) for add_python_resources() in auto-generated pyoxidizer.bzl files.
New Features¶
Python resource scanning now recognizes
*.dist-info
and*.egg-info
directories as package distribution metadata. Files within these directories are exposed to Starlark asPythonPackageDistributionResource
instances. These resources can be added to the embedded resources payload and made available for loading from memory or the filesystem, just like any other resource. The custom Python importer implementsget_distributions()
and returns objects that expose package distribution files. However, functionality of the returned distribution objects is not yet complete. See importlib.metadata Compatibility for details.The custom Python importer now implements
get_data(path)
, allowing loading of resources from filesystem paths (#139).The
PythonDistribution.to_python_executable()
Starlark method now accepts apackaging_policy
argument to control a policy and default behavior for resources on the produced executable. Using this argument, it is possible to control how resources should be materialized. For example, you can specify that resources should be loaded from memory if supported and from the filesystem if not. The argument can also be used to materialize the Python standard library on the filesystem, like how Python distributions typically work.Python resources can now be installed next to built binaries using the new Starlark functions
PythonExecutable.add_filesystem_relative_module_source()
,PythonExecutable.add_filesystem_relative_module_bytecode()
,PythonExecutable.add_filesystem_relative_package_resource()
,PythonExecutable.add_filesystem_relative_extension_module()
,PythonExecutable.add_filesystem_relative_python_resource()
,PythonExecutable.add_filesystem_relative_package_distribution_resource()
, andPythonExecutable.add_filesystem_relative_python_resources()
. Unlike adding Python resources toFileManifest
instances, Python resources added this way have their metadata serialized into the built executable. This allows the special Python module importer present in built binaries to service theimport
request without going through Python’s default filesystem-based importer. Because metadata for the file-based Python resources is frozen into the application, Python has to do far less work at run-time to load resources, making operations faster. Resources loaded from the filesystem in this manner have attributes like__file__
,__cached__
, and__path__
set, emulating behavior of the default Python importer. The custom import now also implements theimportlib.abc.ExecutionLoader
interface.Windows binaries can now import extension modules defined as shared libraries (e.g.
.pyd
files) from memory. PyOxidizer will detect.pyd
files during packaging and embed them into the binary as resources. When the module is imported, the extension module/shared library is loaded from memory and initialized. This feature enables PyOxidizer to package pre-built extension modules (e.g. from Windows binary wheels published on PyPI) while still maintaining the property of a (mostly) self-contained executable.Multiple bytecode optimization levels can now be embedded in binaries. Previously, it was only possible to embed bytecode for a given module at a single optimization level.
The
default_python_distribution()
Starlark function now accepts valuesstandalone_static
andstandalone_dynamic
to specify a standalone distribution that is either statically or dynamically linked.Support for parsing version 4 of the
PYTHON.json
distribution descriptor present in standalone Python distribution archives.Default Python distributions upgraded to CPython 3.7.7.
Other Relevant Changes¶
The directory for downloaded Python distributions in the build directory now uses a truncated SHA-256 hash instead of the full hash to help avoid path length limit issues (#224).
The documentation for the
pyembed
crate has been moved out of the Sphinx documentation and into the Rust crate itself. Rendered docs can be seen by following the Documentation link at https://crates.io/crates/pyembed or by runningcargo doc
from a source checkout.
0.6.0¶
Released February 12, 2020.
Backwards Compatibility Notes¶
The
default_python_distribution()
Starlark function now accepts aflavor
argument denoting the distribution flavor.The
pyembed
crate no longer includes the auto-generated default configuration file. Instead, it is consumed by the application that instantiates a Python interpreter.Rust projects for the main executable now utilize and require a Cargo build script so metadata can be passed from
pyembed
to the project that is consuming it.The
pyembed
crate is no longer added to created Rust projects. Instead, the generatedCargo.toml
will reference a version of thepyembed
crate identical to thePyOxidizer
version currently running. Or ifpyoxidizer
is running from a Git checkout of the canonicalPyOxidizer
Git repository, a local filesystem path will be used.The fields of
EmbeddedPythonConfig
andpyembed::PythonConfig
have been renamed and reordered to align with Python 3.8’s config API naming. This was done for the Starlark type in version 0.5. We have made similar changes to 0.6 so naming is consistent across the various types.
Bug Fixes¶
Module names without a
.
are now properly recognized when scanning the filesystem for Python resources and a package allow list is used (#223). Previously, if filtering scanned resources through an explicit list of allowed packages, the top-level module/package without a dot in its full name would not be passed through the filter.
New Features¶
The
PythonDistribution()
Starlark function now accepts aflavor
argument to denote the distribution type. This allows construction of alternate distribution types.The
default_python_distribution()
Starlark function now accepts aflavor
argument which can be set towindows_embeddable
to return a distribution based on the zip file distributions published by the official CPython project.The
pyembed
crate and generated Rust projects now have variousbuild-mode-*
feature flags to control how build artifacts are built. See PyOxidizer Rust Projects for more.The
pyembed
crate can now be built standalone, without being bound to a specificPyOxidizer
configuration.The
register_target()
Starlark function now accepts an optionaldefault_build_script
argument to define the default target when evaluating in Rust build script mode.The
pyembed
crate now builds against publishedcpython
andpython3-sys
crates instead of a a specific Git commit.Embedded Python interpreters can now be configured to run a file specified by a filename. See the
run_file
argument ofPythonInterpreterConfig
.
Other Relevant Changes¶
Rust internals have been overhauled to use traits to represent various types, namely Python distributions. The goal is to allow different Python distribution flavors to implement different logic for building binaries.
The
pyembed
crate’sbuild.rs
has been tweaked so it can support calling out topyoxidizer
. It also no longer has a build dependency onpyoxidizer
.
0.5.1¶
Released January 26, 2020.
Bug Fixes¶
Fixed bad Starlark example for building
black
in docs.Remove resources attached to packages that don’t exist. (This was a regression in 0.5.)
Warn on failure to annotate a package. (This was a regression in 0.5.)
Building embedded Python resources now emits warnings when
__file__
is seen. (This was a regression in 0.5.)Missing parent packages are now automatically added when constructing embedded resources. (This was a regression in 0.5.)
0.5.0¶
Released January 26, 2020.
General Notes¶
This release of PyOxidizer is significant rewrite of the previous version. The impetus for the rewrite is to transition from TOML to Starlark configuration files. The new configuration file format should allow vastly greater flexibility for building applications and will unlock a world of new possibilities.
The transition to Starlark configuration files represented a shift from static configuration to something more dynamic. This required refactoring a ton of code.
As part of refactoring code, we took the opportunity to shore up lots of the code base. PyOxidizer was the project author’s first real Rust project and a lot of bad practices (such as use of .unwrap()/panics) were prevalent. The code mostly now has proper error handling. Another new addition to the code is unit tests. While coverage still isn’t great, we now have tests performing meaningful packaging activities. So regressions should hopefully be less common going forward.
Because of the scale of the rewritten code in this release, it is expected that there are tons of bugs of regressions. This will likely be a transitional release with a more robust release to follow.
Backwards Compatibility Notes¶
Support for building distributions/installers has been temporarily dropped.
Support for installing license files has been temporarily dropped.
Python interpreter configuration setting names have been changed to reflect names from Python 3.8’s interpreter initialization API.
.egg-info
directories are now ignored when scanning for Python resources on the filesystem (matching the behavior for.dist-info
directories).The
pyoxidizer init
sub-command has been renamed toinit-rust-project
.The
pyoxidizer app-path
sub-command has been removed.Support for building distributions has been removed.
The minimum Rust version to build has been increased from 1.31 to 1.36. This is mainly due to requirements from the
starlark
crate. We could potentially reduce the minimum version requirements again with minimal changes to 3rd party crates.PyOxidizer configuration files are now Starlark instead of TOML files. The default file name is
pyoxidizer.bzl
instead ofpyoxidizer.toml
. All existing configuration files will need to be ported to the new format.
Bug Fixes¶
The
repl
run mode now properly exits with a non-zero exit code if an error occurs.Compiled C extensions now properly honor the
ext_package
argument passed tosetup()
, resulting in extensions which properly have the package name in their extension name (#26).
New Features¶
A glob()` function has been added to config files to allow referencing existing files on the filesystem.
The in-memory
MetaPathFinder
now implementsfind_module()
.A
pyoxidizer init-config-file
command has been implemented to create just apyoxidizer.bzl
configuration file.A
pyoxidizer python-distribution-info
command has been implemented to print information about a Python distribution archive.The
EmbeddedPythonConfig()
config function now accepts alegacy_windows_stdio
argument to control the value ofPy_LegacyWindowsStdioFlag
(#190).The
EmbeddedPythonConfig()
config function now accepts alegacy_windows_fs_encoding
argument to control the value ofPy_LegacyWindowsFSEncodingFlag
.The
EmbeddedPythonConfig()
config function now accepts anisolated
argument to control the value ofPy_IsolatedFlag
.The
EmbeddedPythonConfig()
config function now accepts ause_hash_seed
argument to control the value ofPy_HashRandomizationFlag
.The
EmbeddedPythonConfig()
config function now accepts aninspect
argument to control the value ofPy_InspectFlag
.The
EmbeddedPythonConfig()
config function now accepts aninteractive
argument to control the value ofPy_InteractiveFlag
.The
EmbeddedPythonConfig()
config function now accepts aquiet
argument to control the value ofPy_QuietFlag
.The
EmbeddedPythonConfig()
config function now accepts averbose
argument to control the value ofPy_VerboseFlag
.The
EmbeddedPythonConfig()
config function now accepts aparser_debug
argument to control the value ofPy_DebugFlag
.The
EmbeddedPythonConfig()
config function now accepts abytes_warning
argument to control the value ofPy_BytesWarningFlag
.The
Stdlib()
packaging rule now now accepts an optionalexcludes
list of modules to ignore. This is useful for removing unnecessary Python packages such asdistutils
,pip
, andensurepip
.The
PipRequirementsFile()
andPipInstallSimple()
packaging rules now accept an optionalextra_env
dict of extra environment variables to set when invokingpip install
.The
PipRequirementsFile()
packaging rule now accepts an optionalextra_args
list of extra command line arguments to pass topip install
.
Other Relevant Changes¶
PyOxidizer no longer requires a forked version of the
rust-cpython
project (thepython3-sys
andcpython
crates. All changes required by PyOxidizer are now present in the official project.
0.4.0¶
Released October 27, 2019.
Backwards Compatibility Notes¶
The
setup-py-install
packaging rule now has itspackage_path
evaluated relative to the PyOxidizer config file path rather than the current working directory.
Bug Fixes¶
Windows now explicitly requires dynamic linking against
msvcrt
. Previously, this wasn’t explicit. And sometimes linking the final executable would result in unresolved symbol errors because the Windows Python distributions used external linkage of CRT symbols and for some reason Cargo wasn’t dynamically linking the CRT.Read-only files in Python distributions are now made writable to avoid future permissions errors (#123).
In-memory
InspectLoader.get_source()
implementation no longer errors due to passing amemoryview
to a function that can’t handle it (#134).In-memory
ResourceReader
now properly handles multiple resources (#128).
New Features¶
Added an
app-path
command that prints the path to a packaged application. This command can be useful for tools calling PyOxidizer, as it will emit the path containing the packaged files without forcing the caller to parse command output.The
setup-py-install
packaging rule now has anexcludes
option that allows ignoring specific packages or modules..py
files installed into app-relative locations now have corresponding.pyc
bytecode files written.The
setup-py-install
packaging rule now has anextra_global_arguments
option to allow passing additional command line arguments to thesetup.py
invocation.Packaging rules that invoke
pip
orsetup.py
will now set aPYOXIDIZER=1
environment variable so Python code knows at packaging time whether it is running in the context of PyOxidizer.The
setup-py-install
packaging rule now has anextra_env
option to allow passing additional environment variables tosetup.py
invocations.[[embedded_python_config]]
now supports asys_frozen
flag to control settingsys.frozen = True
.[[embedded_python_config]]
now supports asys_meipass
flag to control settingsys._MEIPASS = <exe directory>
.Default Python distribution upgraded to 3.7.5 (from 3.7.4). Various dependency packages also upgraded to latest versions.
All Other Relevant Changes¶
Built extension modules marked as app-relative are now embedded in the final binary rather than being ignored.
0.3.0¶
Released on August 16, 2019.
Backwards Compatibility Notes¶
The
pyembed::PythonConfig
struct now has an additionalextra_extension_modules
field.The default musl Python distribution now uses LibreSSL instead of OpenSSL. This should hopefully be an invisible change.
Default Python distributions now use CPython 3.7.4 instead of 3.7.3.
Applications are now built into directories named
apps/<app_name>/<target>/<build_type>
rather thanapps/<app_name>/<build_type>
. This enables builds for multiple targets to coexist in an application’s output directory.The
program_name
field from the[[embedded_python_config]]
config section has been removed. At run-time, the current executable’s path is always used when callingPy_SetProgramName()
.The format of embedded Python module data has changed. The
pyembed
crate andpyoxidizer
versions must match exactly or else thepyembed
crate will likely crash at run-time when parsing module data.
Bug Fixes¶
The
libedit
extension variant for thereadline
extension should now link on Linux. Before, attempting to link a binary using this extension variant would result in missing symbol errors.The
setup-py-install
[[packaging_rule]]
now performs actions to appeasesetuptools
, thus allowing installation of packages usingsetuptools
to (hopefully) work without issue (#70).The
virtualenv
[[packaging_rule]]
now properly finds thesite-packages
directory on Windows (#83).The
filter-include
[[packaging_rule]]
no longer requires bothfiles
andglob_files
be defined (#88).import ctypes
now works on Windows (#61).The in-memory module importer now implements
get_resource_reader()
instead ofget_resource_loader()
. (The CPython documentation steered us in the wrong direction - https://bugs.python.org/issue37459.)The in-memory module importer now correctly populates
__package__
in more cases than it did previously. Before, whether a module was a package was derived from the presence of afoo.bar
module. Now, a module will be identified as a package if the file providing it is named__init__
. This more closely matches the behavior of Python’s filesystem based importer. (#53)
New Features¶
The default Python distributions have been updated. Archives are generally about half the size from before. Tcl/tk is included in the Linux and macOS distributions (but PyOxidizer doesn’t yet package the Tcl files).
Extra extension modules can now be registered with
PythonConfig
instances. This can be useful for having the application embedding Python provide its own extension modules without having to go through Python build mechanisms to integrate those extension modules into the Python executable parts.Built applications now have the ability to detect and use
terminfo
databases on the execution machine. This allows applications to interact with terminals properly. (e.g. the backspace key will now work in interactivepdb
sessions). By default, applications on non-Windows platforms will look forterminfo
databases at well-known locations and attempt to load them.Default Python distributions now use CPython 3.7.4 instead of 3.7.3.
A warning is now emitted when a Python source file contains
__file__
. This should help trace down modules using__file__
.Added 32-bit Windows distribution.
New
pyoxidizer distribution
command for producing distributable artifacts of applications. Currently supports building tar archives and.msi
and.exe
installers using the WiX Toolset.Libraries required by C extensions are now passed into the linker as library dependencies. This should allow C extensions linked against libraries to be embedded into produced executables.
pyoxidizer --verbose
will now pass verbose to invokedpip
andsetup.py
scripts. This can help debug what Python packaging tools are doing.
All Other Relevant Changes¶
The list of modules being added by the Python standard library is no longer printed during rule execution unless
--verbose
is used. The output was excessive and usually not very informative.
0.2.0¶
Released on June 30, 2019.
Backwards Compatibility Notes¶
Applications are now built into an
apps/<appname>/(debug|release)
directory instead ofapps/<appname>
. This allows debug and release builds to exist side-by-side.
Bug Fixes¶
Extracted
.egg
directories in Python package directories should now have their resources detected properly and not as Python packages with the name*.egg
.site-packages
directories are now recognized as Python resource package roots and no longer have their contents packaged under asite-packages
Python package.
New Features¶
Support for building and embedding C extensions on Windows, Linux, and macOS in many circumstances. See Native Extension Modules for support status.
pyoxidizer init
now accepts a--pip-install
option to pre-configure generatedpyoxidizer.toml
files with packages to install viapip
. Combined with the--python-code
option, it is now possible to createpyoxidizer.toml
files for a ready-to-use Python application!pyoxidizer
now accepts a--verbose
flag to make operations more verbose. Various low-level output is no longer printed by default and requires--verbose
to see.
All Other Relevant Changes¶
Packaging now automatically creates empty modules for missing parent packages. This prevents a module from being packaged without its parent. This could occur with namespace packages, for example.
pip-install-simple
rule now passes--no-binary :all:
to pip.Cargo packages updated to latest versions.
0.1.3¶
Released on June 29, 2019.
Bug Fixes¶
Fix Python refcounting bug involving call to
PyImport_AddModule()
whenmode = module
evaluation mode is used. The bug would likely lead to a segfault when destroying the Python interpreter. (#31)Various functionality will no longer fail when running
pyoxidizer
from a Git repository that isn’t the canonicalPyOxidizer
repository. (#34)
New Features¶
pyoxidizer init
now accepts a--python-code
option to control which Python code is evaluated in the produced executable. This can be used to create applications that do not run a Python REPL by default.pip-install-simple
packaging rule now supportsexcludes
for excluding resources from packaging. (#21)pip-install-simple
packaging rule now supportsextra_args
for adding parameters to the pip install command. (#42)
All Relevant Changes¶
Minimum Rust version decreased to 1.31 (the first Rust 2018 release). (#24)
Added CI powered by Azure Pipelines. (#45)
Comments in auto-generated
pyoxidizer.toml
have been tweaked to improve understanding. (#29)
0.1.2¶
Released on June 25, 2019.
Bug Fixes¶
Honor
HTTP_PROXY
andHTTPS_PROXY
environment variables when downloading Python distributions. (#15)Handle BOM when compiling Python source files to bytecode. (#13)
All Relevant Changes¶
pyoxidizer
now verifies the minimum Rust version meets requirements before building.
0.1.1¶
Released on June 24, 2019.
Bug Fixes¶
pyoxidizer
binaries built from crates should now properly refer to an appropriate commit/tag in PyOxidizer’s canonical Git repository in auto-generatedCargo.toml
files. (#11)
0.1¶
Released on June 24, 2019. This is the initial formal release of PyOxidizer.
The first pyoxidizer
crate was published to crates.io
.
New Features¶
Support for building standalone, single file executables embedding Python for 64-bit Windows, macOS, and Linux.
Support for importing Python modules from memory using zero-copy.
Basic Python packaging support.
Support for jemalloc as Python’s memory allocator.
pyoxidizer
CLI command with basic support for managing project lifecycle.