generic embedding feedback

[dae]

Carrying on from #467 (comment)

Let me firstly briefly elaborate on my use case/desired outcome:

• I have a medium-sized Python app which uses non-stdlib extension modules, and bundles various data files. I'd like to store the Python bytecode and most of the data files inside the binary, and spill the extension modules and certain data files to external files.
• I'd like to be able to customize the Rust code, so I can set an icon on Windows, print a startup message, and possibly do other things in the future.
• I currently need to specify a non-default Python distribution on Linux to work around PyQt segfaults on modern Linux distros when run with standalone python astral-sh/python-build-standalone#95

If I have correctly understood how generate-python-embedding-artifacts works (please correct me if I'm wrong), I'm not sure it will work for the above use case:

• Because it uses a stock pyoxidizer config, all the non-stdlib files/bytecode that could be stored in the binary would presumably need to be manually copied next to the binary instead. PyOxidizerFinder is one of PyOxidizer's really appealing points, and it would be a shame for that to only work with the stdlib.
• The extension modules and external data files could be copied in from a separate pyoxidizer invocation, though it's a bit wasteful to build another binary just so the artifacts can be extracted.
• It looks like a custom Python distribution can not be provided.
• Presumably it's not possible to customize things like bytecode optimization level?

For now, either #466 or #468 handle my use case pretty well. I like the simplicity of #468, but like the flexibility of #466, and it builds slightly quicker/shows the cargo output in color :-) Not suggesting either of these be included as is, but hopefully they give you an idea of what I'm trying to achieve.

If there were a command like 'generate-all-embedding-artifacts' that used the user-provided pyoxidizer.bzl config and wrote extra_files to disk, I think that would solve all my problems.

[indygreg]

Thanks for taking the time to type this all up. Detailed user reports like this are very valuable feedback.

* Because it uses a stock pyoxidizer config, all the non-stdlib files/bytecode that could be stored in the binary would presumably need to be manually copied next to the binary instead. PyOxidizerFinder is one of PyOxidizer's really appealing points, and it would be a shame for that to only work with the stdlib.

The Starlark configs and related pyoxidizer commands don't have a good way of handling this scenario today but doing what you want is possible.

oxidized_importer exposes enough APIs to Python where you can build your own packed resources files. See https://pyoxidizer.readthedocs.io/en/latest/oxidized_importer_freezing_applications.html for example docs.

In addition, the pyembed crate allows you to register multiple sources for the packed resources. The default config just loads the resources blob that pyoxidizer generates and embeds in the binary. You can generate your own standalone packed resources files, embed them in your custom binary with include_bytes!, then add a reference to this blob to pyembed::OxidizedPythonInterpreterConfig.packed_resources to have them indexed by oxidized_importer.OxidizedFinder during interpreter initialization.

* The extension modules and external data files could be copied in from a separate pyoxidizer invocation, though it's a bit wasteful to build another binary just so the artifacts can be extracted.

I think an area worth exploring is enhancing the Starlark APIs to make it easier to emit packed resources and other build artifacts without going through the entire build an executable code path.

* It looks like a custom Python distribution can not be provided.

It is possible to construct a PythonDistribution from a source path/URL instead of going through default_python_distribution(). See https://pyoxidizer.readthedocs.io/en/pyoxidizer-0.17/pyoxidizer_config_type_python_distribution.html#starlark_pyoxidizer.PythonDistribution.__init__.

* Presumably it's not possible to customize things like bytecode optimization level?

There are attributes on the packaging policy to control bytecode optimization level: https://pyoxidizer.readthedocs.io/en/pyoxidizer-0.17/pyoxidizer_config_type_python_packaging_policy.html#starlark_pyoxidizer.PythonPackagingPolicy.bytecode_optimize_level_zero.

If there were a command like 'generate-all-embedding-artifacts' that used the user-provided pyoxidizer.bzl config and wrote extra_files to disk, I think that would solve all my problems.

Yes, I agree this would solve a lot of problems. However, it gets a bit complicated with how commands like this interact with Starlark. You need to define a Starlark target to evaluate. So I'm thinking that unless we change the Starlark target mechanism (which is a possibility because IMO it feels janky), we should provide a Starlark type to represent build artifacts and config files can provide a target that returns an instance and where pyoxidizer build will write out all the support files without building a Rust project.

[dae]

oxidized_importer exposes enough APIs to Python where you can build your own packed resources files. See https://pyoxidizer.readthedocs.io/en/latest/oxidized_importer_freezing_applications.html for example docs.

In addition, the pyembed crate allows you to register multiple sources for the packed resources.

That's really nice - it sounds like I could use a combination of those two in the future to build and bundle the non-stdlib resources, avoiding the need to patch pyoxidizer.

It is possible to construct a PythonDistribution from a source path/URL instead of going through default_python_distribution()

There are attributes on the packaging policy to control bytecode optimization level:

Yep, I've successfully done those things when using my own pyoxidizer.bzl, but presumably that would not be possible when using generate-python-embedding-artifacts, since it does not use the user-provided config file?

However, it gets a bit complicated with how commands like this interact with Starlark. You need to define a Starlark target to evaluate.

There's already the "resources" target & .to_embedded_resources() - would it make sense if it accepted an optional include_extra_files argument?

[Stradivariuskein]

Hello, I'm trying to add an icon to my .exe built in Python with PyOxidizer, but I can't seem to get it to work. I'm not sure if this is the right place to ask this. Thank you in advance.

code.bzl:

def make_msi(exe):
    # See the full docs for more. But this will convert your Python executable
    # into a `WiXMSIBuilder` Starlark type, which will be converted to a Windows
    # .msi installer when it is built.
    #exe.windows_subsystem("windows")
    msi_builder = exe.to_wix_msi_builder(
        # Simple identifier of your app.
        "epic-alert",
        # The name of your application.
        "Epic Alert",
        # The version of your application.
        "0",
        # The author/manufacturer of your application.
        "autor",        
    )
    msi_builder.product_icon_path= "J:\\Documentos\\python\\auto-buy-game\\build_py2exe\\build_pyobxidian\\warning.ico"
    return msi_builder