write extra_files when building pyembed artifacts?

[dae]

Intended for discussion, not merging as-is. Based on the 0.17 branch; I'm having issues with the main branch which I'll look into more later.

I'd like to be able customize the icon of the produced .exe file on Windows, and run some custom Rust code before the Python interpreter is loaded. I ran into some problems along the way.

I started by using pyoxidizer init-rust-project sample. It tells me:

A new Rust binary application has been created in sample

This application can be built by doing the following:

  $ cd sample
  $ pyoxidizer build
  $ pyoxidizer run

[...]

So I add a print statement to the top of main.rs, and run build/run. Observations:

• run rebuilds everything again. In the main branch the build line appears to have been removed when creating a normal project, but is still shown when using init-rust-project.
• when run, the print statement doesn't appear.

After digging into the sources, I find that pyoxidizer will unconditionally create an ephemeral project when building, ignoring any sources in the current folder. Looks like this was previously reported in #439. Ok, I'll need to try the direct-with-cargo approach instead.

% zstdcat /Users/dae/Library/Caches/pyoxidizer/python_distributions/cpython-3.9.6-x86_64-apple-darwin-pgo-20210724T1424.tar.zst | tar xf -
% cd sample && CARGO_TARGET_DIR=target PYOXIDIZER_EXECUTABLE=$HOME/.cargo/bin/pyoxidizer PYTHON_SYS_EXECUTABLE=$HOME/python/install/bin/python3.9 PYOXIDIZER_CONFIG=$(pwd)/pyoxidizer.bzl cargo build --features cpython-link-unresolved-static,allocator-jemalloc
[...]
% ./target/debug/sample
[src/main.rs:37] "test" = "test"

Ok, that's promising. Now let's add an extension module into the equation:

    policy.resources_location_fallback = "filesystem-relative:prefix"
    python_config.run_command = "import google.protobuf.internal._api_implementation"
    exe.add_python_resources(exe.pip_download(["protobuf"]))

I rebuild and try again:

dae@dip:~/sample% ./target/debug/sample
[src/main.rs:37] "test" = "test"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: dlopen(/Users/dae/sample/target/debug/prefix/google/protobuf/internal/_api_implementation.cpython-39-darwin.so, 2): image not found
zsh: exit 1     ./target/debug/sample
dae@dip:~/sample% find . -name _api_implementation.cpython-39-darwin.so
dae@dip:~/sample%

Hmm, it seems extra_files are nowhere to be found. Back to digging through the source code. It looks like to_embedded_resources() ignores extra_files - they only seem to handled when
building an executable.

With the patch in this PR, I am able to successfully change the icon by adding it to the resources file. But I presume there are other cases where the extra_files should not be included. Which leads me to a few questions:

• Is there a better way to modify the default Rust/resources that I've missed?
• If not, would there be any interest in a change like this? Presumably it would need to be
  gated by an env var?

[dae]

Ok, I've rebased this over the main branch, and confirmed it worked on the Mac and Windows machines I tested when used with the commands mentioned on https://github.com/indygreg/PyOxidizer/pull/467/files

I hit a few stumbling blocks when trying to get 0.18 working:

• I first tried with standalone build mode and PYO3_PYTHON set to a standalone Python. It failed with:

  = note: ld: warning: directory not found for option '-L/install/lib'
          ld: library not found for -lpython3.9
          clang: error: linker command failed with exit code 1 (use -v to see invocation)


error: could not compile `python-oxidized-importer` due to previous error

Presumably pyo3 was trying to autodetect the Python layout, and failing. I was able to work around it by generating the artifacts in advance - is that the intended workflow now?

• I saw mention in the change notes of nightly no longer being required, but it seems that's not the case when building pyoxidizer itself - I ran into an issue that looks like the trait From is not implemented dtolnay/syn#776 (comment) when trying to build with 1.55 on Windows can't reproduce, seems ok now