core-v4 has missing entrypoint manifest

[ascopes]

Hey.

Just raising this as I've spotted it whilst testing things out locally.

It seems that core-v4 is missing a Main-Class manifest entry, so tools such as mine are unable to work out the entrypoint to run it as a command-line application.

I was trying to find an entrypoint but cannot seem to find one define in v4. Is this intentional? If so, is there the possibility of adding one in that is compatible with what is available in the original protoc, since this will make it so that I do not have to rely on project-specific details to provide integrations with this project from my side 🙂. I also imagine this will be desirable for other projects as well looking to swap out implementations.

[andreaTP]

Hi @ascopes thanks for rising this issue!

I was trying to find an entrypoint but cannot seem to find one define in v4. Is this intentional?

Yes, that's completely intentional, we are exposing the protoc functionality as a library and we do not expect this jar to be used as a standalone main.

f so, is there the possibility of adding one in that is compatible with what is available in the original protoc, since this will make it so that I do not have to rely on project-specific details to provide integrations with this project from my side

Is possible, but, I believe, is a lot of work.
One of the key aspects of this repo, is that we are deliberately stripping out the CLI and subprocess handling as they are incompatible with the Wasm build.

If you have an interest in restoring the CLI behavior I'd be happy to evaluate an initial POC, but I'm afraid the effort is going to bring a lot of additional complexity.

[ascopes]

Ahh okay, that's a shame but understandable.

My hope was to not need to plumb specific implementation details of a specific integration but I may need to go back to the drawing board with this one.

[andreaTP]

I think the current incarnation of the library reflects the expectations, i.e. we are NOT saying we do honor all protoc flags, we are saying that we are able to reproduce the functionality by directly linking the C++ protobuf implementation.

Please note that the "command line approach" suffers other kind of weaknesses(for example).

I go ahead and close this issue, but, feel free to re-open or comment further if I missed something relevant.

[ascopes]

@andreaTP regarding the command line length issue, the general way to deal with this is to write an argfile containing the arguments rather than passing them directly to protoc. You should be able to then pass that path as the sole argument with an @ at the start and it be loaded from the virtual file system.

I have to do a similar thing here: https://github.com/ascopes/protobuf-maven-plugin/blob/ef8732eab26ca17d19fb0792f9973ec93e273bb5/protobuf-maven-plugin/src/main/java/io/github/ascopes/protobufmavenplugin/protoc/ProtocExecutor.java#L82

Regarding the subprocess challenge, would an option be utilising Chicory host functions to handoff subprocess handling back to Java? As far as the protoc plugins are concerned, they are just a subprocess that is invoked, reading a proto descriptor from stdin and writing one to stdout.

[andreaTP]

regarding the command line length issue

Ok, thanks

Regarding the subprocess challenge

That's basically what we are doing with the runNativePlugin function.
Triggering the subprocess from protoc should be possible with wasm imported functions, but it likely require some source patching or deeper research on the subject.

[andreaTP]

I thought a little more about this, and ultimately I think it is likely possible to provide a custom version of subprocess.cc, with specific semantics, and get the CLI part to work.

I keep it at low priority on my list, if anyone is willing to pick it up for a spike let me know and I can provide details ( cc. @ascopes @carlesarnal ).

[ascopes]

Thanks! I might be able to look into it in a couple weeks time when I am free, but this'd definitely be useful if possible.

[ascopes]

I tried to take a look into this recently but I am a little stumped as to exactly how this works with regards to which parts of libc/stdc++ need stubbing and which do not... is the expectation that fork/exec on posix and w32api calls on win32 would need to be stubbed or would it be the entire subprocess module itself?

[ascopes]

Spent the afternoon looking into this. I've got to the point where the CLI seems to be working and I am calling out to Java code to invoke a subprocess. Just fighting with some linkage issues at the moment. If I can get it working, I'll push it to a PoC branch as I'm not too sure how to integrate it into the existing entrypoint right now (for now, I'm building a second WASM blob that calls the original main.cc directly instead).

[ascopes]

Work that I've played around with. I've added some notes to the commit on what has been done and where I am at... I'm a bit stuck for how to proceed though and have run out of time for now, but hoping this might be able to be used as a head start @andreaTP?

c92de47

#30: Scaffold to configure build for protoc with original main.cc and delegated subprocess management

• Builds with original protoc.exe main.cc to give full entrypoint.
• Uses a Chicory host function to configure calling out to a subprocess (not
  implemented properly yet)

We need to just make the java-based host function consume a string holding
the codegen request for a downstream subprocess and spawn a java.lang.Process
to handle this. Any output written to the subprocess stdout just needs to
be written to a malloced cstr to be passed back to protoc.

Haven't had time to implement the actual subprocess call as I've been
stuck fighting with malloc to be exported in the right place, I've likely
done something silly there.

I was unsure how to integrate this with the existing logic so for now I've
made it write out a second binary blob with the protoc.exe main.cc included
instead of modifying the original.

Optimisation has been disabled as it appears to have been optimising out
the main.cc logic and just writing an empty string to stdout. Removing
optimisation gives me the full --help 'USAGE' section as expected.

Hopefully, this just needs a tidy up and the java.lang.Process stub to
be implemented properly and it should work?

Current state, with no arguments:

java -classpath ... io.roastedroot.protobuf4j.v3.Protoc

Usage:  [OPTION] PROTO_FILES
Parse PROTO_FILES and generate output based on the options given:
  -IPATH, --proto_path=PATH   Specify the directory in which to search for
                              imports.  May be specified multiple times;
                              directories will be searched in order.  If not
                              given, the current working directory is used.
                              If not found in any of the these directories,
                              the --descriptor_set_in descriptors will be
                              checked for required proto file.
  --version                   Show version info and exit.
  -h, --help                  Show this text and exit.
  --encode=MESSAGE_TYPE       Read a text-format message of the given type
                              from standard input and write it in binary
                              to standard output.  The message type must
                              be defined in PROTO_FILES or their imports.
  --deterministic_output      When using --encode, ensure map fields are
                              deterministically ordered. Note that this order
                              is not canonical, and changes across builds or
                              releases of protoc.
  --decode=MESSAGE_TYPE       Read a binary message of the given type from
                              standard input and write it in text format
                              to standard output.  The message type must
                              be defined in PROTO_FILES or their imports.
  --decode_raw                Read an arbitrary protocol message from
                              standard input and write the raw tag/value
                              pairs in text format to standard output.  No
                              PROTO_FILES should be given when using this
                              flag.
  --descriptor_set_in=FILES   Specifies a delimited list of FILES
                              each containing a FileDescriptorSet (a
                              protocol buffer defined in descriptor.proto).
                              The FileDescriptor for each of the PROTO_FILES
                              provided will be loaded from these
                              FileDescriptorSets. If a FileDescriptor
                              appears multiple times, the first occurrence
                              will be used.
  -oFILE,                     Writes a FileDescriptorSet (a protocol buffer,
    --descriptor_set_out=FILE defined in descriptor.proto) containing all of
                              the input files to FILE.
  --include_imports           When using --descriptor_set_out, also include
                              all dependencies of the input files in the
                              set, so that the set is self-contained.
  --include_source_info       When using --descriptor_set_out, do not strip
                              SourceCodeInfo from the FileDescriptorProto.
                              This results in vastly larger descriptors that
                              include information about the original
                              location of each decl in the source file as
                              well as surrounding comments.
  --retain_options            When using --descriptor_set_out, do not strip
                              any options from the FileDescriptorProto.
                              This results in potentially larger descriptors
                              that include information about options that were
                              only meant to be useful during compilation.
  --dependency_out=FILE       Write a dependency output file in the format
                              expected by make. This writes the transitive
                              set of input file paths to FILE
  --error_format=FORMAT       Set the format in which to print errors.
                              FORMAT may be 'gcc' (the default) or 'msvs'
                              (Microsoft Visual Studio format).
  --fatal_warnings            Make warnings be fatal (similar to -Werr in
                              gcc). This flag will make protoc return
                              with a non-zero exit code if any warnings
                              are generated.
  --print_free_field_numbers  Print the free field numbers of the messages
                              defined in the given proto files. Extension ranges
                              are counted as occupied fields numbers.
  --enable_codegen_trace      Enables tracing which parts of protoc are
                              responsible for what codegen output. Not supported
                              by all backends or on all platforms.
  --plugin=EXECUTABLE         Specifies a plugin executable to use.
                              Normally, protoc searches the PATH for
                              plugins, but you may specify additional
                              executables not in the path using this flag.
                              Additionally, EXECUTABLE may be of the form
                              NAME=PATH, in which case the given plugin name
                              is mapped to the given executable even if
                              the executable's own name differs.
  --cpp_out=OUT_DIR           Generate C++ header and source.
  --csharp_out=OUT_DIR        Generate C# source file.
  --java_out=OUT_DIR          Generate Java source file.
  --kotlin_out=OUT_DIR        Generate Kotlin file.
  --objc_out=OUT_DIR          Generate Objective-C header and source.
  --php_out=OUT_DIR           Generate PHP source file.
  --pyi_out=OUT_DIR           Generate python pyi stub.
  --python_out=OUT_DIR        Generate Python source file.
  --ruby_out=OUT_DIR          Generate Ruby source file.
  --rust_out=OUT_DIR          Generate Rust sources.
  @<filename>                 Read options and filenames from file. If a
                              relative file path is specified, the file
                              will be searched in the working directory.
                              The --proto_path option will not affect how
                              this argument file is searched. Content of
                              the file will be expanded in the position of
                              @<filename> as in the argument list. Note
                              that shell expansion is not applied to the
                              content of the file (i.e., you cannot use
                              quotes, wildcards, escapes, commands, etc.).
                              Each line corresponds to a single argument,
                              even if it contains spaces.

Process finished with exit code 0

Calling a protoc plugin (in this case protoc-gen-grpc-java from Maven Central):

java -classpath ... io.roastedroot.protobuf4j.v3.Protoc --plugin=protoc-gen-grpc-java=/home/ashley/code/thirdparty/protobuf4j/.build/protoc-gen-grpc-java-1.78.0-linux-x86_64.exe --protoc-gen-grpc-java_out=protoc/target/example-out --java_out=/home/ashley/code/thirdparty/protobuf4j/protoc/target/example-out --proto_path=/home/ashley/code/thirdparty/protobuf4j/protoc/src/test/scratch /home/ashley/code/thirdparty/protobuf4j/protoc/src/test/scratch/helloworld.proto

RunJavaDelegatedSubprocess called with protoc-gen-protoc-gen-grpc-java 0 
�helloworld.prot�����"
Exception in thread "main" com.dylibso.chicory.wasm.InvalidException: Unknown export with name malloc
	at com.dylibso.chicory.runtime.Instance$Exports.getExport(Instance.java:204)
	at com.dylibso.chicory.runtime.Instance$Exports.function(Instance.java:218)
	at io.roastedroot.protobuf4j.v3.Protoc.writeCString(Protoc.java:95)
	at io.roastedroot.protobuf4j.v3.Protoc.lambda$runJavaDelegatedSubprocessFunction$0(Protoc.java:88)
	at io.roastedroot.protobuf4j.v3.ProtocV3MachineShaded.callHostFunction(Shaded.java:40)
	at io.roastedroot.protobuf4j.v3.ProtocV3MachineFuncGroup_0.func_0(Unknown Source)
	at io.roastedroot.protobuf4j.v3.ProtocV3MachineFuncGroup_0.func_177(Unknown Source)
	at io.roastedroot.protobuf4j.v3.ProtocV3MachineFuncGroup_0.func_142(Unknown Source)