Skip to content

Expose the full protoc CLI#54

Open
andreaTP wants to merge 3 commits intoroastedroot:mainfrom
andreaTP:issue-30
Open

Expose the full protoc CLI#54
andreaTP wants to merge 3 commits intoroastedroot:mainfrom
andreaTP:issue-30

Conversation

@andreaTP
Copy link
Copy Markdown
Contributor

This is a step forward to, eventually, enable #30 , I finally took the time to look into it.

I know this is not "exactly" in the direction you envision @ascopes , still it should enable the use case.
Please, @ascopes , verify those changes, and we can move from there.

protected abstract ProtobufTestAdapter createAdapter(Path workdir);

@Test
public void shouldInvokeMockSubprocessForExternalPlugin() throws Exception {
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For reviewers, this test shows the exposed "enabling" functionality

@ascopes
Copy link
Copy Markdown
Collaborator

ascopes commented Feb 17, 2026

Think this is a great start! When I have some time again I'll see if I can hook it into the tooling on my side and see how that goes!

Thanks for putting the time into getting this working.

@andreaTP
Copy link
Copy Markdown
Contributor Author

andreaTP commented Mar 5, 2026

@ascopes have you had the chance to try to integrate those changes into your pipeline, does it helps?

@ascopes
Copy link
Copy Markdown
Collaborator

ascopes commented Mar 5, 2026

@andreaTP I haven't just yet unfortunately, things have been pretty manic recently. Need to spend some time trying to work out the best way of exposing this such that it still remains totally agnostic.

Apologies for the delay. It is still on my radar 🙂

@andreaTP
Copy link
Copy Markdown
Contributor Author

andreaTP commented Mar 5, 2026

Thanks for getting back, that's fine, it's enough to hear is on your radar, take your time!

@ascopes
Copy link
Copy Markdown
Collaborator

ascopes commented Mar 8, 2026

@andreaTP regarding getting a build to try this on, am I able to just fudge the WASM binaries in from this branch into my local Maven repository, or does it require a build from scratch to work?

I might have a little time today to fiddle around with some stuff.

@andreaTP
Copy link
Copy Markdown
Contributor Author

andreaTP commented Mar 8, 2026

You can use the binaries committed on this branch 👍

@ascopes
Copy link
Copy Markdown
Collaborator

ascopes commented Mar 15, 2026

Just having a look at this, are we setting the Main-Class Manifest attribute anywhere for the core-v4 package?

➜  protobuf4j git:(issue-30) java -jar ~/.m2/repository/io/roastedroot/protobuf4j-v4/999-SNAPSHOT/protobuf4j-v4-999-SNAPSHOT.jar
no main manifest attribute, in /data/data/com.termux/files/home/.m2/repository/io/roastedroot/protobuf4j-v4/999-SNAPSHOT/protobuf4j-v4-999-SNAPSHOT.jar

Thinking we need to override the Maven JAR Plugin in the core-v3 and core-v4 projects to inject the Main-Class attribute into the manifest so the JAR can be executable without users needing to explicitly find out and specify the main class FQDN.

(Aware the classpath of the dependency tree is also missing from above but I have logic already in place to be able to compute the appropriate values for this already so that side of this should be fine).

@andreaTP
Copy link
Copy Markdown
Contributor Author

are we setting the Main-Class Manifest attribute anywhere for the core-v4 package?

not yet, it should be easy to do so though, either in the core package or in a dependent "CLI" module.
Are you blocked on this detail?

@ascopes
Copy link
Copy Markdown
Collaborator

ascopes commented Mar 16, 2026

@andreaTP not blocked per sé as luckily on my side I've already got the ability to inject the entrypoint. Was mostly mentioning in case you wanted to include it on this PR

@andreaTP
Copy link
Copy Markdown
Contributor Author

Good to hear!
This week I'm surely swamped, but, if you can confirm that this development makes it viable to use protobuf4j in the Maven plugin I'll go ahead on this effort 🙂

@ascopes
Copy link
Copy Markdown
Collaborator

ascopes commented Mar 22, 2026

Spent a little time looking into this today.

I had to add the following to the core-v4 module to get an entrypoint set up (not sure if I was missing where one was defined elsewhere or not):

package io.roastedroot.protobuf4j.v4;

import java.nio.file.Path;
import java.util.List;

public final class Main {

  private Main() {
    throw new UnsupportedOperationException();
  }

  public static void main(String[] args) {
    Protobuf.runProtoc(List.of(args), System.getenv(), Path.of("").toAbsolutePath());
  }
}

On my side in the Maven plugin, I've had to disable module path lookups as the JAR being generated apparently had an invalid file name when deriving an Automatic Module. I think this is likely able to be fixed by just adding an Automatic-Module-Name manifest entry in next to the Main-Class manifest entry we also discussed.

Upon running on my side, I can see that the Java protoc was run and exited successfully (no logs were output)

[INFO] Registering "/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it/protobuf4j-java-main/target/generated-sources/protobuf" as a Maven main source root for compiler plugins
[INFO] All sources will be compiled, as no previous build data was detected
[INFO] Generating source code from 1 proto source file and 0 descriptor files (discovered a total of 1 proto source file and 0 descriptor files)
[INFO] Invoking protoc. Enable debug logs (--debug --errors) for more details
[INFO] protoc (pid 79340) returned exit code 0 (success) after 489ms
[INFO] Embedding sources from "/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it/protobuf4j-java-main/src/main/protobuf" in Maven main class outputs
[INFO] PROTOC SUCCEEDED: Protoc invocation succeeded.

...but I cannot seem to see anything being output by the application into the target/generated-sources directory like would be with protoc itself.

For ref, the args passed to protobuf4j on the command line were:

--java_out=/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it/protobuf4j-java-main/target/generated-sources/protobuf
/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it/protobuf4j-java-main/src/main/protobuf/helloworld.proto
--proto_path=/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it/protobuf4j-java-main/src/main/protobuf
--proto_path=/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it/protobuf4j-java-main/target/protobuf-maven-plugin/generate/default/archives/protobuf-java-4.34.0-eb7cba770206545b5a6ae8014a76c6e6f0522f08

...and the args used to invoke the JVM itself were...

-classpath
/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it-repo/io/roastedroot/protobuf4j-v4/999-SNAPSHOT/protobuf4j-v4-999-SNAPSHOT.jar:/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it-repo/com/google/protobuf/protobuf-java/4.33.5/protobuf-java-4.33.5.jar:/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it-repo/io/roastedroot/protobuf4j-common/999-SNAPSHOT/protobuf4j-common-999-SNAPSHOT.jar:/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it-repo/com/dylibso/chicory/annotations/1.6.1/annotations-1.6.1.jar:/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it-repo/com/dylibso/chicory/wasi/1.6.1/wasi-1.6.1.jar:/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it-repo/com/dylibso/chicory/log/1.6.1/log-1.6.1.jar:/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it-repo/com/dylibso/chicory/runtime/1.6.1/runtime-1.6.1.jar:/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it-repo/com/dylibso/chicory/wasm/1.6.1/wasm-1.6.1.jar:/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it-repo/io/roastedroot/zerofs/0.1.0/zerofs-0.1.0.jar
-XX:TieredStopAtLevel=1
-XX:+CrashOnOutOfMemoryError
-XX:+TieredCompilation
-XX:+UseSerialGC
-Xms32m
-Xshare:auto
io.roastedroot.protobuf4j.v4.Main

I've pushed the code to a branch and the same test can be reproduced via ./mvnw clean integration-test -Dmaven.test.skip -Dinvoker.test=protobuf4j-java-main on my side.

Just want to check I'm not missing anything obvious here?


Edit: might've spotted something my side. Will sort that and drop a comment with any further findings!

@ascopes
Copy link
Copy Markdown
Collaborator

ascopes commented Mar 22, 2026

Okay, update on that. Was a problem on my side it seems. I've pushed a fix to the branch to deal with the new argument propagation correctly and the basic test seems to work.

[INFO] All sources will be compiled, as no previous build data was detected
[INFO] Generating source code from 1 proto source file and 0 descriptor files (discovered a total of 1 proto source file and 0 descriptor files)
[INFO] Invoking protoc. Enable debug logs (--debug --errors) for more details
[INFO] [protoc - stdout pid=81637] Listening for transport dt_socket at address: 5005
[INFO] protoc (pid 81637) returned exit code 0 (success) after 16878ms
[INFO] Embedding sources from "/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it/protobuf4j-java-main/src/main/protobuf" in Maven main class outputs
[INFO] PROTOC SUCCEEDED: Protoc invocation succeeded.
...
[INFO] --- compiler:3.15.0:compile (default-compile) @ protobuf4j-java-main ---
[INFO] Recompiling the module because of changed source code.
[INFO] Compiling 3 source files with javac [debug deprecation parameters release 17] to target/classes
[INFO] 
[INFO] --- resources:3.5.0:testResources (default-testResources) @ protobuf4j-java-main ---
[INFO] skip non existing resourceDirectory /home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it/protobuf4j-java-main/src/test/resources
[INFO] 
[INFO] --- compiler:3.15.0:testCompile (default-testCompile) @ protobuf4j-java-main ---
[INFO] Recompiling the module because of changed dependency.
[INFO] Compiling 1 source file with javac [debug deprecation parameters release 17] to target/test-classes
...
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] ---org.example.helloworld.ProtobufTest - 0.083 s
[INFO]    +-- [OK] generatedProtobufSourcesAreValid - 0.059 s
[INFO]    '-- [OK] generatedProtobufSourcesAreFullMessages - 0.006 s
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO]

So in conclusion for that, the changes needed to support that side of things would be:

  • Define static void main(String[]) entrypoint
  • Add that entrypoint to Main-Class manifest entry
  • Add an Automatic-Module-Name entry for JPMS compliance as tools may attempt to resolve module-info.class on tools to determine if they are JPMS-compatible or not; the current JAR name breaks the JDK tooling for that.

I've also tried to run it with a binary-based gRPC plugin (specifically, protoc-gen-grpc-java) to see if that integration works. Unfortunately, I'm seeing the following output when I do that:

[INFO] All sources will be compiled, as no previous build data was detected
[INFO] Generating source code from 1 proto source file and 0 descriptor files (discovered a total of 1 proto source file and 0 descriptor files)
[INFO] Invoking protoc. Enable debug logs (--debug --errors) for more details
[WARNING] [protoc - stderr pid=5857] Mar 22, 2026 11:10:52 AM io.roastedroot.protobuf4j.common.Protobuf lambda$defaultSubprocessHandler$0
[WARNING] [protoc - stderr pid=5857] WARNING: Unknown plugin: /home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it/protobuf4j-grpc-plugin/target/protobuf-maven-plugin/generate/default/artifacts/protoc-gen-grpc-java-48e6506588b050f251ff028d69f3a929891fd3e7.exe
[INFO] protoc (pid 5857) returned exit code 0 (success) after 739ms

Following this, no sources are being generated at all.

The JVM invocation looks like the following in this case:

<!-- jvm bootstrapping args start here -->
-classpath
/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it-repo/io/roastedroot/protobuf4j-v4/999-SNAPSHOT/protobuf4j-v4-999-SNAPSHOT.jar:/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it-repo/com/google/protobuf/protobuf-java/4.33.5/protobuf-java-4.33.5.jar:/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it-repo/io/roastedroot/protobuf4j-common/999-SNAPSHOT/protobuf4j-common-999-SNAPSHOT.jar:/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it-repo/com/dylibso/chicory/annotations/1.6.1/annotations-1.6.1.jar:/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it-repo/com/dylibso/chicory/wasi/1.6.1/wasi-1.6.1.jar:/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it-repo/com/dylibso/chicory/log/1.6.1/log-1.6.1.jar:/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it-repo/com/dylibso/chicory/runtime/1.6.1/runtime-1.6.1.jar:/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it-repo/com/dylibso/chicory/wasm/1.6.1/wasm-1.6.1.jar:/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it-repo/io/roastedroot/zerofs/0.1.0/zerofs-0.1.0.jar
-XX:TieredStopAtLevel=1
-XX:+CrashOnOutOfMemoryError
-XX:+TieredCompilation
-XX:+UseSerialGC
-Xms32m
-Xshare:auto
io.roastedroot.protobuf4j.v4.Main
<!-- protoc args start here -->
--java_out=/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it/protobuf4j-grpc-plugin/target/generated-sources/protobuf
--plugin=protoc-gen-0_7ae01980bc058f6bf5b3cdf8f5c37c7972fb6808=/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it/protobuf4j-grpc-plugin/target/protobuf-maven-plugin/generate/default/artifacts/protoc-gen-grpc-java-48e6506588b050f251ff028d69f3a929891fd3e7.exe
--0_7ae01980bc058f6bf5b3cdf8f5c37c7972fb6808_out=/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it/protobuf4j-grpc-plugin/target/generated-sources/protobuf
/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it/protobuf4j-grpc-plugin/src/main/protobuf/helloworld.proto
--proto_path=/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it/protobuf4j-grpc-plugin/src/main/protobuf
--proto_path=/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it/protobuf4j-grpc-plugin/target/protobuf-maven-plugin/generate/default/archives/protobuf-java-4.34.0-eb7cba770206545b5a6ae8014a76c6e6f0522f08
--proto_path=/home/ashley/code/protobuf-maven-plugin/protobuf-maven-plugin/target/it/protobuf4j-grpc-plugin/target/protobuf-maven-plugin/generate/default/archives/proto-google-common-protos-2.63.2-2595f0389e266a8c3bf176fea7fd4f62a58dcd4b

Not too sure what is going on there, guessing it is possibly ignoring the --plugin=$id=$path flag or not parsing it correctly? This invocation definitely works under regular protoc.exe though.

I've committed a test project to the branch I mentioned before that can be invoked with ./mvnw clean integration-test -Dmaven.test.skip -Dinvoker.test=protobuf4j-grpc-plugin if desired. I've also added some Maven config in src/it/protobuf4j-grpc-plugin/pom.xml that can be uncommented to allow attaching a remote JVM debugger to the protobuf4j process itself if that is of any help.

@andreaTP
Copy link
Copy Markdown
Contributor Author

Thanks for getting back @ascopes .

I agree that adding Java module definitions is a step forward in this repo, opened #64 to track it.

Having the execution failing when attempting to invoke a plugin that is not compiled into this wasm-module is a design choice, see: https://github.com/roastedroot/protobuf4j/pull/54/changes#diff-b1ca38cad24af3b96f6a0186bfa0f16b6fafdc583716ead57604719ce0aad959R118-R128

I do not want to maintain in this repo OS dependent code, nor ProcessBuilder invocations or similar as it is out of the scope, and, frankly, I think it defeats the value of this pure Java library.

The main was left out on purpose as, I think, you want to provide your own subprocess handler using the last argument:
https://github.com/roastedroot/protobuf4j/pull/54/changes#diff-d7de729dddc5ec8eafcbd1642a8c1dbcac1a5a976f5917763987a3ddafeb8448R189

All in all, probably the best path forward would be if you roll your own "main" using protobuf4j as a library and providing whatever is needed to run external, native protoc plugins.
Happy to hear your thoughts!

@ascopes
Copy link
Copy Markdown
Collaborator

ascopes commented Mar 23, 2026

I see, okay thanks. Will try and have another look when I get some more time then

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants