Add e2e tests for Apache Pinot connector

[mmodzelewski]

Currently the Apache Pinot connector has no proper Java integration tests. There is an existing bash script (external-processors/iggy-connector-pinot/integration-test.sh) that performs manual integration testing, but this should be replaced with proper Java tests that can run in CI.

What needs testing

• End-to-end ingestion flow: send messages to Iggy → verify data lands in Pinot realtime table
• JSON payload parsing and schema mapping
• Bulk message ingestion

Implementation notes

• Use Testcontainers for Pinot (controller, broker, server) and Iggy
• Use JUnit 6 as the test engine
• Use AssertJ for assertions
• Consider using @Testcontainers annotation for lifecycle management
• Schema and table configs can be reused from deployment/ directory
• Remove integration-test.sh once proper tests are in place

Files to reference

• foreign/java/external-processors/iggy-connector-pinot/integration-test.sh - current manual test flow (to be replaced)
• foreign/java/external-processors/iggy-connector-pinot/deployment/schema.json - Pinot schema
• foreign/java/external-processors/iggy-connector-pinot/deployment/table.json - Pinot table config
• foreign/java/external-processors/iggy-connector-pinot/src/main/java/ - connector implementation

Acceptance criteria

• At least 2-3 tests covering basic ingestion and JSON parsing
• Tests use Testcontainers (no manual Docker setup required)
• Tests run in CI via Gradle
• Test failures are informative with clear error messages
• Remove integration-test.sh after Java tests provide equivalent coverage

[Chiicake]

Hi @mmodzelewski , I’d like to take this issue—could you please assign it to me?

[Chiicake]

Hi @mmodzelewski ,

I’m trying to run the integration test script:

foreign/java/external-processors/iggy-connector-pinot/integration-test.sh

The ingestion fails and the Pinot Server logs show:

Caused by: java.lang.AbstractMethodError: 
Receiver class org.apache.iggy.connector.pinot.metadata.IggyStreamMetadataProvider does not 
define or inherit an implementation of the resolved method 'abstract boolean supportsOffsetLag()' 
of interface org.apache.pinot.spi.stream.StreamMetadataProvider.

I think this may suggest a Pinot SPI incompatibility between the connector (pinot-spi 1.4.0 in the repo) and the Pinot image in foreign/java/external-processors/iggy-connector-pinot/docker-compose.yml (apachepinot/pinot:latest, currently it's 1.5.0).

When I pinned Pinot to 1.4.0 in docker-compose.yml

or when I add

public boolean supportsOffsetLag() {
    return false;
}

to foreign/java/external-processors/iggy-connector-pinot/src/main/java/org/apache/iggy/connector/pinot/metadata/IggyStreamMetadataProvider.java, the integration test passed.

Should I

1. pin Pinot to 1.4.0 for now
2. or add supportsOffsetLag() and return false for now
3. or open an issue to support the latest Pinot SPI
4. is there any other recommended approach?

Thanks!

[mmodzelewski]

Hi @Chiicake, as the 1.5.0 is still not released, please pin the 1.4.0 version in the tests. Thanks!

[hubcio]

@Chiicake could you please kindly verify if this issue is still present? we had a release recently

[Chiicake]

@Chiicake could you please kindly verify if this issue is still present? we had a release recently

Yes, the shell-script integration test has not been replaced by Java tests yet.

I had implemented this before, but after syncing my fork with the main Iggy repository, I was unable to run either the Java integration tests or the shell-script test again. I’m currently looking into the cause and trying to identify where the problem comes from.

Sorry about the slow progress. I’m currently working on my graduation thesis, so I have a bit less free time available recently.

[hubcio]

no problem at all - no one is expecting anything from you :) thanks for the contributions.

@mmodzelewski could you please check this again?

[mmodzelewski]

@Chiicake please ping me if you can't resolve the issue. You can also join our discord and we can discuss there. If you're running java integration tests with Testcontainers, make sure that it fetches the latest iggy image.