Describe the bug
I'm seeing issues in the opensearch source that indicate that the PIT is expiring while reading from an index.
2025-12-11T07:59:49.545 [pool-16-thread-1] ERROR org.opensearch.dataprepper.plugins.source.opensearch.worker.PitWorker - Unknown exception while processing index 'my_index':
org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [illegal_argument_exception] Point In Time id: [UJZuozGIhKWjjjWIHGayhg] doesn't exist
at org.opensearch.client.transport.aws.AwsSdk2Transport.extractAndWrapCause(AwsSdk2Transport.java:720)
at org.opensearch.client.transport.aws.AwsSdk2Transport.performRequest(AwsSdk2Transport.java:222)
at org.opensearch.client.opensearch.OpenSearchClient.search(OpenSearchClient.java:1386)
at org.opensearch.dataprepper.plugins.source.opensearch.worker.client.OpenSearchAccessor.searchWithSearchAfter(OpenSearchAccessor.java:270)
at org.opensearch.dataprepper.plugins.source.opensearch.worker.client.OpenSearchAccessor.searchWithPit(OpenSearchAccessor.java:132)
at org.opensearch.dataprepper.plugins.source.opensearch.worker.PitWorker.processIndex(PitWorker.java:189)
at org.opensearch.dataprepper.plugins.source.opensearch.worker.PitWorker.lambda$run$0(PitWorker.java:115)
at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141)
at org.opensearch.dataprepper.plugins.source.opensearch.worker.PitWorker.run(PitWorker.java:115)
at org.opensearch.dataprepper.plugins.source.opensearch.OpenSearchService.lambda$start$0(OpenSearchService.java:108)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
To Reproduce
I'm not sure how to reproduce this.
Expected behavior
I believe there should be two things:
- There seems to be a bug with how we extend the PIT timeout.
- Pipeline authors should be able to configure the PIT timeouts to prevent this.
Screenshots
N/A
Environment (please complete the following information):
Data Prepper 2.13.0
Additional context
N/A
Describe the bug
I'm seeing issues in the
opensearchsource that indicate that the PIT is expiring while reading from an index.To Reproduce
I'm not sure how to reproduce this.
Expected behavior
I believe there should be two things:
Screenshots
N/A
Environment (please complete the following information):
Data Prepper 2.13.0
Additional context
N/A