Describe the bug
When OpenSearch cluster is overloaded, DataPrepper instead of generating the warning and waiting exponentially, attempting to deliver the current payload in the infinite loop, which causing more OpenSearch overload.
The log records have also influence to the overload in case, if the target OpenSearch cluster is using for storing logs as well.
To Reproduce
Steps to reproduce the behavior:
Lower OpenSearch cluster to best possible minimum and start sending huge amount of data from DataPrepper (OTLP metrics and traces) and also have some log processor active to feeding the same OpenSearch cluster (Logstash, FluentD, ...).
Expected behavior
When DataPrepper got reject from OpenSearch, it should waiting some time, before attempting to retrying delivery.
Screenshots
Environment (please complete the following information):
- Kubernetes: 1.34
- Version DataPrepper (official Docker image): 2.7.0
- Version OpenSearch (official Docker image): 2.16
Additional context
As you can see from screenshot, the timestamp of the log records are basically same, so DataPrepper not attempting to wait and only crashing to overloaded OpenSearch cluster.
Describe the bug
When OpenSearch cluster is overloaded, DataPrepper instead of generating the warning and waiting exponentially, attempting to deliver the current payload in the infinite loop, which causing more OpenSearch overload.
The log records have also influence to the overload in case, if the target OpenSearch cluster is using for storing logs as well.
To Reproduce
Steps to reproduce the behavior:
Lower OpenSearch cluster to best possible minimum and start sending huge amount of data from DataPrepper (OTLP metrics and traces) and also have some log processor active to feeding the same OpenSearch cluster (Logstash, FluentD, ...).
Expected behavior
When DataPrepper got reject from OpenSearch, it should waiting some time, before attempting to retrying delivery.
Screenshots
Environment (please complete the following information):
Additional context
As you can see from screenshot, the timestamp of the log records are basically same, so DataPrepper not attempting to wait and only crashing to overloaded OpenSearch cluster.