Clarify Dead letter queue messages reprocessing process

[acharviakou]

Hello

According to documentation fluentbit save rejected chunks to filesystem in internal chunks format

In my specific use case data in general rejected by opensearch output due to mapping conflict issues.
Also I found multiple related issues:

• ElasticSearch output: dead letter index #3924
• Catch Elasticsearch errors #3062
• Improve DLQ usability by labeling failed records within rejected chunks #11425

I have some open questions regarding DLQ:

1. Chunks saved to local file system in custom format.
   • is there any ready tool to read such '.flb' files and understand format? Probably chunkio , but there is not ready binary. I would like to avoid option with custom code, because format looks like internal fluentbit implementation and can be changed in future
   • can fluentbit read and understand '.flb' files via some input? I tried blob input and s3 output to upload '.flb' files to object storage. Some data uploaded to s3, but not all for sure(I see more files on filesystem than on s3; probably something miss configured). Could you please suggest better options?

Could you please assist?

Thanks in advance

[patrick-stephens]

Probably more for the docs repo so tagging @eschabell but @lecaros is the person I'd ask for the technical details.

[lecaros]

is there any ready tool to read such '.flb' files and understand format? Probably chunkio , but there is not ready binary. I would like to avoid option with custom code, because format looks like internal fluentbit implementation and can be changed in future

Yes, you can use https://github.com/chronosphereio/calyptia-chunk-inspector

can fluentbit read and understand '.flb' files via some input? I tried blob input and s3 output to upload '.flb' files to object storage. Some data uploaded to s3, but not all for sure(I see more files on filesystem than on s3; probably something miss configured). Could you please suggest better options?

You can put them in the storage folder of the same input plugin it was created. Fluent Bit will transparently try to deliver it again. However, if it's there it will need some sort of correction to be successfully sent.