Large UDP payloads can be silently dropped

[malish8632]

Due to nature of UDP protocol data could be silently dropped if reached specific limits.
As the data/metrics are dropped silently usually we alerted by teams noticing missing metrics.

One idea discussed is to put some ignorable/optional metric indicating the number of bytes or metrics sent over the wire so FFWD agent could detect the loss and emit a metric to be alerted on.

[sming]

Good suggestion. Or a checksum if we want to be really stingy with bandwidth.

[ao2017]

@malish8632 do we know that limit ?

[malish8632]

Yes the default is ~25MB. Check troubleshooting section in protobuf docs here.

[ao2017]

Well, I wanted more datapoint from teams that are experiencing the problem.I doubt anybody is sending 25MB in the packet. I think the packets drop before reaching UDP max. I noticed, but I am not certain, that anything higher than 2048 bytes is being dropped.
I will look into the issue when I get a chance.