Track superseded mempool errors separately

[tilacog]

Problem definition

When Mempools::execute() runs mempools concurrently, errors from mempools whose results were discarded after another mempool succeeded were still recorded against driver_mempool_submission, biasing the per-mempool success ratio with timing-dependent shadowed failures.

Major change

Replace select_ok with FuturesUnordered + a manual loop so observation runs in the consuming context. Errors from mempools overtaken by a later success are recorded under a new Superseded label via observe::mempool_superseded, which also records the winning mempool in the trace fields. Errors in the all-failed case keep their existing labels (Revert / Expired / Other / Disabled).

Minor change

observe::mempool_executed is also split into mempool_succeeded(&SubmissionSuccess) and mempool_failed(&mempools::Error), dropping the Result<&S, &E> indirection now that each call site already knows which branch it is on. Behavior and emitted metrics are unchanged by the split.

Alert query update needed when deploying

sum by (network) (increase(driver_mempool_submission{cow_fi_environment="prod",result="Success"}[2h]))
/
sum by (network) (increase(driver_mempool_submission{cow_fi_environment="prod",result!~"Disabled|Superseded"}[2h])) < 0.6

[fleupold]

Is there a reason you are not using the PR template for the description?

I agree with the change, however I'd like to suggest that we interpret "superseeded" events as success wrt. how you envision to change the metric. A superseeded submission should be considered a successful one.

This way we receive N (# of mempool) events in the happy case, and N events in the failure case allowing us to keep our alert metric as a ratio of successful to failed ones (otherwise failed events would be weighted N times more than successful ones).