Spike/Investigate flight recorder as an option to help debug and identify issues

[maleck13]

What

From the blog :

A program often knows when something has gone wrong, but the root cause may have happened long ago. The flight recorder lets you collect a trace of the last few seconds of execution leading up to the moment a program detects there’s been a problem.

The flight recorder collects the execution trace as normal, but instead of writing it out to a socket or a file, it buffers the last few seconds of the trace in memory. At any point, the program can request the contents of the buffer and snapshot exactly the problematic window of time. The flight recorder is like a scalpel cutting directly to the problem area.

So this issue is to explore using and accessing flight recorder information. Use case to explore might be

• N number of reconciles has failed start recording