Writing a script that reads the logs and produces a set of ozone freon commands to reproduce the activity

[rnblough]

Unsurprisingly this is for support and testing purposes. What I really hope to be able to accomplish is reproducing problems from clusters too huge to be replicated on a much smaller cluster, because they will both have three OMs/SCMs, and freon can specify updates with sufficient granularity.

First pass at some problems needing solving:

• Problem: all operations may not leave log traces at INFO level. Capturing DEBUG logs is hard in huge clusters.
  • Sub-problem: Can we identify the list of operations only logged at DEBUG or higher, and then use daemonlog to escalate only those classes?
• Problem: We want the processes to be uniquely identifiable through the logs. Is this true of all processes, at least at the DEBUG level?
  • Workaround: If we have distinct START messages for a process, those can just be counted.
• Problem: We want the logs from the leaders. More logs go here and they are authoritative.
• Problem: Log gaps. JVM pauses prevent logging, for example.
• Sanity check: Are there any ozone freon operations with no identifiable logging on the ozone side, at any level? (Assume no, but just in case).
• Test approach: build cluster, test each ozone freon command, identify the corresponding log traces across components. This will answer the question of what is available at each log level.

[errose28]

Hi @rnblough if I'm understanding this correctly, you would like to be able to replay a set of commands that occurred on one cluster onto a different test cluster using freon. The simplest thing seems like making a freon command that takes an OM audit log, which has every read and write request that was executed, and translating this into a set of operations to perform on the other cluster. There would be a few things to note if considering this:

• Many issues in real clusters happen due to timing of multiple events and failures. Simply replaying operations will not capture this.
  • Based on this, I'm not sure how useful such a tool would be, but correct me if my understanding is wrong.
• The OM currently supports ~100 different types of requests, and reverse mapping each type's audit log line back to RPC calls will be a large effort in the beginning, and difficult to maintain going forward.
• Audit logs contain no information about the data written. Freon could write random data if we are only trying to test metadata.

[rnblough]

Pretty close!

My initial plan was to use some kind of pattern recognition over the logs, and then compose the freon commands based on that, using something like bash commands or awk.

I was hoping to use the logs as a source in order to help capture the timing of multiple events problem, since we'd have the timestamps. As a practical matter I wouldn't be able to duplicate the timing exactly, but I we might be able to break the logs into chunks and issue the commands according to a schedule.

My thoughts for the failure issue was that when there is a one, we might be able to simulate some of the features through freon, like the reactive message traffic, but really if we can at least simulate the traffic we can independently take a look at simulating whatever the failure is independently, in the time-honored tradition of sigkill or forced unmounting of disks, etc.

Mostly I am thinking in terms of metadata, because I reason that if the type of data is the crucial detail we can tackle that directly.

Only if this setup gives me any kind of interesting result (or notable gap) would I then look at feeding it back into freon as an actual improvement.

[emshvac]

I think freon can be used as an observability tool as well as a repair tool.