How to compare results?

[mencagli]

Thank you for your hard work with StreamBench. I downloaded and compiled the code by running the YSB with the different frameworks. I was wondering about how to compare the results, since the output format of the various versions with StreamBox, Flink, Saber and Spark is quite different. Do you use some external tool to compare the performance? Any hints to do that?

Many thanks for your help

Gabriele

[grtheod]

Thank you for the comment.

In the current version, we don't support a unified way of comparing the results and we do it manually. For example, in Flink, we collect the Throughput from the workers' logs (maybe you will be more lucky with a new version). Something similar applies to SABER and Streambox (we write the metrics in a log). What you could do, is to pick a representation (e.g. how Spark dumps results in Kafka) and create a JSON file for each one of them that pulls the relevant information from the engine's log. A consistent solution would be to extend the StreamBenchMain with extra runners that gather the data from the aforementioned logs.