Feature request: An option to add user-specified duration to the `!test` commands

[manodeep]

For example, the current !test repro runs for 1-day, which might be too short to uncover a rare issue. It would be convenient if there was an option to specify the duration of the repro test as well (with the default/no-option being the current 1-day). Of course, such a feature would mean that both the base and the PR'ed branch need to be run (instead of using the committed checksums from the base branch), and there might need to be some reasonable lower/upper limits set for the runs for repro tests.

May be other commands might also benefit from a generic solution to parse user-supplied-options.

@CodeGat

[CodeGat]

Thanks for the write up @manodeep - there are a few things to consider in there. I'll let you know when I'm on this issue

[manodeep]

Another related option could be a special command to test performance with and without the changes in a PR for a model config repo. For example, !test perf could run, say a 1-month period and report back any significant performance changes.