Support for custom metrics

[flokde]

Really enjoy working with the package! Would it be possible to implement the use of custom metrics? Some AutoML frameworks like autosklearn support this while others like h2o do not. I think it would be great to have the benchmark support custom metrics for various applications and utilise it in frameworks that in turn support this. If this is already possible (through reconfiguring the frameworks and adding a custom version e.g.), I would really appreciate to be pointed in the right direction. Thank you!

[sebhrusen]

Hey @flokde, thanks for the suggestion.
Custom metrics would require being able to pass the objective function in a declarative way, so I don't see how this would be possible.
Besides for benchmarking, we're trying to use standard metrics that are supported by almost all frameworks.

I see 2 approaches if you really need this.

1. as you suggest, have your custom version for a given framework. It's not ideal (you need to maintain your version), but you it's possible with one limitation: we can't easily build docker images with custom frameworks currently.
   see https://github.com/openml/automlbenchmark/blob/master/docs/HOWTO.md#add-a-custom-framework
   and the extensions example in https://github.com/openml/automlbenchmark/tree/master/examples/custom
2. a much better approach, we could try using custom params, with a custom framework definition:

autosklearn_custom:
    extends: autsklearn
    params:
        _custom_metric_def: "{user}/path/to/custom_metric.py"
        _custom_metric: "my_metric"

then in your custom_metric.py:

def my_metric(...):
     pass

then in autosklearn exec.py:

custom_metric_def= config.framework_params.get('_custom_metric_def')
if custom_metric_def:
    custom_metric_name = config.framework_params.get('_custom_metric', 'metric')
    custom_metrics = {}
    exec(open(custom_metric_def).read(), custom_metrics)
    custom_metric = custom_metrics[custom_metric_name]

now the custom metric is loaded in the environment running the framework, so this should work, besides it's not invasive to integrate, more or less 6 lines above, and it will work locally, in docker, and on aws, as soon as framework_params resolves the {user} placeholder (it doesn't currently, but I can easily fix this).

Otherwise, without the placeholder, you can always hardcode the path:

        _custom_metric_def: "/path/to/custom_metric.py"

but then it will only work locally.

If you're interested in implementing (2), please go on, otherwise, you'll have to wait until I can find time for this.

[sebhrusen]

I think I'll come up with a more generic way to provide custom python extensions for the frameworks integration: custom metrics will just be able to use this mechanism

[sebhrusen]

Hi @flokde , please see the PR above: #141 and tell me if it suits your needs.

[flokde]

Hey Seb!
Thanks so much for your response and taking the time! I tried your method form before and ended up just importing and setting the metric manually. Your proposed new solution in the PR does pretty much that, but more elegant and not as invasive. I think it is a good addition - the benchmark has potential outside of the research community and it is interesting to see how the frameworks deal with "real" datasets and use cases. So I really do think it is a good addition - not just for myself :).
Thanks again and all the best!
Florian