Allow custom metrics to be reported in results

[PGijsbers]

As far as I could tell, the custom metric could be used for optimization, but was never reported.
I set up a separate branch because I wasn't sure if this was correct (maybe I misinterpreted the documentation on the pr).

In general, we can expect AutoML frameworks to expect different signatures for their metric function. For example, TPOT requires a scikit-learn scorer:

from sklearn.metrics import make_scorer

# Make a custom metric function
def my_custom_accuracy(y_true, y_pred):
    y_true = y_true.ravel()  # amlb provides a (N,1)-vector but we need (N,)
    y_pred = y_pred.ravel()  # probably not required
    score = float(sum(y_pred == y_true)) / len(y_true)
    return score

# Make a custom a scorer from the custom metric function
foo = make_scorer(my_custom_accuracy, greater_is_better=True)

However we want a uniform signature for our own score calculations. So in addition to defining the custom metric for the automl framework(s), one needs to be defined for the automl benchmark. This be a function with the same name as the metric, but with a trailing underscore. The signature is Callable[[amlb.results.Result], float]:

def foo_(result):
    return my_custom_accuracy(result.truth, result.predictions)

This provides the user with result.truth, result.predictions and, if applicable, result.probabilities.

[PGijsbers]

This PR should be merged (or rejected) before #141.

[sebhrusen]

There's a difficulty here. If we take TPOT as an example, then the training and the result processing are not done on the same process/venv, this means that if the tpot_extensions.py has a dependency on tpot module, then we will be able to load the extension in the training process, but it will fail in the one processing results...

The lookup logic suggested in #141 is then probably too simple.
Maybe we need something like:

extensions_files: 
   autosklearn:  '{user}/autosklearn_extensions.py'
   autogluon:  '{user}/autogluon_extensions.py'
   tpot: ['{user}/tpot_extensions.py', '{user}/sklearn_extensions.py']
   default:  '{user}/extensions.py'

Now let's imagine that the tpot_extensions.py can be only be loaded in the tpot venv. Then, we can still provide extensions for the results that only require sklearn: to report the custom metrics in the results, the requirement is that it relies only on sklearn, numpy, pandas, scipy..., that is the dependencies available in the main process.

I think you can still merge this PR, I will probably consider this to improve #141 though.

[sebhrusen]

Nice addition, thx