Build GP model when some evaluations failed

[titoiride]

Hi all, I'm trying to understand how to deal with the case when some of the evaluations failed. This is partially related to #300, since I wasn't able to figure out how to interrupt the execution so that trials currently being evaluated finished execution gracefully but no new trial was spawned. In my case, I had to brute force cancel the execution and now I have a history file that includes the last 4 failed attempts. These are marked as failed and I'm able to correctly read the history file. However, whenever I call something like charge_model = diags.build_gp_model("charge", minimize=False), then a command like charge_model.evaluate_model(completed_history.iloc[0:1]) would fail to evaluate for ValueError: Non-finite data found for metric charge: nan.

Is there a way to remove failed evaluations from the history? I tried diags.delete_evaluation_dir, but this does not seem to help.

[jmlarson1]

It is certainly possible to remove the NaNs before building/evaluating the GP model. You can adjust the generator to filter the completed history to non-NaN rows for the relevant metric. Something like:

completed_history = completed_history[np.isfinite(completed_history["charge"])]

I think the relevant GP-based generator looks is AxSingleFidelityGenerator, which uses Bayesian optimization with Ax/BoTorch.

One caveat: if a whole region of parameter space returns NaNs, simply removing those failed evaluations can make the GP acquisition function want to explore that region again, because the model has not learned that it is invalid. (The acquisition functions trade off "not known" parts of parameter space and "good objective" parts. In that case, it may be better to encode failures as a constraint/penalty or steer the acquisition function away from those parts of the parameter space (That is, rather than only deleting them, you should replace the NaNs with large function values.)