Multi-Step Training Predictions

[samuel-gerstein]

Description

Currently when you use MLForecast.forecast_fitted_values() it only can produce one-step ahead predictions on the training set without the ability to customize the horizon.

It would be great if the functionality used in this R example could be implemented here.

Use case

I would like to have training predictions be accurate for multi-step models. It can be quite confusing and is unclear that the horizon is different for multi-step models between training and testing.

[jwiklander]

+1 on this

The easiest would probably be the second option in the above R example, insample 1-step predictions for the validation set(s) based on the model(s) fitted on the cv training set(s)

Then you at least have some way to get some fair comparison of train/validation losses even if your forecast horizon/validation window size is something else

Would also be great to get these train/validation insample predictions also for the LightGBMCV cross validation variant

Now I first run LightGBMCV to be able to access the early stopping greatness. Then to get training loss I have to run the normal .cross_validation (with n_estimators from the early stopping) to get the training loss from the fitted values. Then re-fit the model from the cv on only the cv train set and run loops of 1-step predictions over the validation windows to get the 1-step validation loss. If there is a smarter way please let me know!