Enhancement/faster backtest

[janrth]

Optimized backtest_splits for already sorted data.

When running on large data sets we observed running into memory issues when running backtest functionality (inside mlforecast). This tried to remove some memory overhead and at the same time created speed inreases.

Instead of creating full-length per-row date arrays and boolean masks for every CV window, the new code computes train/validation boundaries per series and slices rows by index ranges. That reduces peak memory and speeds up splitting on large panels. Unsorted inputs still use the old implementation, so existing behavior stays unchanged there.

Performance test showed really good performance compared to old logic.

Old logic took 28 sec and had 5781 MB in peak mem usage, while the suggested logic used finished in 17 sec and only used 3128 MB in peak mem usage on 50 mill rows in the code below:

import time
import tracemalloc

import pandas as pd

from utilsforecast.data import generate_series
from utilsforecast.processing import backtest_splits

n_series = 100_000
min_length = 400
max_length = 600
n_windows = 5
h = 28

df = generate_series(
    n_series=n_series,
    freq="D",
    min_length=min_length,
    max_length=max_length,
    equal_ends=True,
)
print(f"rows={len(df):,} series={df['unique_id'].nunique():,}")

tracemalloc.start()
start = time.perf_counter()

fold_stats = []
for window, (cutoffs, train, valid) in enumerate(
    backtest_splits(
        df,
        n_windows=n_windows,
        h=h,
        id_col="unique_id",
        time_col="ds",
        freq=pd.offsets.Day(),
    )
):
    fold_stats.append(
        {
            "window": window,
            "n_cutoffs": len(cutoffs),
            "train_rows": len(train),
            "valid_rows": len(valid),
            "train_min_ds": train["ds"].min(),
            "train_max_ds": train["ds"].max(),
            "valid_min_ds": valid["ds"].min(),
            "valid_max_ds": valid["ds"].max(),
        }
    )

elapsed = time.perf_counter() - start
_, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()

summary = pd.DataFrame(fold_stats)
print(f"elapsed={elapsed:.3f}s peak_traced_memory={peak / 1e6:.1f} MB")

[janrth]

@codex

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Breezy!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".