Add Permutation Importance

[mayer79]

Implements #201

[mayer79]

This is the current basic call:

import numpy as np
import polars as pl
from sklearn.linear_model import LinearRegression

from model_diagnostics.xai import plot_permutation_importance

rng = np.random.default_rng(1)
n = 1000

X = pl.DataFrame(
    {
        "area": rng.uniform(30, 120, n),
        "rooms": rng.choice([2.5, 3.5, 4.5], n),
        "age": rng.uniform(0, 100, n),
    }
)

y = X["area"] + 20 * X["rooms"] + rng.normal(0, 1, n)

model = LinearRegression()
model.fit(X, y)

_ = plot_permutation_importance(
    predict_function=model.predict,
    X=X,
    y=y,
)

The extended feature API allows to permute groups like this:

_ = plot_permutation_importance(
    predict_function=model.predict,
    features={"size": ["area", "rooms"], "age": "age"},
    X=X,
    y=y,
)

[mayer79]

2 important points:

• Merge with main and resolve merge conflicts. No updates of package versions should be needed.

-> Done

• Is it possible to construct a simple test where we know the answer (=importance values) and check against that answer?

-> Ideally, we can compare with Scikit-Learn's implementation. But I am still working on this

[lorentzenchr]

I think only the np.split point is open. If this is merged, an example or addition to an existing one would be nice.

[mayer79]

I think only the np.split point is open. If this is merged, an example or addition to an existing one would be nice.

The condition now is like this:

        # np.split() does not work on pyarrow arrays and should not be used on Pandas
        if (
            is_pyarrow_array(predictions)
            or is_pandas_df(predictions)
            or is_pandas_series(predictions)
        ):
            predictions = predictions.to_numpy()

Open issues:

1. comparison with another implementation. I was unable to exactly reproduce with Scikit-Learn, even with a single feature / single shuffle situation.
2. Test failures in calling the scorer pre shuffle

[lorentzenchr]

1. comparison with another implementation. I was unable to exactly reproduce with Scikit-Learn, even with a single feature / single shuffle situation.

I guess we can live without it. Do you have any clue as to where this difference might stem from?

2. Test failures in calling the scorer pre shuffle

CI/CD must be green before merge.

[mayer79]

1. comparison with another implementation. I was unable to exactly reproduce with Scikit-Learn, even with a single feature / single shuffle situation.

I guess we can live without it. Do you have any clue as to where this difference might stem from?

I was unable to bring the random number generator to the same state.

2. Test failures in calling the scorer pre shuffle

CI/CD must be green before merge.

test.py3.12 (locally) gives different failtures than I see on github. Will look into it as soon as I have enough time.

[lorentzenchr]

@mayer79 ready to merge?

[lorentzenchr]

🎉