Skip to content

Design a better way to deal with metrics for model evaluation (for the drift detector) #56

Description

@anagainaru

The evaluation metrics are defined here:

examples/mnist/model.py:        self.eval_metrics = [accuracy, self.get_criterion()]

The required values for them are defined in src/main.py:

        drift_signal = detector.update(
            value=0.0,  # dummy value for base class compatibility
            modelHarness=modelHarness,
            reference_validation_metrics=[90, 1.0],
            higher_is_better=[True, False],
        )

These metrics/values are used to evaluate the performance of a model in src/drift_detection/detectors/model_performance_detector.py
This is convoluted and will be hard to understand/debug without proper documentation. I propose we have a list of metrics we support and each example chooses a subset and defines the desired values in their toml files

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions