Skip to content

Fix evaluation split config in lsat_qa #1148

@pjavanrood

Description

@pjavanrood

Describe the bug

The lsat_qa tasks were configured with evaluation_splits = ["validation", "test"], but lighteval/lsat_qa only has test split

To Reproduce

task = "lsat_qa|5"

pipeline = Pipeline(
    tasks=task,
    pipeline_parameters=pipeline_params,
    evaluation_tracker=evaluation_tracker,
    model_config=model_config,
)

pipeline.evaluate()
pipeline.save_and_push_results()
pipeline.show_results()```

## Expected behavior
The configuration should only reference the `test` split to ensure successful loading and evaluation.

## Version info
- OS: mac
- Lighteval version: main (local development)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions