Describe the bug
The lsat_qa tasks were configured with evaluation_splits = ["validation", "test"], but lighteval/lsat_qa only has test split
To Reproduce
task = "lsat_qa|5"
pipeline = Pipeline(
tasks=task,
pipeline_parameters=pipeline_params,
evaluation_tracker=evaluation_tracker,
model_config=model_config,
)
pipeline.evaluate()
pipeline.save_and_push_results()
pipeline.show_results()```
## Expected behavior
The configuration should only reference the `test` split to ensure successful loading and evaluation.
## Version info
- OS: mac
- Lighteval version: main (local development)
Describe the bug
The
lsat_qatasks were configured withevaluation_splits = ["validation", "test"], butlighteval/lsat_qaonly has test splitTo Reproduce