Added Aggregation Logic for 'learning_curves' Artifacts

[adibiasio]

Added Aggregation Logic for 'learning_curves' Artifacts

Summary of Changes:

This pull request introduces enhancements to the aggregate-amlb-results method of agbench. By specifying a new config flag --artifact, you can specify learning_curves artifacts to be collected during the aggregation script, while maintaining backwards compatibility with the previous aggregation of results.csv files. Note that it is expected that you have run agbench with the learning_curves artifact defined in _save_artifacts and _generate_curves set to True.

Intended Usage:

agbench aggregate-amlb-results autogluon-benchmark-metrics tabular ag_bench_learning_curves_20240802T163522 --artifact learning_curves --constraint toy

You can also aggregate across multiple artifacts at a time:

agbench aggregate-amlb-results autogluon-benchmark-metrics tabular ag_bench_learning_curves_20240802T163522 --artifact results --artifact learning_curves --constraint toy

Sample Output:

(base) ubuntu@ip-XXX-XX-XX-XX:~/autogluon-bench$ agbench aggregate-amlb-results andrew-bench-dev tabular ag_bench_learning_curves_full_20240806T084758 --constraint 48h64c --artifact learning_curves
INFO:botocore.credentials:Found credentials in environment variables.
2024-08-07 08:03:03,601 INFO worker.py:1743 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8265 
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:19<00:00, 50.67it/s]
Aggregated output saved to s3://andrew-bench-dev/aggregated/tabular/ag_bench_learning_curves_full_20240806T084758/learning_curves!

Results:

The aggregation script will recursively search all objects in the benchmark and collect learning_curves.json files across all datasets and folds included in the experiment. Afterwards, these files will be organized by their respective datasets and fold numbers:

.
├── dataset_a/
│   ├── 0/
│   │   └── learning_curves.json
│   ├── 1/
│   │   └── learning_curves.json
│   └── 2/
│       └── learning_curves.json
├── dataset_b/
│   ├── 0/
│   │   └── learning_curves.json
│   ├── 1/
│   │   └── learning_curves.json
│   └── 2/
│       └── learning_curves.json
└── ...

[prateekdesai04]

Another high-level comment would be - once the PR #98 is merged you can sync this into your changes and try running it.

I have made some changes which fixes the confusing path names that we had before so it may change your code.

Another TODO: is to add tests for this functionality to future proof the aggregate + eval scripts

[adibiasio]

Regarding the PR #98 you mentioned, I have pulled your changes and the aggregation script runs file (since the path structure that changed was intermediate as explained here).

And I will work on writing some unit tests for these changes, thanks for the suggestion!