Update submission directory to align with new rules.#12
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
| for (system_id, model), runs in groups.items(): | ||
| all_system_runs = runs_by_system[system_id] | ||
| if system_id not in written_systems: | ||
| max_concurrency = max(_extract_concurrency(r["config"]) for r in all_system_runs) |
There was a problem hiding this comment.
This is incorrect. The max concurrency is itself a field within the submission structure. I failed to notice that before, but I'll make that change when we get clarification.
| if _normalize_division(division) == "Standardized": | ||
| (submission_dir / "src").mkdir(exist_ok=True) | ||
| _write_run_entries(submission_dir, system_id, model, runs, max_concurrency) | ||
| _write_model_sweep_stubs(submission_dir, system_id, model) |
There was a problem hiding this comment.
This function definitely shouldn't exist. We'll need to confirm what the csv files are, and generate them on the fly, probably.
| json.dumps(run["result_summary"], indent=2), encoding="utf-8" | ||
| ) | ||
| (result_dir / "mlperf_endpoints_log_detail.json").write_text( | ||
| (run_dir / "mlperf_endpoints_log_detail.json").write_text( |
There was a problem hiding this comment.
This line is also suspect. I can't believe I missed this earlier, but it's worrying that we build a submission with an empty directory here. We need to use real data.
| json.dumps( | ||
| { | ||
| "serving_framework": run["system_info"].get("framework", ""), | ||
| "parallelism": {}, |
| src_dir.mkdir(parents=True, exist_ok=True) | ||
| (src_dir / ".gitkeep").write_text("", encoding="utf-8") | ||
|
|
||
| # Per-run accuracy placeholder |
There was a problem hiding this comment.
Same here. This should be sourced from our database. If that isn't the case, we need to change things to match.
|
|
||
|
|
||
| def _write_accuracy_placeholders( | ||
| def _write_model_sweep_stubs( |
There was a problem hiding this comment.
Noting this again - we'll need to change this to an actual csv creator.
| ] | ||
|
|
||
| _DEFAULT_BASE_URL = "http://localhost:8080" | ||
| _DEFAULT_BASE_URL = "https://mlperf-endpoints-api-50577619532.us-central1.run.app" |
There was a problem hiding this comment.
Note to self: See if there's a better way to store this instead of in the source code. I don't think there will be, but better to check.
Pulling new changes in.
1 participant
Do not merge - this is just for internal review.