Runner drops model outputs for simulation-only jobs (no targets)

[daniel-klein]

Problem

When a study file is submitted with targets: null (simulation-only, no target evaluation), the runner completes successfully but does not persist model outputs to blob storage. The outputs exist in Dask worker memory during execution but are never written to parquet.

The runner code at src/modelops/runners/job_runner.py:339 has a literal TODO:

elif not target_entrypoints:
    logger.warning("Skipping view generation: write_job_view requires aggregated results with targets")
    logger.info("TODO: Implement write_sim_results_view() for jobs without targets")

Impact

This blocks the klebsim incidence sweep analysis, which uses a single-site model with no targets — the baseline Sobol sweep runs 1000 simulations successfully, but the kp_counts.parquet output is not downloadable because it was never written to storage.

Any workflow that runs simulations without target evaluation is affected.

Fix

The default_results list contains all SimReturn objects with their .outputs dictionaries. These need to be concatenated by output name and written to blob storage at jobs/{job_id}/model_outputs/{output_name}.parquet, mirroring the path structure used by jobs with targets.

A fix is implemented on branch fix/runner-no-target-outputs (PR forthcoming).

Workaround

Register a dummy target that always succeeds, forcing the runner through the existing write path. Not ideal since it wastes compute on unnecessary target evaluation.

[daniel-klein]

Investigation Update

The initial fix (writing outputs from default_results) doesn't work because SimReturn objects have empty outputs dicts by the time they reach the runner.

What we found:

• _create_sim_return() correctly populates outputs with TableArtifact(inline=bytes) ✓
• put_sim() stores artifacts to local cache and Azure ✓
• _worker_run_task() returns SimReturn with outputs on the worker ✓
• But gather_sims() (submitted as a separate Dask task depending on sim futures) receives SimReturn objects with outputs: {} ✗

Debug output from runner logs:

Result type: SimReturn
Result attrs: ['cached', 'error', 'error_details', 'logs_ref', 'metrics', 'outputs', 'task_id']
outputs type: dict, len=0

Hypothesis:

The outputs field is populated on the original Dask worker but gets stripped/emptied during Dask's inter-worker serialization when the gather_sims task transfers the SimReturn objects to a different worker. This could be:

1. Dask's serialization dropping bytes fields in dataclass frozen mappings
2. Some interaction with the provenance cache modifying the object after creation
3. The gather_sims receiving "reference" futures rather than materialized values

Current workaround options:

1. Read from provenance cache directory (/tmp/modelops/provenance) on the runner — hacky, depends on runner/worker co-location
2. Read from Azure blob storage directly (provenance put_sim uploads there) — cleaner but adds Azure dependency in the runner output path
3. Skip the gather_sims entirely and iterate sim_futures individually on the runner side instead of via a Dask task

Option 3 seems cleanest but requires restructuring the no-target gather path.

Branch status:

• fix/runner-no-target-outputs has the initial attempt + debug logging
• Needs further investigation into Dask serialization of SimReturn

[daniel-klein]

Root Cause Found

The SimReturn outputs are empty because the bundle's pyproject.toml causes venv install failures on the workers.

Worker logs show:

error: Failed to fetch: `git+https://github.com/starsim/`
OSError: Readme file does not exist: README.md
metadata-generation-failed
RuntimeError: Install failed: .../pip install .../bundles/4fbc835583e3

The sims still ran (falling back to a cached venv), but the model outputs weren't properly extracted into the SimReturn, resulting in outputs: {}.

This is NOT a Dask serialization issue or a provenance cache bug. It's a broken pyproject.toml in the bundle causing a degraded execution environment where output extraction fails silently.

Fix

Fix the pyproject.toml to have valid git URLs and remove the README.md reference, then resubmit. The runner output-writing code (issue #31 fix) should work once the bundle is clean.