feat: resume interrupted dataset generation runs (sync + async engine)#526

Open

przemekboruta wants to merge 33 commits intoNVIDIA-NeMo:mainfrom

przemekboruta:main

Commits on May 4, 2026

docs: add implementation plan for resume mechanism
przemekboruta
committed
feat(storage): add resume flag and clear_partial_results()
przemekboruta
committed
feat(batch-manager): add start_batch param to start()
przemekboruta
committed
feat(builder): implement resume logic in DatasetBuilder
przemekboruta
committed
feat(interface): expose resume on DataDesigner.create()
przemekboruta
committed
test: add tests for resume mechanism
przemekboruta
committed
feat(builder): extend resume to async engine (DATA_DESIGNER_ASYNC_ENGINE=1)
przemekboruta
committed
fix(builder): skip after-generation processors when resume finds dataset already complete
przemekboruta
committed
fix(builder): use filesystem count for initial_total_num_batches on async resume
przemekboruta
committed
feat(results): add export() method and --output-format CLI flag
przemekboruta
committed
fix(builder): handle resume when metadata.json missing (interrupted before first batch)
przemekboruta
committed
docs(interface): fix resume docstring — async engine is supported
przemekboruta
committed
fix(builder): derive initial_actual_num_records from filesystem in async resume
przemekboruta
committed
feat(resume): replace resume: bool with ResumeMode enum (NEVER/ALWAYS/IF_POSSIBLE)
przemekboruta
committed
fix(resume): invalidate resolved_dataset_name cache when IF_POSSIBLE downgrades to NEVER
przemekboruta
committed
fix(builder): move partial-completion warning before return in _build_async
przemekboruta
committed
fix(builder): IF_POSSIBLE now starts fresh when no dataset directory exists
przemekboruta
committed
fix(builder): use original target_num_records in async resume record count
przemekboruta
committed
fix(builder): IF_POSSIBLE starts fresh on empty dataset directory
przemekboruta
committed
fix(builder): ALWAYS raises DatasetGenerationError on config fingerprint mismatch
przemekboruta
committed

Commits on May 5, 2026

Merge branch 'main' into main
nabinchha
authored

Commits on May 7, 2026