Question/Discussion Introduce CaseManager service layer for centralized case lifecycle handling

[sanvishukla]

Type

Question

Context

While working on recent backend stability improvements, I observed that case lifecycle and storage access logic is currently distributed across multiple route files.
During that review, I ran grep -R "DATA_STORAGE" API/
This showed that Config.DATA_STORAGE is referenced across multiple route files and case-related classes, with path construction and filesystem access occurring directly within route handlers.

(Patial grep output attached as screenshot for reference.)

What is blocked?

There does not appear to be a centralized service responsible for:

• Case existence validation
• Case path resolution
• Safe case listing
• Case deletion
• Ensuring consistent session/filesystem state

Requested decision or help

I would like to confirm:

• Is there an intended architectural layer for case lifecycle management that I may have overlooked?
• Is the current design (routes directly accessing DATA_STORAGE) intentional?
• If not, would introducing a minimal CaseManager service layer be aligned with the project’s backend direction?

The goal would not be to change behavior, but simply to understand whether the backend is expected to evolve toward more centralized lifecycle handling or remain route-centric.
I want to make sure I am aligned with the intended architectural direction before proposing any structural refactor.

[NamanmeetSingh]

This is a fantastic architectural observation, @sanvishukla.

As I've been building out the asynchronous simulation backend for the OG-CLEWS integration (PR #24), I’ve noticed the same bottleneck, the current route-centric filesystem access is a bit brittle.

Right now, the ConvergingOrchestrator I'm implementing has to manually resolve Path(Config.DATA_STORAGE / casename) to read the OSeMOSYS outputs and write the OG-Core macroeconomic tables.

With this centralized CaseManager service layer (e.g., exposing a clean CaseManager.get_results_dir(casename)), it would perfectly decouple the filesystem state from the scientific execution logic. I fully support this direction, it will make plugging in the coupled/converging modules much cleaner when we take it to the next phase.

[sanvishukla]

Thanks @NamanmeetSingh
I’ll aim to keep the CaseManager minimal and focused purely on path resolution. I’d be glad if the ConvergingOrchestrator could use it once in place.
Let me know if there are specific access patterns, like- results directory, run directory, etc., that would be helpful to expose.

[NamanmeetSingh]

@sanvishukla Thanks, keeping it minimal and focused on path resolution is definitely the right play.

As of now, for the current status of Track 1 ConvergingOrchestrator, the simulation loop gets quite heavy with file I/O because it has to bounce data back and forth between two different models across multiple iterations.

If the CaseManager could expose these specific access patterns (and safely handle the os.makedirs logic if they don't exist), it would be a massive help:

1. get_clews_results_dir(casename): Path to read the raw OSeMOSYS CSV outputs (e.g., res/REF/csv/ where CapitalInvestment.csv lives).

2. get_macro_inputs_dir(casename): Path where the ETL bridge can safely write the transformed 1D time-series vectors for OG-Core.

3. get_macro_outputs_dir(casename): Path to read the macroeconomic GDP/Interest rate outputs generated by OG-Core.

4. get_iteration_state_dir(casename, iteration_number): Because the orchestrator loops until dynamic equilibrium is reached, having a method that returns a safe, sandboxed directory for a specific iteration (e.g., DataStorage/casename/iterations/iter_3/) would prevent the models from overwriting their previous states during a long run.

If the CaseManager can guarantee those paths, it will make the Track 1 execution engine completely stateless and much more robust. Really appreciate you taking the lead on this structural plumbing. Please let me know in case of any help or input.