Added structured startup validation for runtime dependencies (Issue #3)

[sanvishukla]

Summary

Added a structured startup validation for deterministic runtime prerequisites, addressing the “Add startup validation with clear errors when required binaries/paths are missing” portion of Issue #3.
Additionally, during validation testing, it was observed that Config.py unconditionally set DATA_STORAGE permissions to 0o777, which masked permission-related behavior.
This has been refined to avoid overriding environment-level permissions.

This PR introduces a pre-startup validation layer that ensures required directories and solver binaries are available before the server starts, providing clear and actionable feedback instead of runtime crashes.

• What changed:

  1. Added API/startup_validation.py
  2. Integrated validation call before server startup in API/app.py
  3. Validation now checks:
  • DATA_STORAGE existence and write permissions
  • WebAPP directory existence
  • GLPK (glpsol) availability via PATH
  • CBC availability via PATH
  • Solver executability
  • Basic version detection where possible
  • OS-specific installation hints
  4. Structured validation summary output
  5. Clean exit on failure (no traceback)
  6. Minor Config.py corrections (identified during validation testing):
  • Fixed typo in EmissionActivityRatio parameter definition
  • Removed duplicate dictionary keys causing silent overwrites
  • Refined DATA_STORAGE permission handling:
    • Removed unconditional 0o777 permission override
    • Applied environment-specific (development-only) permission adjustment
    • Prevented masking of permission validation behaviour

• Why:
  Previously, missing solver binaries or required directories could cause unclear runtime failures or crashes before meaningful feedback was shown.
  While validating write-permission behavior, it was discovered that Config.py was unconditionally modifying directory permissions at import time. This masked expected validation outcomes and could override externally configured permissions. The logic has been adjusted to ensure predictable behavior while maintaining development usability.
  This change ensures early, deterministic cross-platform failure detection without modifying existing runtime or path resolution logic.

Related issues

• Issue exists and is linked
• Closes #
• Related Fix platform-dependent paths and runtime assumptions for cross-platform support #3

Validation

• Tests added/updated (or not applicable)
• Validation steps documented
• Evidence attached (logs/screenshots/output as relevant)

Validation Steps:

1. Normal startup (all dependencies installed)

   • From repository root:

     python API/app.py

   • Expected:
     • Structured startup validation summary
     • All checks report OK
     • Server starts normally
   

2. Run from a different working directory

   • From inside API/:

     cd API
     python app.py

   • Expected:
     • A validation output mentioning the error to run from the correct directory
       Note: Running the application from a non-root working directory may still trigger existing path-resolution errors in Config.py. This is part of the broader path refactor tracked under Issue Fix platform-dependent paths and runtime assumptions for cross-platform support #3 and is not modified in this PR.
   

3. Missing solver simulation

   • Temporarily clear PATH (macOS/Linux):

     PATH="" .venv/bin/python API/app.py

   • Expected:
     • GLPK and/or CBC reported as FAIL
     • Clear OS-specific installation hint
     • Clean exit (no traceback)
   

4. Missing WebAPP directory simulation

   • Temporarily rename directory:

     mv WebAPP WebAPP_backup
     python API/app.py

   • Expected:
     • WebAPP directory reported as FAIL
     • Structured summary printed
     • Clean exit
   • Restore directory after test:

     mv WebAPP_backup WebAPP

   

Permission behaviour validation

1. Development environment:

   export ENV=development
   chmod 555 WebAPP/DataStorage
   python API/app.py
   

   Expected:

   • Folder permissions adjusted to 755
   • Validation succeeds
   • Application starts normally
   

2. Production/default environment:

   unset ENV
   chmod 555 WebAPP/DataStorage
   python API/app.py
   

   Expected:

   • Permissions not overridden
   • Validation correctly reports write-permission failure (if applicable)
   • No silent permission modification occurs
   

3. Change path of DataStorage:
   

Documentation

• Docs updated in this PR (or not applicable)
• Any setup/workflow changes reflected in repo docs

No external documentation changes required — internal validation layer only

Scope check

• No unrelated refactors
• Implemented from a feature branch
• Change is deliverable without upstream OSeMOSYS/MUIO dependency
• Base repo/branch is EAPD-DRB/MUIOGO:main (not upstream)

[NamanmeetSingh]

Hi, this is a great initiative! Failing fast before a long scenario run is absolutely critical for the user experience, especially given how heavy these models are. I was reviewing the validation logic and noticed a architectural clash with how the core app currently resolves solver paths, plus a quick thought on extensibility for the upcoming OG-Core integration:

1. System PATH vs Local SOLVERs_FOLDER (Startup Blocker):

Currently, _check_binary uses shutil.which(), which strictly scans the system's global PATH. However, if you look at Classes/Case/OsemosysClass.py, the application historically resolves solvers locally using Config.SOLVERs_FOLDER (e.g., Path(Config.SOLVERs_FOLDER, 'GLPK')) without requiring them to be in the global PATH.

If a user relies on these bundled local solvers, this validation script will trigger a false positive and crash the app on startup. We should probably update the validation to check inside Config.SOLVERs_FOLDER first, and only fall back to shutil.which() if they aren't found locally.

2. Extensibility for OG-Core:

As we start wiring up the OG-Core integration (Phase 2), we will need to validate its specific dependencies as well. Leaving the _check_binary function generalized like you have is perfect as it will make it very easy for us to inject the macroeconomic prerequisites into this array later!

Really solid foundation here, just want to make sure it doesn't accidentally lock out users running the localized solver setup!

[sanvishukla]

Thanks @NamanmeetSingh for pointing out that the initial implementation only relied on shutil.which(), which could incorrectly fail for users relying on bundled/local solvers resolved via Config.SOLVERs_FOLDER. In this repository, Config.SOLVERs_FOLDER points to WebAPP/SOLVERs, so I’ve updated _check_binary() to align with the existing runtime behavior:

• First checks inside WebAPP/SOLVERs
• Falls back to shutil.which() (system PATH)
• Preserves version detection logic
• Only fails if neither location resolves the binary

I validated the following scenarios:

• Solver present only in WebAPP/SOLVERs → correctly detected locally
• Solver present only in system PATH → correctly detected via shutil.which()
• Solver absent in both → validation fails fast with install guidance

This ensures startup validation mirrors the actual solver resolution strategy and avoids false positives for users relying on localized solver setups.

And you are absolutely right about your 2nd point! The _check_binary() helper remains generalized so additional dependency checks can be added cleanly without structural changes to the validation framework. Appreciate you calling that out, that extensibility was intentional!

[NamanmeetSingh]

Looks perfect @sanvishukla. Checking WebAPP/SOLVERs first is the right move to keep the bundled setups running smoothly. Great turnaround on this!

[SeaCelo]

@sanvishukla @NamanmeetSingh

Thanks for this contribution and for the thoughtful startup-validation work. We really appreciate it.

Quick update on merge sequencing: we are about to merge another set of mac-port PRs first (#31, #32, #35, #40, #44, #46, #11). Some of those overlap with parts of this PR (especially Config/path handling and solver-resolution behavior), so we want to avoid introducing conflicts/regressions in this pass.

Could you please take another look after those merges land in mac-port, and then update this PR against the latest mac-port branch? If you can align the startup checks with the merged path/solver logic, we would be happy to review again quickly.

Thanks again for your work and patience on this.

I converted this to draft to make it easier for us to track

[NamanmeetSingh]

Makes perfect sense, @SeaCelo. Staging the mac-port merges first is definitely the safest move to avoid dependency collisions.

@sanvishukla Let me know if you need any help or a second pair of eyes on the solver resolution logic on the mac-port branch.

[sanvishukla]

@SeaCelo @NamanmeetSingh,
I checked using the Mac-port branch using the same validation steps I followed in the PR and the comment above. I am attaching the screenshots for the same.

1. Normal Startup
   

2. Run from a different working directory
   

3. Missing solver simulation
   

4. Missing WebAPP directory simulation
   While checking for this, I realised I renamed WebAPP, WEBAPP_PATH still pointed to /Users/.../MUIOGO/WebAPP but that folder no longer existed. Python just recreated WebAPP/DataStorage. So we can never detect WebAPP missing because it gets recreated before validation runs. - (Might need to be solved if unintentional)

Permission behaviour validation

1. Development environment:
   

2. Production/default environment:
   

3. Change path of DataStorage:
   Same issue as point 4 above where it automatically creates an empty folder.

Validation in the comment above

1.Solver present only in WebAPP/SOLVERs → correctly detected locally

2. Solver present only in system PATH → correctly detected via shutil.which()
   

3. Solver absent in both → validation fails fast with install guidance
   

Is the automatic creation of WebAPP/DataStorage intentional design, or should validation be responsible for detecting and failing when required directories are missing?
I want to make sure I’m fully aligned with the merged path/solver logic before updating the PR. When you mention aligning the startup checks with the merged path/solver logic, could you please clarify which specific behavior you’d like mirrored?
Happy to adjust the implementation once I understand the intended alignment more precisely.

[sanvishukla]

@SeaCelo, I have changed this PR according to the new changes in mac-port branch, please check and let me know if anything needs to be added or changed.