[Bug] Prevent solver subprocess hangs by introducing configurable execution timeout

[parthdagia05]

Summary

The solver execution pipeline currently invokes external solver binaries (such as GLPK) using subprocess.run() without any timeout protection.

If the solver process becomes stuck, enters an infinite loop, or fails to terminate correctly, the Flask server will block indefinitely while waiting for the subprocess to complete.

This can cause several operational issues:

• The backend API becomes unresponsive
• Scenario execution may hang indefinitely
• The server may require a manual restart to recover
• Other requests may be blocked depending on the execution context

Additionally, the codebase contains several legacy commented-out subprocess commands using shell=True. Even though they are currently commented out, keeping these patterns in the codebase increases the risk of unsafe subprocess execution being reintroduced in the future.

Expected behavior

Solver subprocess execution should have a configurable timeout so that long-running or stalled solver processes cannot block the server indefinitely.

If a solver exceeds the allowed execution time:

• The subprocess should be terminated
• A structured error response should be returned
• The backend should remain responsive

The solver timeout should be configurable through environment configuration to allow flexibility for different model sizes and runtime requirements.

Reproduction steps

1. Run a model execution that invokes the solver through the backend pipeline.
2. If the solver process becomes stuck or takes excessively long to complete, the backend will continue waiting indefinitely.
3. The Flask server remains blocked while waiting for the subprocess to terminate.

Since no timeout is enforced, the system cannot recover automatically from stalled solver executions.

Environment

OS: macOS / Linux / Windows (observed during code audit)
Python version: Python 3.x
Repository: EAPD-DRB/MUIOGO
Branch: main

Logs or screenshots

No response

[parthdagia05]

@SeaCelo I have added the issue.

[brightyorcerf]

Great addition on the timeout protection. Preventing indefinite hangs is a solid win for stability.

To build on this, I’m currently auditing the Process Lifecycle for the Stability track (#258). While timeout handles active sessions, we still have a risk of 'Zombie' solver processes if the Flask parent process itself crashes or restarts.

I’m drafting a Global Process Registry utility to complement this, it will use atexit and signal traps to ensure these solvers are reaped even during ungraceful exits. Looking forward to seeing this merge so we can layer that extra safety on top!

[parthdagia05]

Great addition on the timeout protection. Preventing indefinite hangs is a solid win for stability.

To build on this, I’m currently auditing the Process Lifecycle for the Stability track (#258). While timeout handles active sessions, we still have a risk of 'Zombie' solver processes if the Flask parent process itself crashes or restarts.

I’m drafting a Global Process Registry utility to complement this, it will use atexit and signal traps to ensure these solvers are reaped even during ungraceful exits. Looking forward to seeing this merge so we can layer that extra safety on top!

That improves system stability good issue to work on

[SeaCelo]

These models can take a long time to run. sometimes hours. What would be a good approach to including a timeout protection under these conditions? Perhaps a test that the model is still proceeding?

[brightyorcerf]

@SeaCelo I’ve just completed a deep audit of DataFileClass.py and OsemosysClass.py to see how we can practically support those multi-hour models without sacrificing server stability.

I feel like we could move beyond a hard timeout by implementing a State-Aware Heartbeat.

1. File-I/O Pulse: Since we already define self.resFile and use the -solu switch for CBC (line 2186), we don't need to guess if it's stuck. We can use a non-blocking subprocess.Popen and a loop that monitors the st_mtime (last modified time) of the results file. As long as the solver is writing data, we reset the safety timer.

2. Orphan/Zombie Prevention: Currently, there are no atexit or signal handlers. If the Flask parent crashes during a 4-hour solve, that solver becomes a "Zombie" process consuming 100% CPU indefinitely.

3. The Proposal: I suggest we transition from subprocess.run to a managed Popen lifecycle. This allows us to:
   -Support infinite-duration solves (as long as they are active).
   -Auto-reap solvers if the server restarts ([Feature] Standardizing Atomic I/O and Global State Locking for Concurrent Stability  #258).
   -Provide Real-time Progress to the UI (async solver execution with job-based run management  #186).

4. Full-Pipeline Coverage: This "Pulse" logic applies to both the GLPK matrix generation (lp.lp) and CBC optimization (results.txt). Since both paths are already defined in run(), we can monitor st_mtime sequentially across the entire execution without extra path resolution.

I'd be happy to draft a ProcessManager utility to handle this logic if this approach aligns with your vision for the stability track.

[parthdagia05]

That makes sense but my original proposal focused on prevent crash only and then improve architecture, if @brightyorcerf is taking up a follow up its great.

[SeaCelo]

My current view is that this issue is currently framed around timeout enforcement, and that does not match the current product preference for local runs.

The only slice here that currently seems plausibly worth keeping would be some future non-destructive “still running” visibility or status indication. I do not currently want hard timeout counters or automatic kill-after-N-seconds behavior.

Because of that, my default direction right now is to close this unless there is a concrete present-day local-use reason to keep a narrowed version open.

If you want to make that case, please do it in #301.

I’m using that discussion for the first-pass review and will come back here after the review window.