Better Error Recovery #81
Description
In the case of a failure, the error should be identified, and the current WebCT state should be rolled back to a known good configuration, even in the state of simulator communication interrupts and crashes.
Previously, simulators used to be created and let loose, however processes were not correctly reaped, in the case of errors. This lead to broken session states that required the introduction of locks. After the addition of caching, this makes it hard to recover simClient state due to the locking nature, and no cache invalidation methods.
This may be complex, as it requires the program to understand its location in the execution pipeline in order to understand what went wrong, with zero feedback from the error source.
Ideally, this should also recover from simtimeout errors flawlessly;- with no lost state, and just a user message on the frontend stating that simulation failed because it took so long. (and perhaps in the future, display a progress indicator for #32
Right now, the approach is to let the server fail and get into a broken state; This is unacceptable, and ideally we should just reap the session state and start anew.