Add a generic retry evaluator for wrapping CallableModel invocations that may fail transiently (flaky APIs, databases, object stores, etc.).
This pattern is currently re-implemented ad-hoc by users. A first-class evaluator would standardize the behavior and compose cleanly with the rest of the evaluator stack.
Behavior
- Configurable maximum number of attempts.
- Configurable backoff policy (at minimum: constant and exponential, with optional jitter).
- Configurable allow-list and deny-list of exception types to retry on.
- Optional hook invoked between attempts (for logging, metrics, etc.).
Open questions
- Idempotency. Retries assume the wrapped model is safe to re-invoke with the same context. Should
CallableModel gain an explicit idempotency flag that this evaluator respects, or should it be the user's responsibility to only stack retry on idempotent models?
- Interaction with caching. When stacked with
MemoryCacheEvaluator, transient failures should not be cached. Need to confirm or adjust the cache evaluator's behavior on exceptions, and document the canonical stacking order.
Add a generic retry evaluator for wrapping
CallableModelinvocations that may fail transiently (flaky APIs, databases, object stores, etc.).This pattern is currently re-implemented ad-hoc by users. A first-class evaluator would standardize the behavior and compose cleanly with the rest of the evaluator stack.
Behavior
Open questions
CallableModelgain an explicit idempotency flag that this evaluator respects, or should it be the user's responsibility to only stack retry on idempotent models?MemoryCacheEvaluator, transient failures should not be cached. Need to confirm or adjust the cache evaluator's behavior on exceptions, and document the canonical stacking order.