Reliability · bigonotes

Model providers slow down, throttle you and go down. A reliable feature expects this and has a plan for each case. The basics are in Error handling. This note puts them together with two production patterns: fallback models and circuit breakers.

Timeouts on every request#

Set two timeouts:

Total timeout: the maximum time for the whole response.
Idle timeout: for streams, the maximum gap between chunks. If no chunk arrives for 30 seconds, give up.

Without timeouts, a hung request holds a thread or connection forever. Enough of them and your service stops answering anyone.

Retry only what can succeed#

Response	Retry?	How
429 (rate limited)	Yes	Exponential backoff with jitter
5xx (provider error)	Yes	Exponential backoff with jitter
Other 4xx	No	Fix the request; it will fail the same way every time

Cap the number of tries. Each retry resends the full request and can add to the bill.

Fallback model#

When the primary model is unavailable, a fallback model from another provider or a different size can take over. It is not a drop-in copy. It may follow your prompts differently, format output differently or refuse different things. Run your evals on the fallback too, before you need it.

Circuit breaker#

Retries help with short blips. During a real outage they make things worse: every request waits through its full retry sequence, and you add load to a provider that is already struggling.

A circuit breaker tracks the failure rate. When it crosses a threshold, the breaker opens and your code stops calling that provider for a cooldown period. It goes straight to the fallback instead. After the cooldown, it lets a few requests through to test whether the provider has recovered.

Circuit breaker states: closed sends requests to the provider, too many failures open it, open skips the provider for a cooldown, then half-open lets test requests through and either closes again or reopens

Degrade gracefully#

When everything fails, the user should still get something sensible:

Fall back to a non-AI path, such as keyword search or a canned answer.
Show a friendly error that says what happened and what to try next.

Never show a raw stack trace.

Short line#

Put a total timeout and a stream idle timeout on every request, and retry only 429 and 5xx with backoff and jitter. Keep an evaluated fallback model, use a circuit breaker so an outage does not pile up retries, and degrade to a non-AI path or a friendly error.

Key takeaways#

Set a total timeout and, for streams, an idle timeout (for example 30 seconds without a chunk).
Retry 429 and 5xx with exponential backoff and jitter. Do not retry other 4xx errors.
A fallback model can behave differently on your prompts, so it needs its own eval run.
A circuit breaker stops calling a failing provider for a cooldown instead of piling on retries.
When everything fails, use a non-AI path or a friendly message, never a stack trace.

Core idea#

Expect the provider to be slow, throttled or down, and have a plan for each.

Key points#

Total timeout on every request. Idle timeout on streams (for example 30 seconds without a chunk).
Retry 429 and 5xx with exponential backoff and jitter. Cap the tries.
Do not retry other 4xx errors.
A fallback model needs its own eval run.
A circuit breaker stops calling a failing provider for a cooldown, then tests it with a few requests.
When everything fails: non-AI path or a friendly error.

Pitfalls#

No timeout: hung requests hold threads and connections forever.
Retries during an outage pile load on the provider and make users wait longer.
Never show a raw stack trace.

Short line#

Timeouts, retries with backoff, an evaluated fallback, a circuit breaker, graceful degradation.