Error Handling

LLM APIs fail in predictable ways, and each kind of failure needs a different response. Some errors fix themselves if you wait. Others never will, and retrying only wastes time and money. Sorting them correctly is most of the job.

The three kinds of error#

Every failure falls into one of three groups, based on who is at fault and whether time will help.

A bad request will fail the same way every time. Retrying it only adds load and delay, so surface the error and fix the cause.

Backoff with jitter#

Exponential backoff means you wait a short time, retry, and if the call fails again you wait twice as long before the next try. The growing gaps give an overloaded service room to recover.

Jitter adds a random amount to each wait. Without it, many clients that failed together would all retry at the same instant and cause the next spike. With jitter, their retries spread out.

import random
import time

class RetryableError(Exception):
    """Raised for 429 and 5xx responses."""

def call_with_retry(messages, max_tries=5, base=1.0):
    for attempt in range(max_tries):
        try:
            return call_model(messages)
        except RetryableError:
            if attempt == max_tries - 1:
                raise  # out of tries: let the caller degrade
            # Double the wait each time; jitter spreads clients out.
            time.sleep(base * 2 ** attempt + random.uniform(0, 1))

Here call_model stands for whatever function sends one request. It should raise RetryableError for 429 and 5xx, and let every other 4xx error pass straight through.

Always cap the number of tries. Because the API is stateless, a retry resends the full request, and every attempt that reaches the model can add to your cost.

Timeouts#

Set a timeout on every API call. Without one, a stalled connection can hang a request, and the user waiting on it, for as long as the network allows. There are two timeouts to think about.

• Total timeout: the maximum time for the whole response, from sending the request to receiving the last token.
• Idle timeout: for streams, the maximum gap between chunks. If nothing arrives for 30 seconds, give up, even when the total timeout has not fired yet.

The idle timeout matters because a stream can stay open while producing nothing. The total timeout alone would let that stall run for a long time.

Graceful degradation#

When retries run out, your app still has to do something sensible. The user should never see a stack trace because an LLM request timed out. You have three main options:

1. Show a friendly error message that says what happened and what to try next.
2. Fall back to a non-AI path, such as a plain search box or a canned answer.
3. Queue the request and try again later.

Pick the option that fits the feature. A chat reply can show a message, a search feature can fall back to keyword search, and a background summary can wait in a queue.

The trade-off#

The stateless, pay-per-token design keeps the API simple and scalable. It also puts the engineering burden on your app: you manage state, cost, retries and timeouts. A chatbot design therefore always has two layers.

Short line#

The API is stateless and pay-per-token. Your app resends history, manages context, retries with backoff, sets timeouts, and degrades gracefully when the model is unavailable.

Key takeaways#

• Retry 429 and 5xx errors with exponential backoff and jitter. Do not retry other 4xx errors; fix the request.
• Jitter stops many clients from retrying at the same moment.
• Set a total timeout on every call, plus an idle timeout on streams.
• Cap retries, because each one resends the whole request and may cost money.
• When everything fails, show a friendly message, fall back to a non-AI path, or queue the request.
• The provider runs the model layer. You build the application layer.