Streaming

By default, the API waits until the model has finished the whole response, then sends it all at once. Streaming changes that: the API sends the response piece by piece as it is generated. It does not make the model faster, but it makes your app feel much quicker to the user.

Why streaming helps#

Without streaming, a long answer that takes 10 seconds to generate leaves the user looking at a blank screen for 10 seconds. Then the whole answer appears in one go.

With streaming, the API sends each token as it is generated, so the answer appears word by word. Tokens arrive one at a time because that is how the model writes: it predicts a single token, then the next (see Next-Token Prediction).

The model does the same work either way. Generation takes just as long. What changes is when the user first sees something, and seeing progress right away is what makes the app feel fast.

How it works in code#

With streaming on, you do not get one complete response back. You get a stream of chunks, and your app reads them as they arrive.

For each chunk, your app appends it to the text already on screen. When the stream ends, you also get a stop reason that says why the model stopped. The snippet below uses a generic stream_model helper that yields text pieces. The stop reason is empty until the final piece.

def read_stream(messages):
    shown = ""
    stop_reason = None
    for text, reason in stream_model(messages):
        shown += text  # append, so the display grows as chunks arrive
        print(text, end="", flush=True)
        if reason is not None:  # only set on the last chunk
            stop_reason = reason
    return shown, stop_reason

Your app keeps the stop reason so it can check it once the stream is over. That check matters for the second gotcha below.

Gotchas#

Streaming adds two problems that a normal request does not have. Both are about acting on a response before you have all of it.

Structured output cannot be parsed mid-stream#

JSON is only valid once it is complete, so you cannot parse it while chunks are still arriving. A half-finished object is not valid JSON.

The fix is to buffer. Stream the text to the user if that helps the experience, but hold off on any processing until the stream finishes. Then parse the complete response.

A stream can die mid-response#

If the connection drops halfway through, you are left with a partial response. Showing it as if it were finished would give the user half an answer.

Your app needs to detect this in one of two ways:

• Check the stop reason at the end of the stream. A missing or unexpected reason means the answer may be incomplete.
• Catch the connection error when the stream breaks.

Then handle it gracefully, for example by showing a message or retrying, rather than leaving the partial text as the final answer.

Short line#

Streaming sends the response as it is generated, so users see progress at once even though the total time is the same. You must buffer before parsing structured output and check how the stream ended.

Key takeaways#

• Without streaming, the user waits for the whole response. With streaming, tokens appear as they are generated.
• Streaming does not speed up generation. It improves how fast the app feels.
• In code, you read a stream of chunks and append each one to what is displayed.
• Buffer the full response before parsing JSON, because partial JSON is not valid.
• A stream can break mid-response, so check the stop reason at the end or catch the connection error.
• Do not show a partial answer as if it were complete.