Structured Output

Most real applications need model output that code can read, not prose for a person. You can ask for that structure with three levels of strength, and each one guarantees more than the last. None of them guarantees that the values are right, so validation stays your job.

Three levels of control#

The levels differ in what the provider promises and in how a response can still go wrong.

Prompt-only output works some of the time and breaks unpredictably. In JSON mode the provider promises the text will never be malformed, but it makes no promise about shape.

With schema-constrained output, the provider restricts decoding. At each step the model can only pick tokens that keep the output valid against your schema, so the structure is guaranteed.

Valid is not correct#

A response can match your schema perfectly and still contain the wrong date or the wrong number. The schema checks the shape of the answer, not its truth. That is why you validate in code whichever level you use: Pydantic in Python, Zod in TypeScript.

When validation fails, send the error back to the model and retry once or twice. After that, fail gracefully instead of looping. The usual retry rules apply: cap the number of tries, because every attempt costs a request.

from pydantic import BaseModel, ValidationError

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str | None = None  # null is an honest "not in the text"

def extract_invoice(messages, max_tries=3):
    for _ in range(max_tries):
        raw = call_model(messages)
        try:
            return Invoice.model_validate_json(raw)
        except ValidationError as err:
            # Show the model exactly what it got wrong.
            messages = messages + [{"role": "user", "content": str(err)}]
    return None  # out of tries: let the caller degrade

Here call_model stands for whatever function sends the messages and returns the reply text. The due_date field allows null on purpose. If the field were required, a document with no due date would push the model to invent one.

Trade-offs and field order#

Schema-constrained output has a cost of its own. Constraining decoding means the model sometimes cannot produce its most natural token and must take the next best one that fits. On complex schemas this can slightly reduce quality.

The model also writes fields left to right, in the order they appear in the schema, which is the same token-by-token generation as plain text. So field order is reasoning order. Put a reasoning or evidence field before the answer field, and the model works through the problem first and then commits.

{
  "reasoning": "The email mentions a refund and a broken item.",
  "category": "refund_request"
}

If category came first, the model would have to commit to a label before it had written any reasoning. This is the same idea as step-by-step prompting, applied inside a schema.

Streaming and structure#

Structured output and streaming pull in different directions. A half-finished JSON string cannot be parsed, so you normally wait for the whole response before you validate it.

Short line#

Ask for structure at the strongest level your provider offers, put reasoning fields before answer fields, and always validate in code. A schema guarantees shape, never truth.

Key takeaways#

• The three levels are prompt-only, JSON mode and schema-constrained. Each guarantees more than the one before.
• Valid JSON that matches your schema can still hold wrong values.
• Validate with Pydantic or Zod. On failure, send the error back and retry once or twice, then fail gracefully.
• Allow null for optional fields so the model has an honest alternative to inventing a value.
• Schema-constrained decoding can slightly reduce quality on complex schemas.
• Field order is reasoning order: put reasoning before the answer.