The cost of one request is simple: input tokens times the input price, plus output tokens times the output price. The basic levers (shorter prompts, max_tokens, cheaper models, caching) are in Cost. At production volume, three of them need a closer look, because small per-request numbers multiply into a large monthly bill.
Prompt caching#
Repeated content adds up fast. A 2,000-token system prompt sent on 10,000 requests a day is 20 million input tokens a day from the system prompt alone.
Caching works on the start of the prompt, so the order of the parts matters:
| Position | Content | Changes per request? |
|---|---|---|
| First | System prompt, tool definitions, reference documents | No |
| Last | The user's question | Yes |
Put the stable parts first and the changing part last. Then the cache hits on every request. If you put the user's question first, the prefix is different every time and nothing is cached.
Batch APIs#
Some work does not need an answer in real time: nightly classification, processing uploaded documents, back-filling summaries. Batch APIs run these requests asynchronously at roughly half the price.
The trade-off is turnaround time. Results come back in hours, not milliseconds, so batch is for background jobs, never for a user who is waiting.
Model routing#
Requests are not all equally hard. Route easy requests to a cheap small model and hard requests to the expensive large model. A simple classifier or a rule (for example, by task type) decides the route.
Measure quality per route. The savings only count if the cheap model is actually good enough for the requests you send it, and that needs its own evals.
Back-of-the-envelope math#
Interviewers often ask you to estimate a bill. Use the same steps each time: requests per day, times tokens per request, times price.
Example: 10,000 requests a day, 2,000 input tokens and 500 output tokens each, at 15 per million output tokens.
| Part | Tokens per day | Price | Cost per day |
|---|---|---|---|
| Input | 10,000 × 2,000 = 20M | $3 / M | $60 |
| Output | 10,000 × 500 = 5M | $15 / M | $75 |
| Total | 4,000 a month) |
Output is a quarter of the tokens but more than half of the bill, because it is priced higher. The system prompt is most of the input, so caching it is the first lever to pull.
Short line#
Cost is input tokens times input price plus output tokens times output price, on every request. At scale, put stable content first so prompt caching hits, send non-urgent work to a batch API at about half price, and route easy requests to a cheaper model after checking its quality.
Key takeaways#
- Per-request cost: input tokens × input price + output tokens × output price.
- A 2,000-token system prompt on 10,000 requests a day is 20M input tokens a day.
- Put stable content (system prompt, tools, documents) first and the user's question last so the cache hits.
- Batch APIs cost roughly half, but results take hours. Use them for background work only.
- Route easy requests to a small model and hard ones to a large model. Measure quality per route.
- Estimate with requests per day × tokens per request × price, separately for input and output.