When an LLM feature gets slow, expensive or wrong, the logs are how you find out why. Log the same fields on every request, so you can compare requests and spot changes over time.
What to log on every request#
| Field | Why you need it |
|---|---|
| Model and version | Ties a change in behavior to a model change |
| Prompt version | Ties a change in behavior to a prompt change (see Versioning) |
| Input and output token counts | Explains cost and latency |
| Latency | Shows what users wait for |
| Cost per request | Shows where the money goes |
| Stop reason | Shows whether the model finished or hit a limit |
Track latency as p95 and p99, not only the average. The average hides the slow requests, and those are the ones users complain about.
The stop reason is easy to skip, but it is useful. If many responses stop because they hit the token cap, your max_tokens is too tight and users are getting cut-off answers.
Metadata, not content#
Prompts and responses often contain personal data. Do not log raw content when it can contain PII. Log the metadata above instead. Log content only when you have a specific need and the right controls: access limits, redaction and a retention period.
Traces, not flat logs#
One request to an LLM feature is often several steps: retrieval, one or more model calls, tool calls. A flat log line gives you the total. A structured trace splits the request into spans, one per step, each with its own time and token count.
With spans you can see exactly where the time and tokens went. Tools such as LangSmith and Langfuse record traces like this.
Short line#
Log model and version, prompt version, token counts, latency as p95 and p99, cost and stop reason on every request. Log metadata rather than raw content when it may hold PII, and use traces with spans to see where time and tokens went.
Key takeaways#
- Log model and version, prompt version, input and output tokens, latency, cost and stop reason.
- Track p95 and p99 latency, not only the average.
- Many token-cap stops mean
max_tokensis too tight. - Log metadata, not raw content, when content may contain PII.
- Traces with spans (retrieval, model call, tool call) show where time and tokens went. LangSmith and Langfuse do this.