An LLM feature changes when the model changes or when the prompt changes. Both can happen without anyone touching your application code. Versioning makes each change deliberate, testable and traceable.
Pin the model#
Never use a "latest" alias in production. The provider can move the alias to a new model at any time, and your feature changes behavior overnight with no change on your side.
Pin a specific dated snapshot instead. When you want to upgrade, rerun your evals on the new snapshot first, then switch. Track provider deprecation dates as well, so a pinned version is not retired before you have planned the move.
Version the prompts#
Prompts are code. Keep them in version control, not hardcoded as strings across your application.
- Every prompt change gets a version number.
- Every request logs which prompt version handled it (see Logging).
When quality drops, you can then find the exact change that caused it and roll it back.
Roll out safely#
Change model or prompt in three steps:
- Shadow test: run the new version alongside the old one on real traffic. Users only see the old version's output. Compare the two.
- Canary: send a small percentage of real traffic to the new version and watch the metrics.
- Full rollout: move all traffic once the canary looks healthy.
Rerun evals before every change. A prompt change that improves one case can quietly break another.
Short line#
Pin a dated model snapshot, never "latest", and rerun evals before upgrading. Keep prompts in version control with a version number logged on every request, and roll changes out through a shadow test, then a canary, then full traffic.
Key takeaways#
- Never use a "latest" alias in production. Pin a dated snapshot.
- Rerun evals before any model upgrade, and track deprecation dates.
- Keep prompts in version control. Each change gets a version number.
- Log the prompt version on every request so you can trace a regression to its change.
- Roll out with a shadow test, then a canary, then full traffic.
- Rerun evals before every change; an improvement in one place can break another.