A regression is when something that worked before stops working after a change. In ordinary software, a regression often crashes or fails a test. In LLM systems it is usually silent, which is what makes it dangerous.
Why regressions are silent#
After a bad change, the system still runs, still returns responses and still looks fine. The answers are just worse on some cases. You only notice when a user complains, or when you happen to test the right case.
Rerun the golden set after every change#
The fix is to rerun your golden set after every change, before it ships:
- Every prompt edit
- Every model upgrade
- Every chunking change
- Every retrieval tweak
Make the run cheap enough that nobody skips it:
- Automate it, for example as a step in CI.
- Use code assertions for the cheap checks on every case.
- Call the judge model only for cases that need it.
A well-built eval suite runs in minutes, not hours. Compare each run with the last one, case by case, so you see exactly which cases changed.
Two triggers teams forget#
Model upgrades#
Providers update models, and behavior shifts. Pin your model version, so the model only changes when you decide. Rerun evals on the new version before you upgrade.
Prompt changes#
Even small wording changes can swing results a lot. Evals tell you which version of a prompt is actually better, instead of which one looked better on the three examples you tried.
Short line#
LLM regressions are silent: the system still runs but answers get worse. Rerun the golden set on every prompt, model, chunking or retrieval change, automated and fast, and pin the model version so upgrades happen only after an eval run.
Key takeaways#
- A regression is something that worked before and broke after a change.
- In LLM systems, regressions are silent. The system still runs and looks fine.
- Rerun the golden set after every prompt, model, chunking or retrieval change.
- Automate it, use code checks first, and call the judge only where needed. Aim for minutes.
- Pin the model version and rerun evals before upgrading.
- Small prompt wording changes can swing results. Let evals decide.