Evaluation
Evals for LLM systems: golden sets, code checks, LLM judges, regression runs and production signals.
6 notes · 13 min total
- 1Why EvalsWhy LLM systems need evals: outputs vary, small changes break things, and without measurement every change is a guess.2 min
- 2Golden SetWhat a golden set is, the four kinds of test cases it needs, and the rules for keeping it fixed while it grows.2 min
- 3Simple MetricsThe cheap, deterministic checks you can run in code (format, presence, grounding, exact match) before reaching for a judge model.2 min
- 4LLM-as-JudgeHow to grade fuzzy qualities with a judge model and a rubric, why pass/fail beats scores, the three judge biases, and how to validate a judge.2 min
- 5RegressionWhy regressions in LLM systems are silent, why the golden set must rerun after every change, and the two triggers teams forget.2 min
- 6In ProductionWhich production signals reveal failures your golden set missed, how failures feed back into it, and the trade-offs of building evals.3 min