In the evaluator-optimizer pattern, one model drafts, another checks the draft, and the loop repeats until the quality is good enough.
The loop#
- The generator produces a draft.
- The evaluator scores the draft against a rubric.
- If it passes, you are done.
- If not, the evaluator's feedback goes back to the generator.
- The generator revises, and the loop goes back to step 2.
The rubric is the key#
The evaluator needs explicit criteria to check, not "is this good?" Vague criteria produce vague feedback, and the generator spins without improving.
Example: generating a product description. The evaluator checks three things:
| Criterion | Pass when |
|---|---|
| Length | Under 50 words |
| Content | It mentions the key feature |
| Style | It avoids jargon |
If one fails, the generator gets that specific feedback, for example "62 words, cut to under 50", and rewrites. The same rules for writing a good rubric apply as for an LLM judge.
The trade-off#
Every loop adds another generator call and another evaluator call. Two settings keep it in check:
- A maximum number of iterations, so it cannot loop forever. The same idea as the iteration cap in tool calling limits.
- A clear pass threshold, so an evaluator that is too strict does not reject drafts that are good enough.
The evaluator does not have to be a model#
If you can check quality in code, do that. It is faster and cheaper than a model call.
- Count the words.
- Validate the JSON.
- Run the tests on generated code.
Use a model as the evaluator only for criteria that code cannot check, such as tone or whether the key feature is described well.
Short line#
Evaluator-optimizer loops a generator and an evaluator: draft, check against a clear rubric, send specific feedback, revise. Cap the iterations, set a pass threshold, and use code as the evaluator wherever a check can be written in code.
Key takeaways#
- The generator drafts, the evaluator scores against a rubric, and failing drafts get feedback and a rewrite.
- The rubric must be explicit. Vague criteria give vague feedback.
- Each loop costs a generator call and an evaluator call.
- Set a maximum number of iterations and a clear pass threshold.
- Prefer code checks, such as word count, JSON validation or tests, over a model evaluator.