Simple Metrics

Before you reach for anything fancy, start with the checks you can write in code. They are cheap, fast and deterministic, and they catch a surprising number of failures. Run them on every case in the golden set.

Four kinds of code checks#

Format checks#

Does the output parse as valid JSON? Is it under the length limit? Does it contain a citation? Does it use the required format? These pair with structured output.

Presence checks#

For RAG, check whether the right document was retrieved. This tells you whether a wrong answer is a retrieval problem, which the common failures note says to check first.

Grounding checks#

Check that every cited chunk ID is one that was actually provided, as set up in answer rules. Check that any quoted text appears verbatim in those chunks. An invented ID or quote fails at once.

Exact match#

For classification or extraction, where there is one right answer, check whether the output matches it exactly.

What it looks like in code#

Each check is a plain assertion, so it runs in milliseconds and gives the same result every time.

import json
import re

def check_answer(
    output: str, provided_ids: set[str], chunks: dict[str, str]
) -> list[str]:
    failures = []
    try:
        data = json.loads(output)
    except json.JSONDecodeError:
        return ["not valid JSON"]

    answer = data.get("answer", "")
    if len(answer) > 1200:
        failures.append("too long")

    cited = set(re.findall(r"\[(doc_\d+)\]", answer))
    if not cited:
        failures.append("no citation")
    if cited - provided_ids:
        failures.append(f"cites unknown ids: {sorted(cited - provided_ids)}")

    # Quoted text must appear word for word in a cited chunk.
    for quote in re.findall(r'"([^"]+)"', answer):
        if not any(quote in chunks[i] for i in cited & provided_ids):
            failures.append(f"quote not in sources: {quote!r}")
    return failures

The principle#

Use code assertions wherever the check can be written as a rule. Save the expensive, fuzzy checks, such as an LLM judge, for what code cannot handle.

Short line#

Start with checks you can write in code: format, retrieval presence, grounding of citations and quotes, and exact match. They are cheap, fast and deterministic; save the judge model for what code cannot check.

Key takeaways#

• Code checks are cheap, fast, deterministic, and catch many failures.
• Format: valid JSON, length limit, citation present, required format.
• Presence: the right document was retrieved.
• Grounding: cited IDs were provided, and quotes appear verbatim in them.
• Exact match: for tasks with one right answer.
• Use code wherever a rule works. Use a judge only for the rest.