Agentic Evaluation and Benchmarks Cheatsheet

"It seems to work" is not an evaluation. Yet many AI features ship on exactly that: a few good demos, a handful of hand-picked prompts, and a leaderboard number that says the model is smart.

Evaluating AI is measurement under uncertainty. The outputs are probabilistic, the tasks are open-ended, and the same prompt can pass on Monday and fail on Tuesday. The good news: the concepts you need are few and stable. Tasks, criteria, graders, metrics and error bars work the same whether you use one vendor's SDK or another, a hosted eval product or a 50-line script.

This post is a practical cheatsheet. It defines the vocabulary precisely, shows how to score and how much to trust a score, and maps "accuracy" and "quality" to concrete metrics for each kind of system: agents, chatbots, zero-shot LLM features, and generative models such as image generators.

Terms marked with a ⚠️ are used with more than one meaning. Context decides (see "Same word, different meaning" near the end).

1. The vocabulary of an eval

Most confusion in eval discussions comes from people using the same word for different things. These definitions follow the ones Anthropic and OpenAI use in their eval guides.

Capability evals vs. regression evals

Two suites, two jobs:

• Capability evals ask "what can it do?". They should start at a low pass rate, on tasks the system struggles with, so there is room to improve.
• Regression evals ask "does it still do what it used to?". They should pass at nearly 100%, and any drop is a bug.

When a capability task passes reliably, promote it to the regression suite. When a capability suite passes everything solvable, it is saturated and no longer tells you anything: write harder tasks.

2. Graders: who decides what "good" means

Practical rules that hold for any stack:

• Use the cheapest grader that is valid. If code can check it (the refund exists, the test passes, the JSON parses), use code. Use a model judge only for what code cannot see.
• Grade outcomes, not paths. Agents often find valid routes the eval designer did not expect. Checking for one exact sequence of tool calls fails correct solutions. Grade the path only when the path itself is a requirement (e.g. "must verify identity before changing an address").
• Prefer binary and pairwise judgments. "Does the answer state the cancellation fee? yes/no" and "which of these two answers is better?" are more reliable than "rate 1-10".
• Calibrate model judges. Have humans label a sample (often 50-200 items), measure agreement with the judge (percent agreement and Cohen's kappa), and only scale up once agreement is close to human-human agreement.
• Watch for known judge biases: longer answers (verbosity bias), the first option shown (position bias, so swap the order and judge twice), and output from the judge's own model family (self-preference).
• Read transcripts. Regularly. A 0% pass rate on a task that a strong model should handle is usually a broken task or grader, not an incapable agent.

A rubric is data, not prose

Write rubrics as explicit, separately scored criteria with anchors. Each criterion becomes its own grader call or code check:

RUBRIC = {
    "task": "Customer asks to cancel a subscription bought 10 days ago.",
    "criteria": [
        # (id, check, grader, weight, hard_fail)
        ("policy", "Offers a full refund (policy: refunds within 14 days)", "code", 3, True),
        ("action", "Subscription status is 'cancelled' in the database", "code", 3, True),
        ("verify", "Confirms account identity before acting", "model", 2, True),
        ("clarity", "States what happens next and when, in plain language", "model", 1, False),
        ("tone", "Polite, no blame, no upsell after a cancel request", "model", 1, False),
    ],
}


def score(results: dict[str, bool]) -> dict:
    """results maps criterion id -> passed. Hard fails zero the task."""
    crits = RUBRIC["criteria"]
    if any(hard and not results[cid] for cid, _, _, _, hard in crits):
        return {"pass": False, "score": 0.0}
    total = sum(w for *_, w, _ in crits)
    got = sum(w for cid, _, _, w, _ in crits if results[cid])
    return {"pass": True, "score": got / total}

Two ideas carry most of the value here: hard-fail criteria (a polite answer that breaks policy is a failure, not a 7/10) and one check per criterion, so a low score tells you exactly what went wrong.

3. Metrics: what to count

Pick metrics after you define success, not before. Most real evals combine one primary metric (what you optimize) with a few guardrail metrics (what must not get worse).

pass@k vs. pass^k

These two metrics answer opposite questions, and they diverge fast. With a per-trial success rate p on a task:

• pass@k = 1 - (1 - p)^k
• pass^k = p^k

At p = 0.75 and k = 3: pass@3 is about 98%, pass^3 is about 42%. At p = 0.95 and k = 10, pass^10 is only about 60%. In tau-bench, strong agents that solved most retail tasks once dropped below 25% when asked to solve the same task 8 times in a row. A demo shows pass@k. Your customers experience pass^k.

In practice you estimate both from n trials per task with c successes, then average over tasks:

from math import comb, sqrt


def pass_at_k(n: int, c: int, k: int) -> float:
    """P(at least one of k samples passes), from n trials with c passes."""
    if n - c < k:
        return 1.0
    return 1 - comb(n - c, k) / comb(n, k)


def pass_hat_k(n: int, c: int, k: int) -> float:
    """P(all k samples pass), from n trials with c passes."""
    return comb(c, k) / comb(n, k)


def wilson_ci(successes: int, n: int, z: float = 1.96) -> tuple[float, float]:
    """95% confidence interval for a pass rate (works near 0% and 100%)."""
    p = successes / n
    center = (p + z**2 / (2 * n)) / (1 + z**2 / n)
    half = z * sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / (1 + z**2 / n)
    return center - half, center + half


# One task, 8 trials, 6 passed (75% per-trial success)
print(f"pass@1 = {pass_at_k(8, 6, 1):.2f}")   # 0.75
print(f"pass@3 = {pass_at_k(8, 6, 3):.2f}")   # 1.00
print(f"pass^3 = {pass_hat_k(8, 6, 3):.2f}")  # 0.36

# Suite: 40 of 50 tasks passed
lo, hi = wilson_ci(40, 50)
print(f"80% pass rate, n=50 -> 95% CI {lo:.0%} to {hi:.0%}")    # 67% to 89%
lo, hi = wilson_ci(400, 500)
print(f"80% pass rate, n=500 -> 95% CI {lo:.0%} to {hi:.0%}")  # 76% to 83%

The pass@k estimator is the unbiased one from OpenAI's Codex paper; the pass^k estimator is the one from tau-bench. Both need n of at least k trials per task.

4. Variables: what changes the score

An eval score is a function of the whole setup, not only the model. Fix and record every variable you are not testing, or you will compare noise.

Independent variable: the one thing you change (new prompt, new model). Dependent variable: the metric. Controlled variables: everything else in the table above. This is the whole experimental design, and it does not depend on the tool you use.

5. Statistics: how much to trust a number

A score without an interval is an anecdote. These are the few statistical ideas that cover most eval decisions, largely following Anthropic's paper Adding Error Bars to Evals (Miller, 2024).

How big must the eval be? A quick guide for a pass rate near 50% (the worst case):

Paired comparisons on the same tasks detect smaller differences than these numbers suggest, because they remove the "which tasks" variance. A minimal paired comparison:

from statistics import mean, stdev

# Per-task scores for the same 8 tasks (in practice: hundreds), old vs. new prompt
old = [1, 0, 1, 1, 0, 1, 0, 1]
new = [1, 1, 1, 1, 0, 1, 1, 1]

diffs = [b - a for a, b in zip(old, new)]
d = mean(diffs)
se = stdev(diffs) / len(diffs) ** 0.5
print(f"Improvement: {d:+.0%}, 95% CI {d - 1.96 * se:+.0%} to {d + 1.96 * se:+.0%}")
# Improvement: +25%, 95% CI -7% to +57%  -> not conclusive with 8 tasks

6. Accuracy and quality depend on the use case

"Accuracy" is not a property of a model. It is a property of a system on a task, against criteria someone chose. The same answer can be accurate for a brainstorming tool and unacceptable for a billing agent. Define both words per use case:

• Accuracy: does the output match what is true or required? Requires a reference, a verifiable outcome, or a fact check.
• Quality: is it good to receive? Usefulness, clarity, completeness, tone, safety, cost and speed. Usually multi-dimensional and judged by rubric or preference.

A useful habit: write the criteria for "a perfect output" and "an unacceptable output" for ten real examples before choosing any metric. The metric should then be obvious.

GenAI beyond text: image generation

Image models need the same concepts with different graders:

The pattern is the same as for text: a cheap automatic proxy, calibrated against human ratings on a sample, plus explicit checks for what must never happen.

7. Benchmarks: useful, but not your eval

Public benchmarks help you shortlist models. They do not tell you how a model will do on your tasks, with your tools and your definition of success. Snapshot as of November 2026:

Warning signs when reading any benchmark number: no interval, no n or k, a harness you cannot reproduce, a saturated benchmark (everyone above 90%), and a benchmark that has been public long enough to leak into training data.

8. A minimal, stack-independent eval process

Every vendor guide, from Anthropic and OpenAI to Google, Microsoft and Amazon, converges on roughly the same loop:

1. Define success for the use case: the criteria, the hard fails, the primary and guardrail metrics, and the ship threshold.
2. Collect tasks from real usage and real failures. 20-50 is a fine start; grow it with every bug report. Include cases where the right behavior is to refuse, ask, or escalate.
3. Build graders: code first, model judges for the rest, calibrated against human labels.
4. Run with k trials per task in clean, isolated environments, at production settings.
5. Report the metric with n, k, a 95% CI and the dataset, prompt, model and grader versions.
6. Compare versions paired on the same tasks; read the transcripts of the tasks that changed.
7. Gate releases on the regression suite; track capability evals over time.
8. Monitor production: sample real traces, grade them with the same graders, and feed new failures back into step 2.

Amazon describes the same idea in layers: evaluate the final outcome, then the components (intent detection, tool use, memory, reasoning) to find why it failed, and add periodic human audits of traces. Microsoft's agent evaluators split it similarly into intent resolution, tool-call accuracy and task adherence.

Same word, different meaning

Why this matters

Evals are how AI teams replace opinions with evidence. A few practical takeaways:

• Concepts over tools. Tasks, criteria, rubrics, graders, metrics and intervals are the same everywhere. Frameworks and hosted eval products come and go; a well-defined task set and rubric outlives all of them.
• Define success per use case, before measuring. "Accurate" for a billing agent and for a brainstorming chatbot are different metrics.
• Report consistency, not just capability. pass@k sells demos; pass^k predicts customer experience.
• Every number needs n, k and an interval. 82% vs. 79% on 100 tasks is not a result.
• Metrics come in pairs. Accuracy with abstention, quality with cost, automatic judges with human calibration. Optimizing one alone usually breaks the other.

Final Thoughts

Most AI failures in production are not surprises in hindsight. They were in the long tail nobody tested, in the inconsistency nobody measured, or in a score whose error bar nobody computed.

You do not need a large platform to avoid them. You need a few dozen real tasks, clear criteria, an honest grader, and the discipline to report what the numbers can and cannot tell you. Start there, and grow the eval every time the system surprises you.

Sources

• Anthropic, Demystifying evals for AI agents (January 2026): definitions of task, trial, grader, transcript, outcome and harness; grader types; pass@k vs. pass^k; capability vs. regression evals; saturation.
• Anthropic, Quantifying infrastructure noise in agentic coding evals (2026): resource settings shift Terminal-Bench 2.0 scores by up to 6 points.
• Evan Miller (Anthropic), Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (2024): standard errors, clustered SEs, paired differences, power analysis.
• OpenAI, Evaluation best practices: eval process, LLM-as-a-judge guidance, anti-patterns such as vibe-based evals.
• OpenAI, Why language models hallucinate (2025): accuracy-only scoring rewards guessing; report error and abstention rates.
• OpenAI, Why we no longer evaluate SWE-bench Verified (February 2026).
• Chen et al. (OpenAI), Evaluating Large Language Models Trained on Code (2021): unbiased pass@k estimator.
• Yao et al. (Sierra), tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024): pass^k.
• Google Cloud, Evaluate Gen AI agents: trajectory exact, in-order and any-order match, precision and recall.
• Microsoft, Agent evaluators in Microsoft Foundry: intent resolution, tool-call accuracy, task adherence.
• Amazon, Evaluating AI agents: Real-world lessons from building agentic systems at Amazon (2026): layered agent metrics, human audits of traces.
• Meta, ARE: Scaling Up Agent Environments and Evaluations and Hugging Face, Gaia2 and ARE (2025).
• METR, Measuring AI Ability to Complete Long Tasks (2025): 50% time horizon.
• Chiang et al., Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (2024): Bradley-Terry ratings and intervals.
• Ghosh et al., GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment (2023), and Wiles et al. (Google DeepMind), Revisiting Text-to-Image Evaluation with Gecko (2024).