claude-api skill · Claude Code · September 2026

build-eval + hillclimb

Two commands. /claude-api build-eval builds a test set and a grader from production transcripts, bug reports, and hand-written cases, pausing for approval at each step. /claude-api hillclimb iterates against them, one change at a time, with a held-out set to catch overfitting. This page maps the full flow of both.

The primary source is the claude.dev playbook by Lance Martin (Sep 28, 2026) and Hamel Husain's field review.

01 /claude-api build-eval data → grader → baseline

The command walks you through three phases: collecting examples in priority order, proposing and validating a grader, and running a baseline with a confidence interval. Claude pauses for your approval before moving between phases.

Phase A — Data sampling

Claude collects inputs in a fixed priority order, working from the most grounded data toward the most synthetic. The underlying rule: every synthetic example should be anchored in something real. Fabricated cases not tied to actual behavior measure a hypothetical version of your application, not the one running in production.

Adversarial sampling — don't pick cases because today's model fails them. Model capability is jagged. If you pick cases where the current model fails, you are sampling the valleys of that model's capability surface — and the next model will have different valleys. The benchmark ends up measuring that model's failure fingerprint rather than what is intrinsically hard for your application. Pick hard cases because a human judged them hard, not because a particular model got them wrong. You should be able to say why a case is hard before you include it.

Start

/claude-api build-eval

Run in Claude Code with the claude-api skill installed. Claude begins the interview, asking about your application and what you want to measure before touching any data.

↓source 1 — highest signal

Source 1 — Production transcripts

Real conversations your application already had

Claude asks about data retention and sensitive information before touching any logs. This is the strongest signal because it reflects the actual task distribution — what your users actually asked, not what you imagined they might ask.

Upside: closest to production. A conversation where something actually went wrong is worth a thousand synthetic ones.
Caution: users often try what they expect to work, which skews the sample easy. Don't rely on production traffic alone for hard cases — it over-represents the happy path.
If you have none: skip to source 2. The tool doesn't require production data to proceed.

↓source 2 — known failures

Source 2 — Bug reports and support tickets

Documented failures from real use

These are cases where the application demonstrably failed and someone noticed. They are good examples of hard cases because a human already judged them wrong.

Upside: grounded in real failure. Directly addresses the known gap between what the application does and what it should do.
Caution: bug reports bias toward the failure modes users know how to describe. Silent failures — the ones nobody reported — won't appear here.

↓source 3 — human judgment

Source 3 — Hand-written cases (5–10)

Cases you wrote because you judged them hard

You write these yourself. The discipline: you should be able to articulate why a case is hard before you include it. If you can't say why, it probably isn't, and it's filling the benchmark without adding signal.

Upside: covers failure modes that don't appear in logs yet — edge cases you know exist, behaviors you're trying to prevent, scenarios that surface only under specific conditions.
Upside: immune to the model-failure-fingerprint problem, because you are selecting on human judgment, not model behavior.
Caution: reflects the failure modes you can imagine. Combine with production data to catch the ones you didn't.
Good test: would two domain experts reach the same verdict on this case? If not, the case is ambiguous and will produce an unstable grader.

↓source 4 — last resort

Source 4 — Synthetic cases

Generated from your codebase, anchored to real examples

Claude generates synthetic cases from your codebase and anchors them to the real examples you provided in sources 1–3. Synthetic cases not anchored in real data measure a hypothetical application. The anchoring is what keeps them honest.

When to use: you have no production logs, no bug reports, and only a handful of hand-written cases — not enough for a stable baseline.
What anchored means: the synthetic cases are generated to resemble the real examples in structure and difficulty, not invented from scratch. They extend the coverage of real data, not replace it.
Risk: synthetic cases can over-represent the easy, well-formed versions of a task. Always mix with real data when possible.

↓

Approval gate

Review page — you confirm the full input set

Claude generates a plain static page showing every input with tags (e.g. billing, easy, ambiguous). You review and confirm before anything is graded. This is the data-first step — the point Hamel flags as where most eval tools go wrong by skipping straight to criteria selection. Nothing moves forward until you have seen the data.

Phase B — Grader selection

Claude proposes the cheapest grader that fits your output type. The two options are not interchangeable — picking the wrong one produces a number that can't be validated.

Programmatic

Output is constrained

Exact match, a label from a fixed set, JSON matching a schema, tests that pass.

Use when the right answer is unambiguous. Fast, cheap, deterministic. The grader agrees with itself by construction.

LLM-as-judge

Output is open-ended

A second model reads the input, the output, and a rubric written as checkable claims — not a 1-to-5 scale. Returns a score with its reasoning.

If you have a baseline to compare against, the judge reads both outputs in random order, without knowing which is the baseline, and picks the better one. The judge model should not be the model you are testing.

The bundled-grader trap. Nothing in the tool prevents multiple failure types from landing in one rubric. Hamel's run produced a single grader covering four distinct failure types. A compound grader produces unmeasurable agreement — a human and a model can agree at the surface for completely different reasons. One grader, one question.

Phase C — Validation and baseline

Grader validation

Claude grades a sample — you review

Claude scores a handful of cases and asks whether you'd have scored any differently. It also runs the grader twice on the same output and reports if the verdict changed. Consistency is necessary, not sufficient — a grader can agree with itself every time and still be wrong. Human review is the real check here.

↓

Diagnostic checks

Three things Claude checks before the baseline

Headroom: if the baseline already scores ~95% or above, Claude warns and suggests the hillclimb focus on cost or latency instead of quality.
Plumbing: checks for timeouts, API errors, and cut-off answers so infrastructure noise doesn't pass as model variance.
Grader consistency: runs the grader twice on identical output; a changed verdict flags a variance problem before it infects the baseline.

↓

Output

Baseline score + results page

You get: the cases, the grader, the runner, one JSON line and one full transcript per case, and a plain page listing each case's score with a link to its transcript. Score is reported with a confidence interval. Now you are ready to hillclimb.

02 /claude-api hillclimb iterate without fooling yourself

Hillclimbing iterates against the evaluation one change at a time. The discipline is keeping the test set genuinely held out, never pasting failures into the prompt, and reverting anything that only improves on training data. When the score stalls, Claude stops proposing changes and diagnoses the remaining failures instead.

Start

/claude-api hillclimb

Claude asks what you want to optimize: performance improvement, or cost reduction while performance holds. For cost goals it looks at prompt caching, model choice, effort level, and prompt compatibility first.

↓

Setup

Split cases at random: train / test

The test set is never seen by the hillclimber. Claude reads only train failures when proposing changes. If a train improvement doesn't hold on the test set, that is the overfitting signal.

↓

Pre-flight

Noise floor check

Before the first round, Claude checks that the eval's variance (how far the score can move by chance) is smaller than the smallest improvement you'd act on. If it isn't, Claude says so and suggests more cases or repeats rather than starting rounds that can't measure anything.

↓loop begins

Each round

Read train failures → propose one change

Claude reads the previous round's train transcripts and proposes one change as a patch. The target is a change whose effect can show above the eval's noise: it rewrites the section that causes the failure or adds a missing rule, rather than rewording a line. What can be changed:

System prompt (rewrites, new rules, migration tables for old API shapes)
Skill or instruction files
Tool descriptions
Model choice, effort level, other API parameters
Harness code (higher risk of overfitting — Claude flags this)

↓

Evaluation

Run eval on both train and test splits

The patched change is evaluated on both splits simultaneously. The test result is the authoritative one — the train result is only a leading indicator.

↓decision

Both improve Keep the patch. Continue to next round.

Train up, test flat Overfitting signal. Revert. The change addressed the specific examples, not the underlying behavior.

Either regresses Revert. The change made things worse somewhere it shouldn't have.

↓

Stall detection

Score stalls for 2–3 rounds → diagnosis

When no single fix can gain more than the eval's noise, Claude stops proposing changes. Instead it reads every remaining train failure and buckets them by root cause. This step makes no edits — it only sorts. What this surfaces:

Ambiguous tasks — cases that fail every run regardless of changes. A sign the task or grader is the problem, not the model.
Grader errors — a grader marking a correct answer wrong, or instructions contradicting the actual API behavior.
Harness bugs — leftover state, timeouts, environment issues that hand the model the answer or deny it context.
Coverage gaps — features present in the skill but untested; or cases where the model is following old trained priors rather than the current API.

Claude may reword tasks, fix graders, or suggest more repeats. Only legitimate failures continue into more rounds.

↓

Done

Leave code at the best test-set version

Claude reports the final test result against the baseline with confidence intervals. If the gain is within noise, it says so and recommends against merging. The code stays at the version that performed best on the held-out test set — not the highest train score.

The overfitting rules — always in force. Never paste failures into the prompt. If Claude reads failing transcripts, the content of those failures must not appear verbatim in the system instructions — that's memorization, not learning. Keep answers structurally out of the model's reach — a model that can find the eval answers (via a public repo, a file left in the harness, a git history) will find them. Split is mandatory — without a held-out set, there is no way to tell improvement from overfitting.

03 The four properties of a good eval design principles behind the tool

These are the principles the tool is built around. An eval that violates any of them will produce misleading hillclimb results — the tool checks for all four before and during a run.

Tasks mirror production

Grade what you actually care about

Tasks picked because they are easy to generate or easy to grade will produce a benchmark that doesn't track production performance. Sample from what the system will actually be used for.

Don't blindly trust user traffic either: users often try what they expect to work, skewing the sample easy. Mix production logs with hand-written hard cases.

Performance scales with capability

Stronger model, higher score

More capable models at higher effort levels should score higher. If they don't, the tasks are ambiguous or the grader is miscalibrated.

This is a sanity check you can run before starting hillclimb. A benchmark that doesn't discriminate between model sizes isn't measuring anything real.

Headroom below perfect

The frontier model should not be at 100%

If the best model at the highest effort already scores ~95% or above, you can't reliably judge whether a change helped. The gap should not be explained by impossible or ambiguous tasks.

A task that fails every run regardless of the model is a broken task, not a hard one. The tool flags this and suggests rewording before hillclimb starts.

Low run-to-run variance

The score should be stable on identical output

High variance usually comes from poorly designed tasks, a grader that answers differently on repeats, or inconsistent effort settings in the harness. Leftover state from a prior run (a file, a git history) can also hand the model the answer.

The tool checks grader consistency before the baseline and flags variance before hillclimb starts.

04 What the tool actually produced two real benchmarks

Two examples from Anthropic's own playbook. Both ran the full build-eval + hillclimb loop.

Customer support benchmark — cost-focused hillclimb

Started on Opus 4.8 at high effort. Hillclimb goal: reduce cost while holding accuracy. 44 tickets total, 30 for training, 14 held out.

78.6%

Starting accuracy (held-out)

Opus 4.8, high effort, 4.6¢ per ticket.

90.5%

Final accuracy (held-out)

Sonnet 5, low effort, ~1¢ per ticket. Roughly one-fifth the starting cost.

The hillclimber's path: audited the prompt first (removed contradictory rules, a mandatory scratchpad, tool-call rituals), stepped down to Opus 5.5 at low effort (cleared the accuracy bar at half the cost), then stepped down again to Sonnet 5. Final prompt added routing rules and a refund-cap cross-reference.

claude-api skill benchmark — performance-focused hillclimb

Eval set derived from API documentation. Testing whether the skill correctly implements code using current API shapes.

66%

Starting pass rate

Skill missing coverage for eight features.

87.9%

Final pass rate

After three phases: missing sections added, type tables fixed, migration table added for old API shapes.

The key diagnostic insight from the stall analysis: the skill content was present, but the model was writing older API shapes from its trained priors. The fix was a migration table near the top of the skill mapping from forms the model remembered to current ones — not more content, but a redirect at the point of failure.

What Hamel found in his run. The tool proposed a single grader bundling four distinct failure types without asking, described it in language that gave no view of the underlying prompt, and when asked to validate labels returned aggregate counts rather than individual cases. His core critique: if an eval tool doesn't put looking at data at the center of your workflow, it's not worth using. The data-first design of build-eval is a direct response to that principle — but the bundled-grader problem requires the user to enforce one-grader-one-question as a discipline the tool won't enforce for them.