AESOP OS / EVAL TOOLCHAIN

AESOP → Hillclimb Handoff

AESOP evaluates your agent and produces a structured evidence pack — cases, baseline stats, failure clusters, and a grader rubric — that feeds directly into /claude-api hillclimb.

One export closes the loop between measurement and improvement.

The Problem

AESOP produces

Per-scenario verdicts — pass / fail / inconclusive, with scores

Root-cause clusters — dominant failure patterns named and assigned

Rubric criteria — six dimensions as checkable claims

Baseline score — pass rate, mean, confidence intervals

/claude-api hillclimb wants

Labeled cases — verdict + split assignment per scenario

Failure summary — what failed and why, before round 1

A grader — checkable yes/no claims for the judge prompt

A baseline — score + CI to measure improvement against

Export + Hillclimb Pack

One button. One zip. One hillclimb/ directory. Without it, you re-derive all of the above by hand.

The Pack

cases.jsonl

One line per scenario from the baseline run.

scenario_name, dimension, verdict, score
failure_bucket from AESOP clustering (null if not run)
split: train or test — 70/30, seeded by run ID

baseline.json

Baseline statistics over all cases.

pass_rate with Wilson 95% CI
mean_score with standard error CI
n_train, n_test, warning when n < 6

grader.md

AESOP's rubric as checkable yes/no claims.

Six dimensions: Instructions, Ethos, Bias, Safety, Legal, InfoSec
Each expressed as a set of yes/no claims a judge can verify
Ready to paste as a judge prompt into the hillclimber

failures.json

Root-cause clusters from AESOP's failure analysis.

Dominant failure patterns with scenario assignments
Present only when AESOP's clustering ran on this run
Gives the hillclimber a starting point before round 1

README.md

How to use the pack in a Claude Code session.

Drop the archive into a Claude Code session
Run /claude-api hillclimb, point it at hillclimb/cases.jsonl and hillclimb/baseline.json
The train/test split is pre-assigned — use it directly

How It Works

AESOP Evaluation

Scenarios scored across 6 dimensions. Failures clustered by root cause.

↓

Export + Hillclimb Pack

hillclimb/ directory assembled. Split seeded by run ID.

↓

hillclimb/

cases.jsonl • baseline.json • grader.md • failures.json • README.md

↓

/claude-api hillclimb

Reads cases + grader + baseline. Iterates one patch per round.

↓

Train ↑ + Test flat → Revert

Train ↑ + Test ↑ → Keep

↓

Improved Instructions

Best test-set version. Confidence intervals. No blind hillclimb.

Overfitting Guard

The hillclimber reads train transcripts to propose a change. It never looks at test cases until it decides whether to keep the patch.

A change that lifts train but leaves test flat is reverted — that is the overfitting check. Without it, you optimise for the examples you have, not for the agent you want.

AESOP pre-assigns the split: 70% train, 30% test, seeded by the evaluation run ID. The same run always produces the same split. Re-export the archive next week and the split is identical — no accidental reassignment between rounds.

70% train cases

30% test cases

Seeded by run ID

Stable across re-exports

Min. 6 cases recommended

vs. Build-Eval

/claude-api build-eval

Start from zero

No existing eval set required
Interviews you: prod traffic, bug reports, hand-written cases
Validates the grader before use
Runs a baseline and prints confidence intervals
Right tool when you do not yet have an eval

AESOP Hillclimb Pack

Start from a completed AESOP run

Cases are already scored by a multi-dimensional judge
Grader is derived from AESOP's rubric — no design step
Failure clusters already name the dominant failure modes
Baseline stats already computed with CI
Right tool when AESOP has already run