Instruction repair — AESOP

Repair

Automatically repair AI system instructions from evaluation findings, with version tracking and regression detection.

How to access it

Repair is available after any completed evaluation that included system instructions — from the evaluation report page or the dedicated Repair list in the sidebar.

1From the
report page

After an evaluation completes, click “Repair” on the report page to go straight to Repair for that evaluation.

2From the
Repair list

Open the “Repair” page in the sidebar to see every evaluation eligible for repair; click any to open its Repair page.

3Fix
regression

If a re-evaluation after repair shows a score drop, a red “Fix Regression” button appears on the report page, launching Repair in regression mode with context about what went wrong.

Key features

CoreEvidence-based
repair

Every change traces to a specific finding from evaluation reports. The Repair agent uses extended thinking to plan minimal, targeted fixes rather than rewriting from scratch.

TrackingVersion
lineage

Each repair increments the instruction version (v1.0, v1.1, v1.2). Versions are tracked across evaluations so you can see how scores change over time on the Agent Trajectory page.

Safety netRegression
detection

When a re-evaluation scores lower than the previous version, it is flagged as a regression. Repair can then run in regression mode, analyzing what went wrong and trying a different approach.

Output modes

DefaultFull
rewrite

Produces the complete revised instructions with all changes applied inline. Best when you want a ready-to-use replacement for your current system prompt.

AdvancedSurgical
patches

Shows only the changed sections as before/after patches with location references. Best when you want to review and apply changes selectively.

Repair output

Every Repair run produces two artifacts.

Artifact 1Repaired
instructions

The revised system instructions (full rewrite or patches, per mode). Format is preserved — XML in, XML out; markdown headers in, markdown headers out.

Artifact 2Changelog

instruction_changes lists every modification with the source finding, what changed, and why. knowledge_base_recommendations lists issues that can’t be fixed in instructions alone and suggests actions like adding documents or updating retrieval configuration.

Optional inputTiger Team
findings

Paste stakeholder feedback from the Launch Package’s Tiger Team review into the repair page. Repair treats it as real-world user-testing data, surfacing usability and adoption issues automated evaluation cannot detect. Tiger Team findings are weighted alongside evaluation reports when planning changes.

Version lifecycle

1Evaluate

Run an evaluation against your agent’s system instructions. The first evaluation establishes v1.0.

2Repair

Run Repair to produce repaired instructions (v1.1). The changelog documents every change.

3Re-evaluate

Apply the repaired instructions and run a new evaluation. The system computes the score delta (+3.2, −1.5) and flags regressions.

4Iterate

Continue the cycle. Each repair increments the minor version. The Agent Trajectory page shows the full history — scores, deltas, certification levels — across all versions.

The repair pipeline

InputCompleted
evaluation

Original instructions + specialist reports + synthesis + scenario results.

AgentRepair agent

v1.0 — extended thinking Analyzes findings, classifies each as instruction-fixable or knowledge-base, plans minimal changes.

Fork — outputTwo
artifacts

Repaired instructions Full rewrite or surgical patches, preserving original format and voice.

Changelog instruction_changes + knowledge_base_recommendations with finding traceability.

SystemVersion
increment

v1.0 → v1.1 Server-side version detection and increment across multiple format patterns.

Next stepRe-evaluate

Apply repaired instructions to the agent and run a new evaluation to measure improvement.

AutomaticScore
comparison

The system computes the score delta and flags regressions by comparing against the previous version.

Regression detected? If the new score is lower than the previous version, the evaluation is flagged as a regression. A “Fix Regression” button appears on the report page, launching Repair in regression mode with analysis of what the previous repair changed and why it hurt.

Regression repair pipeline

TriggerRegression
detected

Previous repair changelog + current evaluation reports + the score delta showing decline.

AgentRepair agent
— regression mode

Analyzes which previous changes caused the score drop, reverts or refines the harmful changes.

OutputRegression analysis
+ repaired instructions

Includes reverted_changes, root_cause, and new_approach alongside the corrected instructions.

Regression mode forces the Repair agent to explicitly acknowledge what went wrong. The changelog references the specific changes from the previous repair that caused the decline, explains the root cause, and documents the alternative approach taken.