Automatically repair AI system instructions using evaluation findings, with version tracking and regression detection
How to Access
Repair is available after any completed evaluation that included system instructions. You can access it from the evaluation report page or from the dedicated Repair list page in the sidebar.
1
From the Report Page
After an evaluation completes, click the "Repair" button on the report page. This takes you directly to the Repair repair page for that evaluation.
2
From the Repair List
Navigate to the "Repair" page in the sidebar to see all evaluations eligible for repair. Click any evaluation to open the Repair repair page.
3
Fix Regression
If a re-evaluation after repair shows a score drop, a red "Fix Regression" button appears on the report page. This launches Repair in regression mode with context about what went wrong.
Key Features
Core
Evidence-Based Repair
Every change traces to a specific finding from evaluation reports. The Repair agent uses extended thinking to plan minimal, targeted fixes rather than rewriting from scratch.
Tracking
Version Lineage
Each repair increments the instruction version (v1.0, v1.1, v1.2). Versions are tracked across evaluations so you can see how scores change over time on the Agent Trajectory page.
Safety Net
Regression Detection
When a re-evaluation scores lower than the previous version, it is flagged as a regression. Repair can then run in regression mode, analyzing what went wrong and trying a different approach.
Output Modes
Default
FR
Full Rewrite
Produces the complete revised instructions with all changes applied inline. Best when you want a ready-to-use replacement for your current system prompt.
Advanced
SP
Surgical Patches
Shows only the changed sections as before/after patches with location references. Best when you want to review and apply changes selectively.
Repair Output
Every Repair repair produces two artifacts:
Repaired Instructions -- The revised system instructions (full rewrite or patches depending on mode). Format is preserved: if your input uses XML tags, the output uses XML tags. If it uses markdown headers, output uses markdown headers.
Changelog -- Two sections: instruction_changes lists every modification with the source finding, what changed, and why. knowledge_base_recommendations lists issues that cannot be fixed in instructions alone and suggests actions like adding documents or updating retrieval configuration.
Tiger Team Findings (optional input) -- Paste stakeholder feedback from the Launch Package's Tiger Team review into the repair page. Repair treats these as real-world user testing data, surfacing usability and adoption issues that automated evaluation cannot detect. Tiger Team findings are weighted alongside evaluation reports when planning changes.
Version Lifecycle
Evaluate -- Run an evaluation against your agent's system instructions. The first evaluation establishes version v1.0.
Repair -- Run Repair to produce repaired instructions (v1.1). The changelog documents every change.
Re-evaluate -- Apply the repaired instructions and run a new evaluation. The system automatically computes the score delta (+3.2, -1.5) and flags regressions.
Iterate -- Continue the cycle. Each repair increments the minor version. The Agent Trajectory page shows the full history with scores, deltas, and certification levels across all versions.
• Repair Pipeline
Input
Source
Completed Evaluation
Original instructions + specialist reports + synthesis + scenario results
Agent
Repair Agent
v1.0 -- Extended Thinking
Analyzes findings, classifies each as instruction-fixable or knowledge-base, plans minimal changes
Output
Artifact
Repaired Instructions
Full rewrite or surgical patches preserving original format and voice
Artifact
Changelog
instruction_changes + knowledge_base_recommendations with finding traceability
System
Version Increment
v1.0 -> v1.1
Server-side version detection and increment across multiple format patterns
Next Step
Re-evaluate
Apply repaired instructions to the agent and run a new evaluation to measure improvement
Automatic
Score Comparison
System computes score delta and flags regressions by comparing against previous version
Regression detected? If the new score is lower than the previous version, the evaluation is flagged as a regression. A "Fix Regression" button appears on the report page, launching Repair in regression mode with analysis of what the previous repair changed and why it hurt.
Analyzes which previous changes caused the score drop, reverts or refines the harmful changes
Output
Regression Analysis + Repaired Instructions
Includes reverted_changes, root_cause, and new_approach alongside the corrected instructions
Regression mode forces the Repair agent to explicitly acknowledge what went wrong. The changelog references the specific changes from the previous repair that caused the decline, explains the root cause, and documents the alternative approach taken.