aesop // 15 methods · 6 stages · 1 hard veto
These aren't isolated tools. Each stage feeds the next — KB Audit validates the foundation, Quality Analysis scores instructions, Trace Analysis converts scores into specific failure patterns, three adversarial modes attack from different angles, Regulatory Mapping produces compliance artifacts, Drift Detection catches post-deployment degradation. No single lens catches everything.
Safety and Bias carry 1.5× weight.
If either dimension scores below 75, the composite score caps at 74.9 —
no certification, regardless of performance on other dimensions.
A brilliant agent that is unsafe does not ship. This veto is human judgment encoded as code.
graph TD
subgraph PRE["Pre-Build"]
KB["KB Audit"]
end
subgraph BUILD["Build-Time"]
QC["Quality Analysis
Quick Check"]
QA["Quality Analysis
Full · 7 agents"]
CTS["Custom Test Suite"]
end
subgraph FW["First Wave"]
TR["Trace Analysis"]
end
subgraph ADV["Pre-Deployment Adversarial"]
COS["Cave of Shadows
surgical"]
HYD["Hydra
20 concurrent heads"]
BP["Bias Probe
behavioral"]
BBI["Black Box Invest.
reconstructive"]
end
subgraph PE["Post-Evaluation"]
RM["Regulatory Mapping"]
end
subgraph PD["Post-Deployment"]
DD["Drift Detection"]
end
subgraph SPAN["Continuous / Spanning"]
NERVE["NERVE · public leaderboard"]
AS["Accuracy Scorer · forensic"]
SEN["Sentinel · QA plans"]
CICD["CICD Testing · 842 tests"]
end
PRE --> BUILD
BUILD --> FW
FW --> ADV
ADV --> PE
PE --> PD
QC -.->|"fast iteration"| QA
QC -.->|"score handoff"| TR
QA -.->|"failure patterns"| TR
TR -.->|"targeted vulns"| COS
TR -.->|"weak domains"| HYD
TR -.->|"differential gaps"| BP
RM -.->|"compliance evidence"| DD
classDef prebuild fill:#1a2e2e,stroke:#22d3ee,color:#cffafe,stroke-width:1.5px
classDef build fill:#241a35,stroke:#a78bfa,color:#e8e0f0,stroke-width:1.5px
classDef firstwave fill:#2a2618,stroke:#fbbf24,color:#fef3c7,stroke-width:1.5px
classDef adversarial fill:#2a1a1a,stroke:#f87171,color:#fde0e0,stroke-width:1.5px
classDef posteval fill:#1a2a1a,stroke:#4ade80,color:#dcfce7,stroke-width:1.5px
classDef postdeploy fill:#1a2828,stroke:#2dd4bf,color:#ccfbf1,stroke-width:1.5px
classDef spanning fill:#1a1830,stroke:#c084fc,color:#ede0ff,stroke-width:1.5px
classDef subg fill:transparent,stroke:#3a3050,stroke-dasharray:6 4,color:#9b8fb8
class KB prebuild
class QC,QA,CTS build
class TR firstwave
class COS,HYD,BP,BBI adversarial
class RM posteval
class DD postdeploy
class NERVE,AS,SEN,CICD spanning
class PRE,BUILD,FW,ADV,PE,PD,SPAN subg
Each stage feeds the next. Pre-build validates foundations. Build-time scores instructions. Trace Analysis runs as the first wave after build — qualitative coding converts scores into specific, categorized failure patterns before the heavy adversarial testing begins. Three adversarial modes then attack from three different angles: surgical (known vulnerabilities), saturation (20 concurrent heads), and behavioral (paired differential testing). Regulatory Mapping produces compliance artifacts. Drift Detection catches post-deployment degradation. Spanning methods run continuously across the full lifecycle.
Targets known vulnerabilities identified in Quality Analysis and Trace Analysis. Generates adversarial personas and attack scenarios — prompt injection, jailbreak, social engineering, data extraction. Produces per-scenario pass/fail with full transcripts.
20 concurrent adversarial heads running adaptive multi-turn conversations. Each head probes from a different angle simultaneously. Throws everything at once and sees what breaks. Weakness-weighted allocation concentrates firepower.
Paired equivalent scenarios differing only on a protected dimension. Measures differential treatment using behavioral scoring, not sentiment analysis. Does the agent treat a startup founder's query differently than a GC's identical query?
Scans knowledge base documents for quality issues before any agent is built. Detects readability gaps, cross-document conflicts, terminology inconsistencies, images-stored-as-text, and contradictory guidance. Produces scored report with per-document issues and repair recommendations. 10 standardized issue types.
Instructions + Safety agents only. ~2 minutes, ~$0.02. Fast feedback during rapid iteration. No composite score — just the two most critical dimensions. Designed for the inner dev loop.
7-agent parallel pipeline: Instructions, Ethics, Bias, Safety, Functionality, Tone, Synthesis. Each scores independently on 100-point scale. Synthesis aggregates into composite with Platinum/Gold/Silver/Bronze certification. Hard veto active.
Generates tailored rubric with 3 core categories (Safety, Core Function, Structure) plus up to 4 domain-specific categories. Creates test scenarios matched to criteria. For legal: privilege boundaries, jurisdictional variation, confidentiality tests.
The bridge between "scores are low" and "here's exactly what to fix." Qualitative coding of evaluation conversation traces. Annotates specific failure moments, groups them into a taxonomy of failure patterns (axial codes), and produces concrete repair recommendations. Runs first and fast after build — before heavy adversarial testing — so downstream attacks target known weaknesses with precision.
Surgical adversarial testing targeting known vulnerabilities. Three scenario types: Report-Driven (targeting Quality Analysis findings), Framework-Driven (standard AI safety patterns), and Safety Probes (direct attacks). Produces resilience score with per-scenario pass/fail and full conversation transcripts. Surgical, not saturation.
20 concurrent adversarial heads running multi-turn adaptive conversations. Weakness-weighted allocation concentrates firepower on lowest-scoring domains. Each head adapts based on target responses. Saturation, not surgical — throws everything at once and sees what breaks. Domains that improve get fewer heads on subsequent runs.
Detects differential treatment using behavioral scoring. Pairs equivalent scenarios that differ only on a protected dimension and measures response divergence. For legal use: does the agent treat a startup founder's contract query differently than an enterprise GC's identical query? Behavioral, not sentiment-based.
Structured interview with a chatbot endpoint — no system instructions needed. Reconstructs understanding of what the system actually does. For third-party tools, Harvey custom agents where prompts aren't accessible, or any system where documented and actual behavior may differ. Reconstructive.
Pre-populates compliance control templates — validation protocols, change control forms, risk registers — with cited evidence from the agent's own evaluation history. Turns audit preparation into a generated artifact. For APC: maps evaluation evidence to ABA 512 duty compliance, producing the audit trail that proves the boundary held.
Continuous monitoring for 6 drift types: tone, scope, refusal, instruction erosion, hallucination, and safety drift. Compares production behavior against baseline instructions. If the model underlying Harvey changes tomorrow, the evaluation pipeline reruns against the same test scenarios and surfaces what broke before the client notices.
Public leaderboard evaluation system. Multi-turn probing on human-centered dimensions: consistency, nuance, edge cases. NERVE Director generates follow-up probes; NERVE Content evaluates responses; NERVE Scorer produces dimensional scores; NERVE Findings synthesize results. Curious and persistent — distinct from adversarial Cave of Shadows.
Forensic citation verification. Claim-by-claim document comparison. Decomposes agent responses into discrete factual claims, then exhaustively searches source documents for support or contradiction. Catches cherry-picked citations, omitted conditions, and approximate numbers presented as exact. Essential for legal use cases.
QA testing plan generator. Produces structured, actionable testing plans that validate AI agent behavior after evaluation and repair. Generates both human-readable QA documentation and executable test scenarios in Cave of Shadows format. Bridges evaluation findings to ongoing quality assurance.
842 backend pytest tests, frontend Playwright + Jest. Automated testing integrated into the development pipeline. Deterministic test infrastructure ensuring the evaluation platform itself works correctly. Not AI evaluation per se — the engineering foundation everything else runs on.
APC published its AI governance framework — 5 principles — on day one. The principles are clear. What the firm needs is the operationalization: the evaluation pipeline that proves, with evidence, that every agent adheres to those principles before it touches a client matter. These 15 methods aren't a philosophy. They're a working system — architected, built, and tested. For a firm that proved it will go to constitutional war to protect client confidentiality, governance isn't a policy memo. It's engineered into every stage of the agent lifecycle. That's the standard the role demands. That's the standard this system enforces.