aesop // 15 methods · 6 stages · 1 hard veto

Evaluation System Lifecycle

These aren’t isolated tools. Each stage feeds the next — KB Audit validates the foundation, Quality Analysis scores instructions, Trace Analysis converts scores into specific failure patterns, three adversarial modes attack from different angles, Regulatory Mapping produces compliance artifacts, Drift Detection catches post-deployment degradation. No single lens catches everything.

Hard Veto Rule. Safety and Bias carry 1.5× weight. If either dimension scores below 75, the composite score caps at 74.9 — no certification, regardless of performance on other dimensions. A brilliant agent that is unsafe does not ship. This veto is human judgment encoded as code.

Lifecycle Map

      graph TD
        subgraph PRE["Pre-Build"]
          KB["KB Audit"]
        end

        subgraph BUILD["Build-Time"]
          QC["Quality Analysis
Quick Check"]
          QA["Quality Analysis
Full · 7 agents"]
          CTS["Custom Test Suite"]
        end

        subgraph FW["First Wave"]
          TR["Trace Analysis"]
        end

        subgraph ADV["Pre-Deployment Adversarial"]
          COS["Cave of Shadows
surgical"]
          HYD["Hydra
20 concurrent heads"]
          BP["Bias Probe
behavioral"]
          BBI["Black Box Invest.
reconstructive"]
        end

        subgraph PE["Post-Evaluation"]
          RM["Regulatory Mapping"]
        end

        subgraph PD["Post-Deployment"]
          DD["Drift Detection"]
        end

        subgraph SPAN["Continuous / Spanning"]
          NERVE["NERVE · public leaderboard"]
          AS["Accuracy Scorer · forensic"]
          SEN["Sentinel · QA plans"]
          CICD["CICD Testing · 842 tests"]
        end

        PRE --> BUILD
        BUILD --> FW
        FW --> ADV
        ADV --> PE
        PE --> PD

        QC -.->|"fast iteration"| QA
        QC -.->|"score handoff"| TR
        QA -.->|"failure patterns"| TR
        TR -.->|"targeted vulns"| COS
        TR -.->|"weak domains"| HYD
        TR -.->|"differential gaps"| BP

        RM -.->|"compliance evidence"| DD

        classDef prebuild fill:#EAF2F9,stroke:#2F6FA8,color:#241A12,stroke-width:1.5px
        classDef build fill:#ECE6FB,stroke:#7C3AED,color:#241A12,stroke-width:1.5px
        classDef firstwave fill:#FAF0DC,stroke:#A5650A,color:#241A12,stroke-width:1.5px
        classDef adversarial fill:#FBEBEA,stroke:#C23A30,color:#241A12,stroke-width:1.5px
        classDef posteval fill:#E4F1EE,stroke:#0E7C6E,color:#241A12,stroke-width:1.5px
        classDef postdeploy fill:#E9E7FB,stroke:#4F46E5,color:#241A12,stroke-width:1.5px
        classDef spanning fill:#F0E9E1,stroke:#8C7E6E,color:#241A12,stroke-width:1.5px
        classDef subg fill:transparent,stroke:#CDBBA8,stroke-dasharray:6 4,color:#8C7E6E

        class KB prebuild
        class QC,QA,CTS build
        class TR firstwave
        class COS,HYD,BP,BBI adversarial
        class RM posteval
        class DD postdeploy
        class NERVE,AS,SEN,CICD spanning
        class PRE,BUILD,FW,ADV,PE,PD,SPAN subg

Pre-Build

Build-Time

First Wave

Adversarial

Post-Evaluation

Post-Deployment

Spanning

Pre-Build→ Build-Time→ First Wave→ Pre-Deployment→ Post-Eval→ Post-Deployment→ Continuous

Each stage feeds the next. Pre-build validates foundations. Build-time scores instructions. Trace Analysis runs as the first wave after build — qualitative coding converts scores into specific, categorized failure patterns before the heavy adversarial testing begins. Three adversarial modes then attack from three different angles: surgical (known vulnerabilities), saturation (20 concurrent heads), and behavioral (paired differential testing). Regulatory Mapping produces compliance artifacts. Drift Detection catches post-deployment degradation. Spanning methods run continuously across the full lifecycle.

Three Adversarial Angles

Cave of Shadows — Surgical

Targets known vulnerabilities identified in Quality Analysis and Trace Analysis. Generates adversarial personas and attack scenarios — prompt injection, jailbreak, social engineering, data extraction. Produces per-scenario pass/fail with full transcripts.

Hydra — Saturation

20 concurrent adversarial heads running adaptive multi-turn conversations. Each head probes from a different angle simultaneously. Throws everything at once and sees what breaks. Weakness-weighted allocation concentrates firepower.

Bias Probe — Behavioral

Paired equivalent scenarios differing only on a protected dimension. Measures differential treatment using behavioral scoring, not sentiment analysis. Does the agent treat a startup founder’s query differently than a GC’s identical query?

Pre-Build — Foundation Validation

1 method

Pre-BuildKB Audit

Scans knowledge base documents for quality issues before any agent is built. Detects readability gaps, cross-document conflicts, terminology inconsistencies, images-stored-as-text, and contradictory guidance. Produces scored report with per-document issues and repair recommendations. 10 standardized issue types.10 issue typespre-build gatescored report

Build-Time — Instruction Quality & Test Design

3 methods

Build-TimeQuality Analysis — Quick Check

Instructions + Safety agents only. ~2 minutes, ~$0.02. Fast feedback during rapid iteration. No composite score — just the two most critical dimensions. Designed for the inner dev loop.~2 min~$0.022 agents

Build-TimeQuality Analysis — Full Analysis

7-agent parallel pipeline: Instructions, Ethics, Bias, Safety, Functionality, Tone, Synthesis. Each scores independently on 100-point scale. Synthesis aggregates into composite with Platinum/Gold/Silver/Bronze certification. Hard veto active.7 agents100-pt scale4-tier cert

Build-TimeCustom Test Suite

Generates tailored rubric with 3 core categories (Safety, Core Function, Structure) plus up to 4 domain-specific categories. Creates test scenarios matched to criteria. For legal: privilege boundaries, jurisdictional variation, confidentiality tests.3–7 categoriestailored rubricdomain-aware

First Wave After Build — Score-to-Fix Translation

1 method

First WaveTrace Analysis

The bridge between “scores are low” and “here’s exactly what to fix.” Qualitative coding of evaluation conversation traces. Annotates specific failure moments, groups them into a taxonomy of failure patterns (axial codes), and produces concrete repair recommendations. Runs first and fast after build — before heavy adversarial testing — so downstream attacks target known weaknesses with precision.qualitative codingaxial codesrepair targetsfirst wave

Pre-Deployment Adversarial — Three Angles of Attack

4 methods

AdversarialCave of Shadows

Surgical adversarial testing targeting known vulnerabilities. Three scenario types: Report-Driven (targeting Quality Analysis findings), Framework-Driven (standard AI safety patterns), and Safety Probes (direct attacks). Produces resilience score with per-scenario pass/fail and full conversation transcripts. Surgical, not saturation.3 scenario typesper-scenario pass/failsurgical

AdversarialHydra

20 concurrent adversarial heads running multi-turn adaptive conversations. Weakness-weighted allocation concentrates firepower on lowest-scoring domains. Each head adapts based on target responses. Saturation, not surgical — throws everything at once and sees what breaks. Domains that improve get fewer heads on subsequent runs.20 headsmulti-turnadaptivesaturation

AdversarialBias Probe

Detects differential treatment using behavioral scoring. Pairs equivalent scenarios that differ only on a protected dimension and measures response divergence. For legal use: does the agent treat a startup founder’s contract query differently than an enterprise GC’s identical query? Behavioral, not sentiment-based.paired testingbehavioral scoringdifferential detection

AdversarialBlack Box Investigation

Structured interview with a chatbot endpoint — no system instructions needed. Reconstructs understanding of what the system actually does. For third-party tools, Harvey custom agents where prompts aren’t accessible, or any system where documented and actual behavior may differ. Reconstructive.no system prompt neededthird-party toolsreconstructive

Post-Evaluation — Compliance Artifacts

1 method

Post-EvalRegulatory Mapping

Pre-populates compliance control templates — validation protocols, change control forms, risk registers — with cited evidence from the agent’s own evaluation history. Turns audit preparation into a generated artifact. For APC: maps evaluation evidence to ABA 512 duty compliance, producing the audit trail that proves the boundary held.ABA 512 mappingauto-generatedaudit trail

Post-Deployment — Continuous Monitoring

1 method

Post-DeployDrift Detection

Continuous monitoring for 6 drift types: tone, scope, refusal, instruction erosion, hallucination, and safety drift. Compares production behavior against baseline instructions. If the model underlying Harvey changes tomorrow, the evaluation pipeline reruns against the same test scenarios and surfaces what broke before the client notices.6 drift typescontinuousmodel-change detection

Continuous / Spanning — Running Across the Full Lifecycle

4 methods

SpanningNERVE

Public leaderboard evaluation system. Multi-turn probing on human-centered dimensions: consistency, nuance, edge cases. NERVE Director generates follow-up probes; NERVE Content evaluates responses; NERVE Scorer produces dimensional scores; NERVE Findings synthesize results. Curious and persistent — distinct from adversarial Cave of Shadows.public leaderboardmulti-turn4 sub-agents

SpanningAccuracy Scorer

Forensic citation verification. Claim-by-claim document comparison. Decomposes agent responses into discrete factual claims, then exhaustively searches source documents for support or contradiction. Catches cherry-picked citations, omitted conditions, and approximate numbers presented as exact. Essential for legal use cases.claim decompositionsource verificationforensic

SpanningSentinel Agent

QA testing plan generator. Produces structured, actionable testing plans that validate AI agent behavior after evaluation and repair. Generates both human-readable QA documentation and executable test scenarios in Cave of Shadows format. Bridges evaluation findings to ongoing quality assurance.QA planshuman + machine readableCoS-compatible

SpanningCICD Testing Pipeline

842 backend pytest tests, frontend Playwright + Jest. Automated testing integrated into the development pipeline. Deterministic test infrastructure ensuring the evaluation platform itself works correctly. Not AI evaluation per se — the engineering foundation everything else runs on.842 pytestPlaywrightJestCI/CD

Why This Matters for APC

APC published its AI governance framework — 5 principles — on day one. The principles are clear. What the firm needs is the operationalization: the evaluation pipeline that proves, with evidence, that every agent adheres to those principles before it touches a client matter. These 15 methods aren’t a philosophy. They’re a working system — architected, built, and tested. For a firm that proved it will go to constitutional war to protect client confidentiality, governance isn’t a policy memo. It’s engineered into every stage of the agent lifecycle. That’s the standard the role demands. That’s the standard this system enforces.