AI Strategy · Build → Evaluate → Repair → Deploy · Q226

PuRDy — AI Agent Quality at Scale

We built a system where AI agents are not shipped based on “looks good” — they are scored against business-defined rubrics, red-teamed across difficulty tiers, and improved through structured repair cycles until they meet a measurable quality bar. PuRDy is the first agent through this process. This is the model for every AI deployment going forward.

The Problem We Solved

Why PRD intake was broken, and what it costs

Page PRD Template

Required Sections

60%+

Incomplete Submissions

Linear

Bottleneck Scaling

Employees face a blank 15-page template and either submit incomplete documents or don’t submit at all. IS and AI Strategy then spend significant time on clarification calls. This bottleneck scales linearly with request volume — every new project request costs the same manual effort to get right.

The deeper issue: Incomplete requirements don’t just slow IS down — they lead to misscoped projects, scope creep, and rework. The cost isn’t measured in hours spent on intake; it’s measured in projects that go wrong because they started with bad requirements.

What PuRDy Is

A self-service PRD interview agent + automated formatting pipeline

User step Agent step Automation Output

Glean AgentPuRDy

Self-service interview — any employee, any time. Classifies the project into 7 types, then adapts interview depth per section. Simple automations get 5-6 questions. Complex platform builds get full coverage across 10 sections.

Google ScriptsApps Script

Automated formatting — maps PuRDy output to Michael Stratton’s official IS PRD template. Generates the cover page, all 9 numbered sections, reviews/approvals tracker, and pre-fills the Jira intake form. Flags gaps with [NEEDS REVIEW] markers.

What changed: Employees no longer face a blank template. They have a conversation. The agent does the requirements thinking. IS receives complete, structured, submission-ready documents.

The Build-Eval-Repair Loop

How we went from v1 to production-ready through structured improvement cycles

We do not build an agent, look at the output, and say “that seems fine.” We run it through a rigorous, repeatable improvement cycle where every change is driven by scored evidence, not intuition.

Build Evaluate / Score Repair Reference

Each cycle is the same: Run evaluation scenarios against the current agent version. Score every dimension. Generate a weakness report with specific, prioritized findings. Run a repair session that produces targeted instruction changes. Deploy the new version. Repeat. No guesswork. No “let’s just tweak this and see.”

The Agentic Toolchain

Four purpose-built tools working together

AESOP
Studio

BuildRepairBuilds initial agent instructions from a PRD and knowledge base. Also runs repair sessions — takes evaluation findings and produces targeted instruction changes with a detailed changelog. Built by AI Strategy.

Evaluation
Engine

EvaluateScoreSimulates real user conversations with the agent using scenario personas (FP&A analyst, RevOps manager, etc). Scores each conversation against a 16-criteria rubric across 5 dimensions. Produces a suite report with weakness patterns.

Rubric +
Scenarios

DefineRed TeamBusiness-defined evaluation criteria anchored to real IS values: Does the PRD have enough substance for ELT to decide? Are stakeholders fully mapped? Are risks surfaced? Scenarios range from basic (simple automation) to challenging (adversarial, ambiguous).

Apps Script
Pipeline

FormatDeliverAutomated downstream formatting. Parses PuRDy output, maps it to the official IS PRD template, generates a new Google Doc with cover page + 9 sections + intake fields. Flags anything missing. Zero manual formatting effort.

How Evaluations Work

Not “does this look good?” but “does this meet a measurable standard?”

The Rubric: 5 Dimensions, 16 Criteria

Every criterion has anchored scoring — specific descriptions of what a 1, a 5, and a 10 look like. Scores are not vibes. They are measurements.

25% weightGoal
Completion

PRD production — does the artifact exist with all 4 minimum elements?
Depth calibration — lean PRD for simple projects, full for complex?
Problem-first orientation — describes a problem, not a solution?
Open questions captured — real gaps documented, not perfunctory?

20% weightProcess
Quality

Stage sequence adherence — 8 stages in order, full depth on Stage 2?
Classification accuracy — correct project type, user confirmation?
Depth matrix application — right number of questions per section?
Information state tracking — knows what’s missing, probes to fill gaps?

10% weightConversational
Skill

Question pacing — one question per response, natural rhythm?
Active listening — references prior answers before next probe?
Coaching move deployment — uses prescribed techniques?
Summary quality — accurate confirmation before generating PRD?

20% weightRed Flag
Detection

Gap and inconsistency surfacing — catches contradictions, missing stakeholders?
Scope and routing assessment — identifies non-IS projects, routes appropriately?

25% weightOutput
Quality

Content specificity — names, numbers, systems from the conversation?
Section structure — matches expected PRD format?
Downstream readiness — IS team can act on this document?
Synthesis quality — overview captures the whole story?

Scenario Library: Tiered Difficulty

We don’t just test the happy path. Scenarios are designed to stress-test the agent across different project types, user behaviors, and difficulty levels.

Basic
Tier

Finance Analyst — simple process automation. User comes prepared with specifics. Tests: depth calibration for lean PRDs, scope discipline, clean interview flow.

Standard
Tier

RevOps Manager — Salesforce-NetSuite integration. User leads with solution language, hasn’t thought through all details. Tests: solution-first redirect, stakeholder probing, ROI quantification.

Challenging
Tier

Adversarial and ambiguous inputs. Users who resist structure, give conflicting answers, or describe projects that aren’t IS scope. Tests: red flag detection, routing, coaching under pressure.

Score Progression Across Versions

Measurable improvement from v1 through v8+

Evaluation Scores by Dimension Across Repair Cycles — each dimension weighted, scored 1-10, tracked v1 through v8.

Version	Goal Completion (25%)	Process Quality (20%)	Conversational Skill (10%)	Red Flag Detection (20%)	Output Quality (25%)
v1	4.5	4.25	5.0	4.0	5.0
v2	7.5	7.0	7.0	6.0	7.5
v3	8.2	7.8	7.5	6.5	8.0
v4	8.8	8.5	8.7	7.3	8.7
v5	8.5	8.3	8.3	7.3	8.7
v6	8.5	8.0	8.0	7.3	8.3
v7	8.8	8.0	8.0	8.0	8.3
v8	8.8	8.8	8.7	8.0	8.7

v1 — Initial Build

AESOP Studio generates first instructions
Weighted average: ~4.5/10. Hallucinated routing details, no classification confirmation, missing metadata section, no executive summary. Functional but not IS-ready.

v2 — First Repair Cycle

Eliminated hallucinations, added classification gate
Fixed fabricated contact names and Slack channels. Added confirmation gate before proceeding. Added metadata and exec summary sections. One-question-per-response rule added.

v3 — Template Alignment

Closed structural gaps against IS PRD template
Added “Why Now” urgency question, constraints capture, Enablement section. Agent now covers ~95% of Stratton’s template requirements. Output went from ~60% to ~95% template coverage.

v4-v5 — Coaching Precision

Targeted conversational discipline
Solution-first redirect enforcement. Multi-question batching eliminated. End-user perspective probing added. Red flag detection improved from 7.3 to 8.0.

v6-v8 — Cross-Scenario Hardening

Tested against Standard tier, tightened coaching moves
Vacation test coaching move mandated. Active listening callbacks required. Metadata pre-verification step. ROI arithmetic validation. Named routing destinations instead of generic advice. Scores stabilized at 8.0+ across all dimensions.

What the Scores Mean for the Business

From “we hope this works” to “we can prove it works”

4.5

v1 Average

8.2

Latest Average

+82% improvement

Repair Cycles

30+

Findings Resolved

Quality

PRDs contain specific names, numbers, systems, and timelines drawn from the conversation — not generic summaries. Content specificity scored 8.7/10 in the latest evaluation.

Consistency

Every PRD follows the same structure regardless of who the user is. The agent applies the same depth matrix, same coaching moves, same quality gates every time. No variability from analyst to analyst.

Predictability

Scores are stable across different scenario types. The agent doesn’t perform well on easy inputs and poorly on hard ones. Standard-tier scenarios score within 0.5 points of basic-tier — the quality holds under pressure.

Auditability

Every version has a scored evaluation, a weakness report, and a repair changelog. We can trace any instruction change back to the specific finding that drove it. Full paper trail from v1 to production.

Governance

The rubric is defined by people, based on real business values. What does IS actually need in a PRD? What does ELT use to make decisions? Those answers became the scoring criteria. The AI doesn’t grade itself.

Explainability

When someone asks “how do we know this agent is good?” — we don’t say “it seems to work.” We show 16 scored criteria, anchored definitions, scenario transcripts, and improvement trajectory. The evidence speaks.

Post-Deploy: Continuous Monitoring

The eval loop doesn’t stop at launch

Once PuRDy is live with real users, the same evaluation infrastructure becomes a continuous quality monitoring system.

In production Evaluation Repair Insight

Prompt drift detection — scores trending down across conversations signal the agent’s behavior is drifting from its instructions. Catch it before users notice.
Meta-analysis across conversations — are certain project types consistently scoring lower? Are users in certain departments struggling? Pattern recognition at scale.
Continuous improvement cycles — the same Build-Eval-Repair loop runs on production data, not just test scenarios. The agent gets better from real usage, not just our imagination of usage.
Quality reporting for leadership — IS leadership gets visibility into agent quality metrics over time. Not “it’s working” — but “here are the scores, here are the trends, here’s what we fixed this month.”

Why This Matters

The difference between AI that’s deployed and AI that’s governed

We are not just shipping AI tools. We are building a quality system.

Anyone can put an agent in Glean and call it done. What we’ve built is different:

Rubrics defined by people, not AI — the business decides what “good” means. Criteria are anchored to what IS actually needs to approve and scope a project.
Scenarios that stress-test, not just validate — we don’t test with ideal inputs. We test with solution-first language, missing sponsors, vague quantification, adversarial behavior.
Evidence-based improvement, not intuition — every change to the agent traces back to a scored finding. We can show exactly why v7 is better than v6.
Continuous monitoring, not “launch and forget” — the same eval infrastructure that built the agent continues to monitor it in production.
Full audit trail — version history, evaluation reports, repair changelogs, score progression. Complete traceability from first draft to production.

The Model Going Forward

PuRDy is agent #1. This is how we build every AI deployment.

The toolchain and methodology we built for PuRDy is not PuRDy-specific. It’s a reusable framework for any AI agent deployment across the organization and beyond.

Process Agents in pipeline

For every future deployment: Define what “good” looks like with the business stakeholders (rubric). Build with AESOP Studio. Eval with tiered scenarios. Repair until the quality bar is met. Deploy with monitoring. Improve continuously. The same loop, the same rigor, the same auditability — regardless of the domain.

Agents already in this pipeline:

PuRDy (IS) — PRD interview agent. 9 repair cycles complete. Approaching production.
Sales Ops Agent — Sales workflow support. In requirements phase.
Legal Separation Agent (Legal/HR) — Agreement generation + severance calculation. In development.
Discovery PuRDy (Cross-functional) — Product requirements interviews. Pipeline integration planned.

Built with AESOP Studio, Glean, AESOP Evaluation Engine.