PuRDy: AI Agent Quality at Scale
AI Strategy · Build → Evaluate → Repair → Deploy · Q226
We built a system where AI agents are not shipped based on "looks good" -- they are scored against business-defined rubrics, red-teamed across difficulty tiers, and improved through structured repair cycles until they meet a measurable quality bar. PuRDy is the first agent through this process. This is the model for every AI deployment going forward.
Why PRD intake was broken, and what it costs
Employees face a blank 15-page template and either submit incomplete documents or don't submit at all. IS and AI Strategy then spend significant time on clarification calls. This bottleneck scales linearly with request volume -- every new project request costs the same manual effort to get right.
A self-service PRD interview agent + automated formatting pipeline
graph LR
A["Employee has a project idea"] --> B["Opens PuRDy in Glean"]
B --> C["Guided interview\n8 adaptive stages"]
C --> D["Agent classifies project type\nand adjusts depth"]
D --> E["Produces substance-focused PRD"]
E --> F["Paste into Google Doc"]
F --> G["Apps Script formats to\nIS PRD template"]
G --> H["Submission-ready PRD\n+ intake fields"]
classDef user fill:#b4530922,stroke:#b45309,stroke-width:2px
classDef agent fill:#1d4ed822,stroke:#1d4ed8,stroke-width:2px
classDef auto fill:#15803d22,stroke:#15803d,stroke-width:2px
classDef output fill:#7e22ce22,stroke:#7e22ce,stroke-width:2px
class A,B,F user
class C,D,E agent
class G auto
class H output
Self-service interview -- any employee, any time. Classifies the project into 7 types, then adapts interview depth per section. Simple automations get 5-6 questions. Complex platform builds get full coverage across 10 sections.
Automated formatting -- maps PuRDy output to Michael Stratton's official IS PRD template. Generates the cover page, all 9 numbered sections, reviews/approvals tracker, and pre-fills the Jira intake form. Flags gaps with [NEEDS REVIEW] markers.
What changed: Employees no longer face a blank template. They have a conversation. The agent does the requirements thinking. IS receives complete, structured, submission-ready documents.
How we went from v1 to production-ready through structured improvement cycles
We do not build an agent, look at the output, and say "that seems fine." We run it through a rigorous, repeatable improvement cycle where every change is driven by scored evidence, not intuition.
graph TD
AF["AESOP Studio\nBuild initial instructions"] --> V1["Agent v1\nFirst draft instructions"]
V1 --> EVAL["Evaluation Suite\nRun scenario conversations"]
EVAL --> SCORE["Score against rubric\n5 dimensions, 16 criteria"]
SCORE --> REPORT["Suite Report\nWeakness patterns + punch list"]
REPORT --> REPAIR["Repair Session\nTargeted instruction fixes"]
REPAIR --> VN["Agent v(n+1)\nUpdated instructions"]
VN --> EVAL
SCORE -.-> RUBRIC["Business-defined rubric\nAnchored to real IS values"]
EVAL -.-> SCENARIOS["Scenario library\nBasic / Standard / Challenging"]
classDef build fill:#b4530922,stroke:#b45309,stroke-width:2px
classDef eval fill:#1d4ed822,stroke:#1d4ed8,stroke-width:2px
classDef repair fill:#15803d22,stroke:#15803d,stroke-width:2px
classDef ref fill:#7e22ce11,stroke:#7e22ce,stroke-width:1px,stroke-dasharray:5 5
class AF,V1,VN build
class EVAL,SCORE,REPORT eval
class REPAIR repair
class RUBRIC,SCENARIOS ref
Four purpose-built tools working together
Builds initial agent instructions from a PRD and knowledge base. Also runs repair sessions -- takes evaluation findings and produces targeted instruction changes with a detailed changelog. Built by AI Strategy.
Simulates real user conversations with the agent using scenario personas (FP&A analyst, RevOps manager, etc). Scores each conversation against a 16-criteria rubric across 5 dimensions. Produces a suite report with weakness patterns.
Business-defined evaluation criteria anchored to real IS values: Does the PRD have enough substance for ELT to decide? Are stakeholders fully mapped? Are risks surfaced? Scenarios range from basic (simple automation) to challenging (adversarial, ambiguous).
Automated downstream formatting. Parses PuRDy output, maps it to the official IS PRD template, generates a new Google Doc with cover page + 9 sections + intake fields. Flags anything missing. Zero manual formatting effort.
Not "does this look good?" but "does this meet a measurable standard?"
The Rubric: 5 Dimensions, 16 Criteria
Every criterion has anchored scoring -- specific descriptions of what a 1, a 5, and a 10 look like. Scores are not vibes. They are measurements.
- PRD production -- does the artifact exist with all 4 minimum elements?
- Depth calibration -- lean PRD for simple projects, full for complex?
- Problem-first orientation -- describes a problem, not a solution?
- Open questions captured -- real gaps documented, not perfunctory?
- Stage sequence adherence -- 8 stages in order, full depth on Stage 2?
- Classification accuracy -- correct project type, user confirmation?
- Depth matrix application -- right number of questions per section?
- Information state tracking -- knows what's missing, probes to fill gaps?
- Question pacing -- one question per response, natural rhythm?
- Active listening -- references prior answers before next probe?
- Coaching move deployment -- uses prescribed techniques?
- Summary quality -- accurate confirmation before generating PRD?
- Gap and inconsistency surfacing -- catches contradictions, missing stakeholders?
- Scope and routing assessment -- identifies non-IS projects, routes appropriately?
- Content specificity -- names, numbers, systems from the conversation?
- Section structure -- matches expected PRD format?
- Downstream readiness -- IS team can act on this document?
- Synthesis quality -- overview captures the whole story?
Scenario Library: Tiered Difficulty
We don't just test the happy path. Scenarios are designed to stress-test the agent across different project types, user behaviors, and difficulty levels.
Finance Analyst -- simple process automation. User comes prepared with specifics. Tests: depth calibration for lean PRDs, scope discipline, clean interview flow.
RevOps Manager -- Salesforce-NetSuite integration. User leads with solution language, hasn't thought through all details. Tests: solution-first redirect, stakeholder probing, ROI quantification.
Adversarial and ambiguous inputs. Users who resist structure, give conflicting answers, or describe projects that aren't IS scope. Tests: red flag detection, routing, coaching under pressure.
Measurable improvement from v1 through v8+
From "we hope this works" to "we can prove it works"
PRDs contain specific names, numbers, systems, and timelines drawn from the conversation -- not generic summaries. Content specificity scored 8.7/10 in the latest evaluation.
Every PRD follows the same structure regardless of who the user is. The agent applies the same depth matrix, same coaching moves, same quality gates every time. No variability from analyst to analyst.
Scores are stable across different scenario types. The agent doesn't perform well on easy inputs and poorly on hard ones. Standard-tier scenarios score within 0.5 points of basic-tier -- the quality holds under pressure.
Every version has a scored evaluation, a weakness report, and a repair changelog. We can trace any instruction change back to the specific finding that drove it. Full paper trail from v1 to production.
The rubric is defined by people, based on real business values. What does IS actually need in a PRD? What does ELT use to make decisions? Those answers became the scoring criteria. The AI doesn't grade itself.
When someone asks "how do we know this agent is good?" -- we don't say "it seems to work." We show 16 scored criteria, anchored definitions, scenario transcripts, and improvement trajectory. The evidence speaks.
The eval loop doesn't stop at launch
Once PuRDy is live with real users, the same evaluation infrastructure becomes a continuous quality monitoring system.
graph TD
LIVE["PuRDy in Production\nReal user conversations"] --> SAMPLE["Sample real interactions\nPeriodic or triggered"]
SAMPLE --> EVAL["Run through eval rubric\nSame 16 criteria"]
EVAL --> META["Meta-Analysis\nCross-conversation patterns"]
META --> DRIFT{"Prompt drift\ndetected?"}
DRIFT -->|"Yes"| REPAIR["Repair cycle\nTargeted fix"]
DRIFT -->|"No"| MONITOR["Continue monitoring"]
REPAIR --> DEPLOY["Deploy updated instructions"]
DEPLOY --> LIVE
META --> INSIGHTS["Quality insights\nfor IS leadership"]
classDef live fill:#15803d22,stroke:#15803d,stroke-width:2px
classDef eval fill:#1d4ed822,stroke:#1d4ed8,stroke-width:2px
classDef repair fill:#b4530922,stroke:#b45309,stroke-width:2px
classDef insight fill:#7e22ce22,stroke:#7e22ce,stroke-width:2px
class LIVE,DEPLOY live
class SAMPLE,EVAL,META,DRIFT,MONITOR eval
class REPAIR repair
class INSIGHTS insight
- Prompt drift detection -- scores trending down across conversations signal the agent's behavior is drifting from its instructions. Catch it before users notice.
- Meta-analysis across conversations -- are certain project types consistently scoring lower? Are users in certain departments struggling? Pattern recognition at scale.
- Continuous improvement cycles -- the same Build-Eval-Repair loop runs on production data, not just test scenarios. The agent gets better from real usage, not just our imagination of usage.
- Quality reporting for leadership -- IS leadership gets visibility into agent quality metrics over time. Not "it's working" -- but "here are the scores, here are the trends, here's what we fixed this month."
The difference between AI that's deployed and AI that's governed
We are not just shipping AI tools. We are building a quality system.
Anyone can put an agent in Glean and call it done. What we've built is different:
- Rubrics defined by people, not AI -- the business decides what "good" means. Criteria are anchored to what IS actually needs to approve and scope a project.
- Scenarios that stress-test, not just validate -- we don't test with ideal inputs. We test with solution-first language, missing sponsors, vague quantification, adversarial behavior.
- Evidence-based improvement, not intuition -- every change to the agent traces back to a scored finding. We can show exactly why v7 is better than v6.
- Continuous monitoring, not "launch and forget" -- the same eval infrastructure that built the agent continues to monitor it in production.
- Full audit trail -- version history, evaluation reports, repair changelogs, score progression. Complete traceability from first draft to production.
PuRDy is agent #1. This is how we build every AI deployment.
The toolchain and methodology we built for PuRDy is not PuRDy-specific. It's a reusable framework for any AI agent deployment across the organization and beyond.
graph TD
REQ["New agent request\nfrom any team"] --> PRD["Requirements + rubric\ndefined with stakeholders"]
PRD --> BUILD["AESOP Studio\nbuilds initial version"]
BUILD --> LOOP["Eval-Repair Loop\nuntil quality bar met"]
LOOP --> DEPLOY["Deploy to Glean\nwith monitoring"]
DEPLOY --> IMPROVE["Continuous improvement\nfrom real usage data"]
LOOP -.-> PURDY["PuRDy\nPRD interviews"]
LOOP -.-> DEAL["Sales Ops Agent\nsales support"]
LOOP -.-> SEP["Legal Separation Agent\nAgreement generation"]
LOOP -.-> FUTURE["Future agents\nacross teams..."]
classDef process fill:#b4530922,stroke:#b45309,stroke-width:2px
classDef agent fill:#1d4ed822,stroke:#1d4ed8,stroke-width:1px,stroke-dasharray:5 5
class REQ,PRD,BUILD,LOOP,DEPLOY,IMPROVE process
class PURDY,DEAL,SEP,FUTURE agent
Agents already in this pipeline:
- PuRDy (IS) -- PRD interview agent. 9 repair cycles complete. Approaching production.
- Sales Ops Agent -- Sales workflow support. In requirements phase.
- Legal Separation Agent (Legal/HR) -- Agreement generation + severance calculation. In development.
- Discovery PuRDy (Cross-functional) -- Product requirements interviews. Pipeline integration planned.