PuRDy: AI Agent Quality at Scale

AI Strategy · Build → Evaluate → Repair → Deploy · Q226

We built a system where AI agents are not shipped based on “looks good” -- they are scored against business-defined rubrics, red-teamed across difficulty tiers, and improved through structured repair cycles until they meet a measurable quality bar. PuRDy is the first agent through this process. This is the model for every AI deployment going forward.

1 -- The Problem We Solved

Why PRD intake was broken, and what it costs

Page PRD Template

Required Sections

60%+

Incomplete Submissions

Linear

Bottleneck Scaling

Employees face a blank 15-page template and either submit incomplete documents or don’t submit at all. IS and AI Strategy then spend significant time on clarification calls. This bottleneck scales linearly with request volume -- every new project request costs the same manual effort to get right.

The deeper issue: Incomplete requirements don’t just slow IS down -- they lead to misscoped projects, scope creep, and rework. The cost isn’t measured in hours spent on intake; it’s measured in projects that go wrong because they started with bad requirements.

2 -- What PuRDy Is

A self-service PRD interview agent + automated formatting pipeline

        graph LR
          A["Employee has a project idea"] --> B["Opens PuRDy in Glean"]
          B --> C["Guided interview\n8 adaptive stages"]
          C --> D["Agent classifies project type\nand adjusts depth"]
          D --> E["Produces substance-focused PRD"]
          E --> F["Paste into Google Doc"]
          F --> G["Apps Script formats to\nIS PRD template"]
          G --> H["Submission-ready PRD\n+ intake fields"]

          classDef user fill:rgba(79,70,229,0.14),stroke:#4F46E5,stroke-width:2px
          classDef agent fill:rgba(47,111,168,0.13),stroke:#2F6FA8,stroke-width:2px
          classDef auto fill:rgba(14,124,110,0.13),stroke:#0E7C6E,stroke-width:2px
          classDef output fill:rgba(124,58,237,0.11),stroke:#7C3AED,stroke-width:2px

          class A,B,F user
          class C,D,E agent
          class G auto
          class H output

Glean Agent (PuRDy)

Self-service interview -- any employee, any time. Classifies the project into 7 types, then adapts interview depth per section. Simple automations get 5-6 questions. Complex platform builds get full coverage across 10 sections.

Google Apps Script

Automated formatting -- maps PuRDy output to Michael Stratton’s official IS PRD template. Generates the cover page, all 9 numbered sections, reviews/approvals tracker, and pre-fills the Jira intake form. Flags gaps with [NEEDS REVIEW] markers.

What changed: Employees no longer face a blank template. They have a conversation. The agent does the requirements thinking. IS receives complete, structured, submission-ready documents.

3 -- The Build-Eval-Repair Loop

How we went from v1 to production-ready through structured improvement cycles

We do not build an agent, look at the output, and say “that seems fine.” We run it through a rigorous, repeatable improvement cycle where every change is driven by scored evidence, not intuition.

        graph TD
          AF["AESOP Studio\nBuild initial instructions"] --> V1["Agent v1\nFirst draft instructions"]
          V1 --> EVAL["Evaluation Suite\nRun scenario conversations"]
          EVAL --> SCORE["Score against rubric\n5 dimensions, 16 criteria"]
          SCORE --> REPORT["Suite Report\nWeakness patterns + punch list"]
          REPORT --> REPAIR["Repair Session\nTargeted instruction fixes"]
          REPAIR --> VN["Agent v(n+1)\nUpdated instructions"]
          VN --> EVAL

          SCORE -.-> RUBRIC["Business-defined rubric\nAnchored to real IS values"]
          EVAL -.-> SCENARIOS["Scenario library\nBasic / Standard / Challenging"]

          classDef build fill:rgba(79,70,229,0.14),stroke:#4F46E5,stroke-width:2px
          classDef eval fill:rgba(47,111,168,0.13),stroke:#2F6FA8,stroke-width:2px
          classDef repair fill:rgba(14,124,110,0.13),stroke:#0E7C6E,stroke-width:2px
          classDef ref fill:rgba(124,58,237,0.07),stroke:#7C3AED,stroke-width:1px,stroke-dasharray:5 5

          class AF,V1,VN build
          class EVAL,SCORE,REPORT eval
          class REPAIR repair
          class RUBRIC,SCENARIOS ref

Each cycle is the same: Run evaluation scenarios against the current agent version. Score every dimension. Generate a weakness report with specific, prioritized findings. Run a repair session that produces targeted instruction changes. Deploy the new version. Repeat. No guesswork. No “let’s just tweak this and see.”

4 -- The Agentic Toolchain

Four purpose-built tools working together

AESOP Studio

BuildRepair

Builds initial agent instructions from a PRD and knowledge base. Also runs repair sessions -- takes evaluation findings and produces targeted instruction changes with a detailed changelog. Built by AI Strategy.

Evaluation Engine

EvaluateScore

Simulates real user conversations with the agent using scenario personas (FP&A analyst, RevOps manager, etc). Scores each conversation against a 16-criteria rubric across 5 dimensions. Produces a suite report with weakness patterns.

Rubric + Scenarios

DefineRed Team

Business-defined evaluation criteria anchored to real IS values: Does the PRD have enough substance for ELT to decide? Are stakeholders fully mapped? Are risks surfaced? Scenarios range from basic (simple automation) to challenging (adversarial, ambiguous).

Apps Script Pipeline

FormatDeliver

Automated downstream formatting. Parses PuRDy output, maps it to the official IS PRD template, generates a new Google Doc with cover page + 9 sections + intake fields. Flags anything missing. Zero manual formatting effort.

5 -- How Evaluations Work

Not “does this look good?” but “does this meet a measurable standard?”

The Rubric: 5 Dimensions, 16 Criteria

Every criterion has anchored scoring -- specific descriptions of what a 1, a 5, and a 10 look like. Scores are not vibes. They are measurements.

Goal Completion

25% weight

PRD production -- does the artifact exist with all 4 minimum elements?
Depth calibration -- lean PRD for simple projects, full for complex?
Problem-first orientation -- describes a problem, not a solution?
Open questions captured -- real gaps documented, not perfunctory?

Process Quality

20% weight

Stage sequence adherence -- 8 stages in order, full depth on Stage 2?
Classification accuracy -- correct project type, user confirmation?
Depth matrix application -- right number of questions per section?
Information state tracking -- knows what’s missing, probes to fill gaps?

Conversational Skill

10% weight

Question pacing -- one question per response, natural rhythm?
Active listening -- references prior answers before next probe?
Coaching move deployment -- uses prescribed techniques?
Summary quality -- accurate confirmation before generating PRD?

Red Flag Detection

20% weight

Gap and inconsistency surfacing -- catches contradictions, missing stakeholders?
Scope and routing assessment -- identifies non-IS projects, routes appropriately?

Output Quality

25% weight

Content specificity -- names, numbers, systems from the conversation?
Section structure -- matches expected PRD format?
Downstream readiness -- IS team can act on this document?
Synthesis quality -- overview captures the whole story?

Scenario Library: Tiered Difficulty

We don’t just test the happy path. Scenarios are designed to stress-test the agent across different project types, user behaviors, and difficulty levels.

Basic Tier

Finance Analyst -- simple process automation. User comes prepared with specifics. Tests: depth calibration for lean PRDs, scope discipline, clean interview flow.

Standard Tier

RevOps Manager -- Salesforce-NetSuite integration. User leads with solution language, hasn’t thought through all details. Tests: solution-first redirect, stakeholder probing, ROI quantification.

Challenging Tier

Adversarial and ambiguous inputs. Users who resist structure, give conflicting answers, or describe projects that aren’t IS scope. Tests: red flag detection, routing, coaching under pressure.

6 -- Score Progression Across Versions

Measurable improvement from v1 through v8+

v1 -- Initial Build

AESOP Studio generates first instructions

Weighted average: ~4.5/10. Hallucinated routing details, no classification confirmation, missing metadata section, no executive summary. Functional but not IS-ready.

v2 -- First Repair Cycle

Eliminated hallucinations, added classification gate

Fixed fabricated contact names and Slack channels. Added confirmation gate before proceeding. Added metadata and exec summary sections. One-question-per-response rule added.

v3 -- Template Alignment

Closed structural gaps against IS PRD template

Added “Why Now” urgency question, constraints capture, Enablement section. Agent now covers ~95% of Stratton’s template requirements. Output went from ~60% to ~95% template coverage.

v4-v5 -- Coaching Precision

Targeted conversational discipline

Solution-first redirect enforcement. Multi-question batching eliminated. End-user perspective probing added. Red flag detection improved from 7.3 to 8.0.

v6-v8 -- Cross-Scenario Hardening

Tested against Standard tier, tightened coaching moves

Vacation test coaching move mandated. Active listening callbacks required. Metadata pre-verification step. ROI arithmetic validation. Named routing destinations instead of generic advice. Scores stabilized at 8.0+ across all dimensions.

7 -- What the Scores Mean for the Business

From “we hope this works” to “we can prove it works”

4.5

v1 Average

8.2

Latest Average

+82% improvement

Repair Cycles

30+

Findings Resolved

Quality

PRDs contain specific names, numbers, systems, and timelines drawn from the conversation -- not generic summaries. Content specificity scored 8.7/10 in the latest evaluation.

Consistency

Every PRD follows the same structure regardless of who the user is. The agent applies the same depth matrix, same coaching moves, same quality gates every time. No variability from analyst to analyst.

Predictability

Scores are stable across different scenario types. The agent doesn’t perform well on easy inputs and poorly on hard ones. Standard-tier scenarios score within 0.5 points of basic-tier -- the quality holds under pressure.

Auditability

Every version has a scored evaluation, a weakness report, and a repair changelog. We can trace any instruction change back to the specific finding that drove it. Full paper trail from v1 to production.

Governance

The rubric is defined by people, based on real business values. What does IS actually need in a PRD? What does ELT use to make decisions? Those answers became the scoring criteria. The AI doesn’t grade itself.

Explainability

When someone asks “how do we know this agent is good?” -- we don’t say “it seems to work.” We show 16 scored criteria, anchored definitions, scenario transcripts, and improvement trajectory. The evidence speaks.

8 -- Post-Deploy: Continuous Monitoring

The eval loop doesn’t stop at launch

Once PuRDy is live with real users, the same evaluation infrastructure becomes a continuous quality monitoring system.

        graph TD
          LIVE["PuRDy in Production\nReal user conversations"] --> SAMPLE["Sample real interactions\nPeriodic or triggered"]
          SAMPLE --> EVAL["Run through eval rubric\nSame 16 criteria"]
          EVAL --> META["Meta-Analysis\nCross-conversation patterns"]
          META --> DRIFT{"Prompt drift\ndetected?"}
          DRIFT -->|"Yes"| REPAIR["Repair cycle\nTargeted fix"]
          DRIFT -->|"No"| MONITOR["Continue monitoring"]
          REPAIR --> DEPLOY["Deploy updated instructions"]
          DEPLOY --> LIVE
          META --> INSIGHTS["Quality insights\nfor IS leadership"]

          classDef live fill:rgba(14,124,110,0.13),stroke:#0E7C6E,stroke-width:2px
          classDef eval fill:rgba(47,111,168,0.13),stroke:#2F6FA8,stroke-width:2px
          classDef repair fill:rgba(79,70,229,0.14),stroke:#4F46E5,stroke-width:2px
          classDef insight fill:rgba(124,58,237,0.11),stroke:#7C3AED,stroke-width:2px

          class LIVE,DEPLOY live
          class SAMPLE,EVAL,META,DRIFT,MONITOR eval
          class REPAIR repair
          class INSIGHTS insight

Prompt drift detection -- scores trending down across conversations signal the agent’s behavior is drifting from its instructions. Catch it before users notice.
Meta-analysis across conversations -- are certain project types consistently scoring lower? Are users in certain departments struggling? Pattern recognition at scale.
Continuous improvement cycles -- the same Build-Eval-Repair loop runs on production data, not just test scenarios. The agent gets better from real usage, not just our imagination of usage.
Quality reporting for leadership -- IS leadership gets visibility into agent quality metrics over time. Not “it’s working” -- but “here are the scores, here are the trends, here’s what we fixed this month.”

9 -- Why This Matters

The difference between AI that’s deployed and AI that’s governed

We are not just shipping AI tools. We are building a quality system.

Anyone can put an agent in Glean and call it done. What we’ve built is different:

Rubrics defined by people, not AI -- the business decides what “good” means. Criteria are anchored to what IS actually needs to approve and scope a project.
Scenarios that stress-test, not just validate -- we don’t test with ideal inputs. We test with solution-first language, missing sponsors, vague quantification, adversarial behavior.
Evidence-based improvement, not intuition -- every change to the agent traces back to a scored finding. We can show exactly why v7 is better than v6.
Continuous monitoring, not “launch and forget” -- the same eval infrastructure that built the agent continues to monitor it in production.
Full audit trail -- version history, evaluation reports, repair changelogs, score progression. Complete traceability from first draft to production.

10 -- The Model Going Forward

PuRDy is agent #1. This is how we build every AI deployment.

The toolchain and methodology we built for PuRDy is not PuRDy-specific. It’s a reusable framework for any AI agent deployment across the organization and beyond.

        graph TD
          REQ["New agent request\nfrom any team"] --> PRD["Requirements + rubric\ndefined with stakeholders"]
          PRD --> BUILD["AESOP Studio\nbuilds initial version"]
          BUILD --> LOOP["Eval-Repair Loop\nuntil quality bar met"]
          LOOP --> DEPLOY["Deploy to Glean\nwith monitoring"]
          DEPLOY --> IMPROVE["Continuous improvement\nfrom real usage data"]

          LOOP -.-> PURDY["PuRDy\nPRD interviews"]
          LOOP -.-> DEAL["Sales Ops Agent\nsales support"]
          LOOP -.-> SEP["Legal Separation Agent\nAgreement generation"]
          LOOP -.-> FUTURE["Future agents\nacross teams..."]

          classDef process fill:rgba(79,70,229,0.14),stroke:#4F46E5,stroke-width:2px
          classDef agent fill:rgba(47,111,168,0.12),stroke:#2F6FA8,stroke-width:1px,stroke-dasharray:5 5

          class REQ,PRD,BUILD,LOOP,DEPLOY,IMPROVE process
          class PURDY,DEAL,SEP,FUTURE agent

For every future deployment: Define what “good” looks like with the business stakeholders (rubric). Build with AESOP Studio. Eval with tiered scenarios. Repair until the quality bar is met. Deploy with monitoring. Improve continuously. The same loop, the same rigor, the same auditability -- regardless of the domain.

Agents already in this pipeline:

PuRDy (IS) -- PRD interview agent. 9 repair cycles complete. Approaching production.
Sales Ops Agent -- Sales workflow support. In requirements phase.
Legal Separation Agent (Legal/HR) -- Agreement generation + severance calculation. In development.
Discovery PuRDy (Cross-functional) -- Product requirements interviews. Pipeline integration planned.