Full trace: 19 executive intake fields through parsing, scoring, multi-pass analysis, and roadmap generation. Every heuristic, decision, and confidence signal at its point of injection.
What Goes In — The 19-Field Executive Intake
About Your Business
business_name industry employee_count what_they_do what_makes_different
Each of the 19 fields maps to one of 4 org_context rows. No AI involved — this is a deterministic mapping in _create_org_context_entries(). The function unpacks the flat 19-field dict into typed context records with titles, content, and content_summary.
Organization Creation
Business name → slug (lowercase, hyphenated). Organization record created in organizations table with org_type="smb". Submitter email added as org admin (best-effort — public submissions may not have an account).
"They use QuickBooks and Excel. Documents are in a shared Drive but inconsistently organized."
confidence scoring runs in parallel — same input, different path
Step 2 — Confidence Scoring: How Much Do We Trust the Output?
This is the scoring engine Aparna asked about. Every field that's empty reduces confidence — and every reduction is tied to a specific, hand-authored impact statement that names exactly what the roadmap can't say because of that gap. This isn't a generic "confidence score" — it's a gap-by-gap audit trail.
1. Field-Level Check
Each of 19 fields is checked: is it empty? None, empty string, or whitespace-only = empty. Non-empty strings and numbers count as filled. This is a binary check — no content quality assessment.
2. Section Scoring (5 × 20 pts)
Each of 5 sections is worth 20 points. Section score = 20 × (filled_fields / total_fields_in_section). So if "About Your Business" has 4 of 5 fields filled, that section scores 16/20.
3. Total & Cap
Sum of 5 section scores = raw total. Then capped at 95. The cap is deliberate: "perfect confidence is a false signal" — even with all fields filled, the system acknowledges there's always uncertainty in automated analysis of self-reported data.
Section Score= 20 × (filled / total_fields) Raw Total= sum of 5 section scores Final Score= min(Raw Total, 95) Level=high ≥75|medium ≥50|low <50
Gap Analysis — Impact Per Missing Field
Plain-Language Impact (Hand-Authored)
Every one of 19 fields has a unique, hand-written impact statement in _executive_gap_impact(). Not a template — each names exactly what the roadmap cannot determine without that field. Examples:
"Without the founder's day-to-day role, the roadmap cannot identify delegation opportunities." "Without a success definition, there is no way to measure whether the roadmap is working."
Confidence Summary (3-Tier)
High (≥75): "The intake covers all four areas in detail — the roadmap is well-grounded."
Medium (≥50): "Direction is sound but specific recommendations may shift as gaps are filled."
Low (<50): "This roadmap is a starting point for conversation, not a finished plan. The gaps below become cohort exercises that will sharpen the picture week by week."
Why three passes? A single "write a roadmap" prompt is fragile — the LLM skips evidence, invents details, or collapses into generic advice. Three independent passes, each with a narrow lens, force the system to ground claims in evidence before synthesizing. If any pass fails, the pipeline degrades gracefully to the remaining passes.
Input: All 4 org_context rows + confidence data What it produces:
• 5×5 Maturity Assessment — scores each of 5 dimensions at each of 5 levels with specific evidence from the intake justifying each placement
• Dysfunctionality Severity — scores 5 dysfunction axes (knowledge-in-heads, unwritten-rules, rework-loops, broken-handoffs, founder-blocking) on a 1-10 scale
• Confidence Gaps Ranked — which missing fields most impact roadmap reliability, ordered by severity
• Boundary Cases — dimensions where placement could shift to adjacent level with better data ("this is a 2, but with one more data point it could be a 3")
Pass 2 — Roadmap Synthesis Agent temperature=0.2
Input: Pass 1 traits analysis + org_context + confidence What it produces:
• Movements — three strategic phases, each with: what to do, risk signals to watch, dependencies, and success indicators. Cross-references dysfunctionality traits to explain why each movement is hard for this specific organization
• The Honest Truth — the uncomfortable finding in their data they may not want to hear. (e.g., "Your estimating bottleneck isn't a process problem — it's a founder-as-single-point-of-failure problem.")
• The One Move — single highest-impact action they can take now, grounded in their specific bottleneck
• What Changes — concrete before/after picture if they execute the movements
Pass 3 — Prose Roadmap Agent temperature=0.2
Input: Pass 1 traits + Pass 2 synthesis + org_context + confidence What it produces:
• Maturity Snapshot — current level, bottleneck dimension, bottleneck explanation (grounded in their specifics), stage-by-stage gap map (1→2, 2→3, 3→4, 4→5) with 2-3 sentences each
• The One Move — rendered as actionable prose
• What Changes — concrete before/after picture
• The Honest Truth — named directly, not softened
• Source Manifest — audit trail of which intake fields informed which claims
Graceful degradation: If Pass 1 fails → skip to single-pass Pass 3. If Pass 1 succeeds but Pass 2 fails → Pass 3 runs with Pass 1 data only. Every pass is independently persisted to adoption_roadmap_traits and adoption_roadmap_synthesis tables for audit.
LLM output → validation → persistence
Step 4 — Validation, Persistence & Output
JSON Validation
Agent output is validated against the expected JSON schema. Up to 2 automatic retries if JSON is malformed, missing required fields, or has wrong types. Each retry includes the specific validation error so the LLM can correct it.
Database Persistence
adoption_roadmaps — final roadmap with maturity_snapshot adoption_roadmap_traits — Pass 1 audit artifact adoption_roadmap_synthesis — Pass 2 audit artifact smb_intake_submissions — status updated to "completed"
Output: 5-Page Roadmap
~1,900 words of founder-ready prose. Maturity placement with bottleneck dimension. Stage-by-stage gap map. Three movements with risk signals. The one move. The honest truth. What changes. Source manifest showing which intake fields informed which claims.
The Confidence-Maturity Connection
Confidence score is not the maturity score. Confidence says "how much do we trust this roadmap?" Maturity says "where is this organization on the AI readiness spectrum?" The two interact: a low-confidence roadmap with high maturity placement is flagged as suspect. Every maturity dimension score in Pass 1 includes a confidence_modifier that expresses how much the placement could shift with better data.
the full picture — all heuristics and decision points
Every Heuristic & Decision Point — At a Glance
#
Decision Point
What It Does
Where It Lives
1
Field completeness check
Binary: is each field non-empty? Drives confidence score and gap analysis.
_compute_confidence_executive()
2
Section weighting
5 sections × 20 points each. Equal weighting — no section is privileged. Reflects the view that company profile is as important as dysfunctionality data for an honest assessment.
_compute_confidence_executive()
3
Confidence cap at 95
Deliberate ceiling. "Perfect confidence is a false signal." Even 19/19 fields filled, self-reported data has inherent uncertainty.
_compute_confidence_executive() — line min(total, 95)
4
Gap impact mapping
19 hand-authored impact statements — each names exactly what the roadmap can't determine without that specific field. Not a template, not LLM-generated.
_executive_gap_impact()
5
Confidence level tiering
High ≥75, Medium ≥50, Low <50. Each tier has a plain-language summary that sets expectations about roadmap reliability.
_executive_confidence_summary()
6
Maturity bottleneck principle
Overall maturity = lowest dimension score. A business with great processes but no governance is stuck at governance. Prevents cherry-picking the strongest dimension.
Before every claim: "Did they tell us this?" If not, don't write it — or explicitly flag it as a confidence gap. This is the hardest rule and the one that separates honest analysis from generic AI output.
Each of 5 dimensions scored at each of 5 levels. Evidence required per score. Boundary cases flagged where placement could shift with better data.
DysfunctionalityTraitsAgent (Pass 1)
9
Dysfunctionality severity (1-10)
5 dysfunction axes scored independently. Not averaged — each is a separate signal. High scores on "founder-blocking" or "knowledge-in-heads" trigger specific movement recommendations.
DysfunctionalityTraitsAgent (Pass 1)
10
Multi-pass triangulation
Pass 1 (traits) → Pass 2 (synthesis) → Pass 3 (prose). Each pass sees the prior output but is prompted to challenge, not rubber-stamp. Graceful degradation if any pass fails.
_generate_roadmap_background()
11
Jargon ban
Banned words: LLM, API, vector database, embedding, token, agent, platform, ecosystem, solution. Enforced in the prompt — if the LLM uses these, the validation layer catches and retries.
Every output claim traces back to a specific intake field. "This recommendation exists because they told us X." Audit trail for both the system and the human reviewer.
Source manifest in output JSON
what the pipeline doesn't do — constraints by design
What the Pipeline Does NOT Do — Constraints by Design
Guess at missing data
When a field is empty, the system doesn't infer or assume. It names the gap and reduces confidence. The "no assumptions" rule is enforced at every LLM pass.
Recommend specific AI tools
The roadmap says "you need a way to capture estimating rules" — not "buy Tool X." Tool recommendations without context are dangerous and the prompt explicitly forbids them.
Claim 100% confidence
The 95-cap means the system always says "this could be sharper." Even with complete intake data, the output carries appropriate uncertainty.
Use the same framework for everyone
The maturity model is universal — but every score is grounded in that organization's specific intake. Two businesses at "AI Ready" got there differently and the output names those differences.
Summary — The Full Trace in One Sentence
19 fields enter → deterministic parsing into 4 context records → confidence scoring with hand-authored gap impacts → 3-pass LLM analysis (traits → synthesis → prose) each grounded in evidence and cross-validated → JSON validated with retries → persisted with full audit trail → 1,900-word founder-ready roadmap with maturity placement, bottleneck identification, stage gaps, movements, the honest truth, and a source manifest showing which intake field informed every claim.
19
Input Fields
4
Context Rows
12
Heuristics
3
LLM Passes
95
Confidence Cap
AESOP Transformation OS — Executive Intake Trace v1.0 — June 2026