Solomon Engine · Architecture & Beta Readiness

The voice front-door to a personalized executive AI.

A ~10–15 min voice interview → structured extraction → an admin-reviewed, custom AI-assistant configuration. This one-pager reconciles the as-built system with its April–May beta testing history to inform post-feedback next steps.

Beta in market — 10+ exec cohort Baseline locked 2026-05-15 Prepared by Charlie Fuller 2026-06-04

01 Snapshot at a glance

Interview segments

~6 min each, own agent

60%

Coverage threshold

to advance / trigger Solomon

5/5

Integration suite

pipeline verified end-to-end

~15

Test/fix cycles

Apr 8 → May 19

Dominant open bug

Segment-2 early wrap-up

Scope. This documents the Solomon voice-interview engine in beta — not the wider platform's chat / RAG / knowledge-base layer. Where deployed reality differs from earlier collateral, see §06 Documentation Drift.

02 Architecture as-built, verified against code

Tech stack

Frontend: Next.js 16 · React 19 · TS · Vercel
Backend: FastAPI 0.115 · Python 3.11 · Railway
Voice: ElevenLabs Conversational AI (2 agents)
Scoring: Claude Haiku
Instructions: Claude Sonnet 4-6
Data: Supabase (Postgres + RLS)
Email: Resend

Two segments

S1 — Pain Points & Workflow
pain_points · impact_baseline · TRIPS capture

S2 — Work Style & Tools
comms · strategy · team · frameworks · doc storage · org context

Prompt wiring

Segment prompts live in prompts/*.md, injected per-session via the ElevenLabs SDK overrides field. Dashboard base prompts are fully replaced at session start — historically the single biggest source of bugs.

Reliability guardrails

60% threshold · skips excluded from denominator
Silence 10s timeout · 45s auto-end · 90s min-age
English-only · no identity re-ask · repeat handling
Duplicate-webhook + late-fetch + autoplay fixes

Key tables

interview_sessions · interview_segments
interview_extractions (+ eval JSON)
solomon_reviews — pending admin gate
api_usage_logs — token / cost

Admin tooling

Solomon review & approval workflow
Interviews history + transcripts
API-usage page (ElevenLabs credits)
On-demand AI quality-evaluation report

03 Processing Pipeline what happens after the voice call

graph TD
  COL[Columbus
lead capture] --> V
  V["Voice interview
ElevenLabs · 2 agents"] -->|WebRTC disconnect| WH["Webhook
/interview-complete
HMAC-verified"]
  WH --> SC["Segment scoring
Claude Haiku · 0–100
skips excluded"]
  SC --> G{"Both segments
≥ 60%?"}
  G -->|No| RD["Flag for redo
can_redo = true"]
  G -->|Yes| S1["Solomon Stage 1
Extraction · Haiku
12 structured fields"]
  S1 --> G2{"≥ 60%?"}
  G2 -->|No| NR["needs_review
admin notified"]
  G2 -->|Yes| S2["Solomon Stage 2
Instructions · Sonnet
top 5 of 10 functions"]
  S2 --> REV["solomon_reviews
status = pending"]
  REV --> AG{"Admin
approve?"}
  AG -->|Reject| NR
  AG -->|Approve| DEP["Deploy to
client profile"]
  DEP --> PRD["PRD interview
(Inject & Skip)"]

  classDef gold fill:#2a2410,stroke:#f8c51c,color:#f5e9c8;
  classDef teal fill:#10302c,stroke:#3bc9b0,color:#cdeee7;
  classDef gate fill:#2e2418,stroke:#f0a23e,color:#f5dfc0;
  classDef stop fill:#321c1b,stroke:#f0726b,color:#f3cfcc;
  classDef done fill:#13301d,stroke:#5fd07f,color:#cdeed4;
  class V,WH gold;
  class SC,S1,S2 teal;
  class G,G2,AG gate;
  class RD,NR stop;
  class DEP,PRD,REV done;
  class COL,PRD gold;

Nothing auto-deploys. Generated instructions always land in solomon_reviews as pending and require human approval. The Apr 29 integration suite confirmed every stage after the voice call — including duplicate-webhook and stuck-session recovery — passes 5/5.

04 Testing & Results Apr 8 – May 19, 2026

C1 · Completion

Both segments reach coverage; no premature wrap-up.

C2 · Scoring

Score reflects quality; no false "All Done"; skips not penalized.

C3 · Control

Silence, English-only, no re-ask, graceful exit, recovery.

C4 · UI state

Results always load; stable buttons; accurate timer; recap.

C5 · Infra

Mic/WebSocket, audio on start, webhooks, email, credits.

Date	Round & key finding	Resolution
Mar 4	Stuck at 50% — invalid model ID (404); backend defaulted to 50% on scoring failure	FixedModel ID corrected
Apr 8	16-item UX changelog (scripting, lag, End button, recap, rating)	All verifiedBE 69 · FE 99 · E2E 130/1
Apr 21	Stress test (Menchie/Niharika): false green @2–35%, skip penalty, 11s silence, S2 ran S1 script, language drift, vanishing results	→ Apr 22
Apr 22	Post-stress-test hardening	11 fixesthreshold, skip-exclude, silence, coverage req, guardrail, race
Apr 23–24	S2 stuck @52% (never reached Q5/Q6); brand color	FixedS2 prompt rewrite · Sun Gold palette
Apr 29	Root cause: override prop `conversationConfigOverride` silently ignored → agents ran on placeholder prompt the whole time	Fixed→ `overrides`; +8-item reliability pass
Apr 29	Automated integration suite (Charlie)	5/5 pass92% / 95% high-quality runs
Apr 30–May 1	Hard block: "Couldn't connect" everywhere — 3 bugs (allowlist typo, timeout race, override fields disabled →1008)	All fixed
May 5–6	Paige blocked @50% "Scoring error" — production Anthropic key had zero credits; audio silent-start; repeat-question	Credits + codeautoplay fix · repeat guideline
May 12	Result capture + nav good; S2 premature wrap-up persists (28% despite good input)	Open
May 18–19	Beta go/no-go: closing screen broke; refresh → "interview closed" lock-out	Fixedresults-wait 30→90s + retry; Submit Anyway; auto-deploy

The pattern: nearly every blocking failure was infrastructure wiring, not UX logic — prompt overrides never reaching ElevenLabs, a depleted API key silently scoring 50%, an allowlist hostname typo. The UX fixes stuck; the infra failures kept recurring in new forms.

05 Open Items entering / during beta

#	Item	Severity	Status / decision
O1	Segment 2 premature wrap-up — processes after 2–3 Q, skips Tech Stack / Work Styles (~28%)	High · UX	Baseline-locked 05-15; deep fix deferred to avoid regression. Topic-checklist guardrail proposed
O2	Anthropic credit depletion silently degrades scoring to 50% fallback	High · Ops	Needs proactive credit alerting; partial via API-usage page
O3	Audio silent-start on Segment 2	Medium	Autoplay fix shipped; monitor for recurrence
O4	Contradiction-flagging guardrail never fully verified	Low	Segment kept ending before it could be tested
O5	Email by manual link only	Process	Deliberate — avoids spam-filter risk during beta
O6	TBG logo is styled text, not image asset	Cosmetic	Awaiting logo file
O7	STATS-02 E2E failure	None	Pre-existing test-infra CORS issue, unrelated

06 Documentation Drift reconcile before external use

Older collateral claims

As-built reality

Four 5–7 min segments
70% completeness threshold
Voyage AI embeddings in interview pipeline
Extracts "20+ parameters"
Interview "15–20 min" / agent says "30 min"

Two ~6-min segments (no S3/S4 prompts exist)
60% across all enforcement points
Voyage powers chat/RAG only — not Solomon extraction
Stage 1 outputs 12 structured fields
Actual ~10–15 min; agents now say "about 5 min/segment"

07 Next Steps post-beta-feedback, prioritized

Immediate

1 · Permanent fix for credit depletion (O2)

Automated low-credit alerting on the Anthropic key — the highest-leverage reliability gap; it silently breaks scoring for every user.

Immediate

2 · Instrument beta sessions

Capture per-segment scores, completion rate, drop-off point, and Help/bug submissions to quantify O1's real field frequency.

Near-term

3 · Fix Segment-2 wrap-up (O1)

Hard topic-checklist guardrail — but only after field data, in an isolated branch with the integration suite as a regression gate.

Near-term

4 · Verify contradiction-flagging (O4)

Run a fresh full Segment-2 session to confirm the guardrail fires.

Structural

5 · Reduce the "Double Interview"

Implement CIPHER "Inject & Skip" — Solomon JSON pre-populates / hides ~65% of PRD fields. Largest UX-fatigue lever; shapes the Solomon→PRD roadmap.

Structural

6 · Reconcile documentation (§06)

Align beta & sales collateral with the system before external sharing.

Roadmap

7 · Hybrid voice + clickable · 8 · Transcript-quality protocol

Lower ElevenLabs token cost; target 90% accuracy on core pain-point extraction. Re-evaluate late June–July.