2 seeds · 16 sprints · 1 adversarial audit

I built a simulation to test a thesis. Then I audited it and found it was cheating.

You cannot fork a real company to test whether the AI operating model is discovered or designed. So I simulated one. It returned the answer I wanted, which is exactly when a result deserves the least trust. So I traced the win to its source, found a rig, cut it out, and re-ran the whole test. The tidy answer did not come back — and that is the finding.

Overview Full run output Cross-run comparison Measurement & validation Revision log

The shape of the investigation

Build the instrument.

It confirms the thesis I’d arrived at.

Distrust that.

Trace every metric to its mutation.

Found the rig.

Fixed it.

Re-ran the honest test.

The claim

Stuart Winter-Tear puts it most precisely: the AI operating model is discovered through implementation, not designed before it. A claim I came to believe by running AI work inside organizations, not by reading about it — worth testing rather than asserting. Testing means running one company through two change processes at once. Nobody has a spare company.

Track ADiscovered,
not designed

Deploy without reorganizing. Put AI into the existing process. Watch where it breaks. Redesign from evidence you actually observed. Slow, narrow, and blind to any seam it has not yet seen fail — that blindness is the honest cost of the method.

Track BTraditional change
management

Design it, then roll it out. Kotter, Prosci, McKinsey 7S. Assess up front, rank seams by risk, design the target state, roll out broadly, govern by steering committee. Fast and wide, but only as accurate as the a-priori design.

The instrument

Progressive Insurance, run twice. A calibrated model: 70,000 employees, $86B in premiums, already running 55% of claims through AI. Financials from Q2 2026 filings, culture from Glassdoor, incentives from the disclosed Gainshare structure. Eleven stakeholders carry OCEAN traits, cognitive biases, and motivational drivers — organizations are not rational optimizers, they are people protecting metrics and repairing broken processes with unwritten knowledge. Both tracks face the same world: same hurricanes, same attrition, same seed, same claims. The only variable is how change happens.

Scale

16 quarterly sprints · 2 independent seeds · ~5,400 decisions per track.

Seam profile is
the regime dial

Tacit seams are judgment-heavy and start gappy — liability calls, negotiation, settlement. A-priori design cannot template work that lives in someone’s head, so this is where Track A should win. Codifiable seams are schemas, thresholds, payment rails — work Track B can genuinely design in advance. Whether B ever wins there is the whole test.

The measurement that mattersTranslation debt. The share of decisions where information had to be reconstructed, re-requested, or repaired at a handoff. It is the tax an organization pays when AI is inserted into a seam nobody redesigned.

The finding — the instrument failed its own test, and that caught the rig

Validity gate 5 asks one question: can we construct conditions where Track B wins? If not, the simulation asserts its answer instead of discovering it. In the original build, Track B did not win a single cell — including the cell built for it to win. That was the tell.

Cell	Designed to test	Seed A	Seed B	Expected
S1_B_should_win	Codifiable seams, calm, short horizon	tie	tie	B
S2_A_should_win	Tacit seams, turbulent, long horizon	A	tie	A
S3_honest_middle	Mixed, should vary by seed	A	A	seed-dependent

An instrument that cannot produce the answer you do not want is not measuring. So I traced every metric back to the state field it reads, and every state mutation back to the mechanic that writes it.

C1 · the rig

I was giving Track A a free, permanent, omniscient advantage

The headline metric is a direct function of one variable. Track A cut that variable by 40% every two sprints, targeting the worst seam in the entire workflow, whether or not A had ever observed that seam fail. Permanent. Never regressed.

step.ai_error_rate = max(0.02, step.ai_error_rate * 0.6)   # unconditional

Track B’s only comparable lever was smaller, probabilistic, blocked on tacit seams, and reversible. Track A was structurally incapable of losing. The subtle part is which half was legitimate. A’s fixes being durable is real — the team owns the fix because it built it from its own evidence, and that is the actual mechanism the whole thesis rests on. Keep that. What was rigged is that the fix was ungated, larger, and guaranteed every cycle. The code handed A omniscience and deleted its defining constraint.

How I fixed itOne line, one gate. The cut is now allowed only on seams Track A has actually observed failing — the same evidence gate that already governed everything else A does.

if step.name in observed_gap_steps:      # only what A has watched break
    step.ai_error_rate = max(0.02, step.ai_error_rate * 0.6)

Durability survives, because it is real. Omniscience is gone, because A can no longer repair a seam it has never seen break. The winner becomes conditional on the regime instead of guaranteed. I also deleted the second rig — the hand-written trust deltas — and re-derived trust from experience for both tracks.

What the fix revealed

Deleting a differentiator produced a real one.

Second rigTrust was
written by fiat

Fixed per-track deltas moved employee trust every sprint, calibrated — the source comment said outright — so B would climb and A would hold flat. Two rigs pointing opposite ways do not cancel; they make two metrics worthless. Removed, replaced with derivation from the already-tracked positive and negative AI-experience counters, identically for both tracks. What emerged: trust stays near baseline (~4.3) and lands near-equal in both tracks. That is what an honest null looks like. The change process is not what moves trust. Working AI is.

Does anything replicate?

Six of seven headline metrics hold direction across two independent seeds. Exception rate is the lone flip, a sub-point gap that changes sign; trust sits near-equal rather than flipping.

Translation debt — seed 42, sprint by sprint

Sprint	A	B	Sprint	A	B
S1	24.77	24.42	S9	10.26	20.38
S2	25.23	25.11	S10	2.61	17.01
S3	37.97	38.75	S11	1.27	16.78
S4	25.33	31.33	S12	0.0	16.56
S5	27.27	33.12	S13	4.67	20.67
S6	21.57	22.29	S14	2.56	18.97
S7	21.66	22.01	S15	2.67	19.33
S8	14.94	24.2	S16	2.01	23.49

A falls from ~15 to under 2 between sprint 8 and 11 and stays there; B plateaus in the 16–24 band and never converges. The inflection is structural — where accumulated observation finally covers enough seams for durable fixes to compound. Seed 7 shows the same: 10.60, 13.27, 3.69, 1.90.

Back-half means — both seeds

Metric	A s42	B s42	A s7	B s7	Direction
translation_debt_index	3.26	19.15	3.40	17.01	holds
exception_rate	5.32	4.92	8.68	8.75	flips
cost_per_claim	$272.88	$344.76	$303.92	$394.04	holds
first_pass_accuracy	71.5%	44.0%	42.0%	23.5%	holds
cycle_time	4.79	4.95	6.45	6.68	holds
trust_polarization	3.67	3.88	3.60	3.98	holds
employee_ai_trust	4.26	4.31	3.98	4.71	holds

A’s debt collapses
at the same point twice

Sprint 8 to 11, Track A falls from roughly 15 to under 2 and stays there — seed 42: 14.94, 10.26, 2.61, 1.27; seed 7: 10.60, 13.27, 3.69, 1.90. B plateaus and never converges. The inflection is structural: it is where accumulated observation finally covers enough seams for durable fixes to compound.

B’s failures are
specific, not random

The traces name them. B ranks seams by a static risk score and polishes clean handoffs while broken ones leak. Its completed seams regress under turbulence and change fatigue — shallow adoption that doesn’t stick, so fnol_intake completes and then reverts when conditions turn. And it repeatedly templates settlement_human, judgment work it structurally cannot template.

Conclusions — what holds, and what I cannot claim

Supported

In the tacit regime, Track A wins on outcomes. Six of seven metrics hold direction across two seeds, and the de-rigged sweep confirms it: a clean A win by 9 to 15 points on both seeds of the tacit cell.
Durability is the mechanism, not speed or breadth. A’s fixes stick because the team built them from evidence it owns.
Trust is not the differentiator. It stays near baseline in both tracks. Look at outcomes, not sentiment.

Tested — and it refused to flip

The sweep has been re-run on the de-rigged code. The audit’s prediction was specific: with A’s advantage evidence-gated, the codifiable and calm cell should flip to a clean B win. It did not.

S1, codifiable and calm. Still a tie. Widened from two to six stress-probed calm seeds: a mean translation-debt gap of +0.19 in B’s favor, an order of magnitude inside the 5-point win threshold. B leans on four of the six and never clears the bar.

B’s theoretical best case. Stacked every lever in B’s favor — perfect vendor, instant decisions, longest horizon, the three calmest seeds — and ran it again. Three cells, mean gap −0.03, a dead tie that if anything leans a hair toward A. Zero B-wins.

S2, tacit and turbulent. A clean A win on both seeds, by 9 to 15 points. The instrument discriminates hard when the regime calls for it. It now produces the honest asymmetry the thesis predicts: A wins the tacit regime decisively, and B, at best, ties its own.

So the prediction is refuted on the merits, not by rigging. The de-rig worked. What it revealed is smaller than hoped and more believable than the original: B’s a-priori coverage is real but too weak to beat A’s durability even on B’s home turf. Gate 5 is answered honestly, and the answer is uncomfortable, which is the whole point of building the gate.

The four open items, since closed

Attrition now fires. Baseline churn was ~5× too low; it now produces three to seven departures per run, and departures backfill a fresh hire so the pipeline keeps measuring.

The psychological layer now does mechanical work. Trust is driven by loss aversion, overconfidence, and status quo, not a single openness scalar; the eleven profiles land on distinct values instead of converging to a flat 8.0.

The seed objection is closed by power analysis. The codifiable tie is parity — mean Δ +0.19 with a 95% interval that straddles zero; the tacit win is decisive at mean Δ −12. More seeds cannot flip either.

The 0.00-debt sprint was a genuine, evidence-gated result. Track A can only repair a seam it has actually watched fail; the near-zero is the durable payoff of that, not the C1 rig.

What the experiment is actually for

The winner is the least interesting output. The two tracks are two ends of one axis, speed versus absorption, and the traces show exactly where each leaks. A leaks time: it cannot fix what it has not yet watched break. B leaks trust: it templates the untemplatable and its changes wash off.

A better Track A borrows B’s breadth. A better Track B borrows A’s ownership. The instrument earns its keep by being sharp enough to say which, and honest enough to say when it cannot.