And I wrote the whole thing down as I went.
One real problem, run from the first question to the finished tool. The tool forecasts where legal AI is heading and what controls firms will be forced to adopt, it grades its own calls against what actually happened, and it keeps its misses on the page, because a forecast that only shows its hits is just hype.
What follows is that build, start to finish.This is the loop I run on every build. Here it is on one real problem, from the first question to the fix.
Discovery is where I refuse to shrink the problem, and this one fought back. The effects run in orders, capability then ruling then adoption then what a firm can afford, but never in a tidy line, some are pure market reactions and some run backwards when a firm files AI output and provokes the rule, and that non-linearity is the thing to model. So no forced single order, each fault line carries the way its own effects actually move.
Press the problem against the solution space until the constraints pick the stack: deterministic and replayable, weighted by who sets the rules not by volume, gradeable against what happened, auditable after the fact. That ruled out a model freehanding the forecast and pointed at plain scored code, and the forks resolved fast:
| Decision | Options | Chosen | Why |
|---|---|---|---|
| Generate the forecast | LLM free-generation vs deterministic weighted scoring | Deterministic scoring | Replayable and auditable. No hallucinated confidence. |
| Weight the sources | By volume vs by authority tier | Authority tiers | T1 binding at 1.0 down to T5 vendor at 0.008. Who sets the rule matters, not how loud. |
| Grade the calls | Predict dates vs called-versus-actual at 90-day lead | Called-versus-actual | Checkable against reality. A miss reads as a miss. |
| Prediction depth | Single ruling meter vs three-order cascade | Cascade | Capability → ruling → adoption. Each layer only counts if the one it rests on held. |
| Conflicting evidence | Average it out vs discount on conflict | Conflict discount | Disagreement should lower confidence, not wash out. |
I paired with AI to move fast and leaned on it hard, but the scoring stayed deterministic code I can read and rerun and defend, five small files that each run on their own.
Then I graded it against reality, replaying ninety days out to check which calls actually held, and I left the miss in on purpose, because a panel that only shows hits is marketing, not calibration.
A miss isn't something to bury, it's the next thing to build. This one said the base of the forecast was too thin, so the next work added a proper capability lane and widened the market evidence the calls sit on.
Every run gets logged, hashed and replayable, the inputs and the reasoning and the output and the sources each score rested on, so the whole thing audits end to end. And that record is where the improvements come from, the gaps surface on their own and the loop reopens with evidence instead of a guess.