Xcel demo · The loop

The Audit–Monitor Feedback Loop

The KB audit grades documents before they ship. The monitor watches what the agent actually does with them in production. On their own, each has a blind spot. Wired together, the monitor becomes the audit's missing ground truth.

The audit says a document is clean. The monitor says it got served to the wrong state anyway. Feed that back, and the next audit knows which of its grades the real world actually cares about.

Storage — the audit, grading docs at rest Consumer — the agent in production Compute — the monitor and the calibration

Two tools, two blind spots

They are not redundant. Each sees something the other structurally cannot, which is exactly why one can correct the other.

KB Audit · before ship

Grades the documents, cold

Reads every document and scores its writing: clear, complete, consistent, up to date. It sees the whole library at once. It cannot see which documents customers actually hit, or whether retrieval serves them to the wrong person.

Monitor · in production

Watches what actually happened

Reads the record behind each live answer and catches the document served to the wrong state, the expired policy that answered, the FAQ that outranked a tariff. It only sees what real traffic asked. It says nothing about documents no one happened to hit.

What the monitor taught the audit

Run against the same 30 Xcel documents, the two tools lined up in three telling ways. This is the loop in real numbers, not in theory.

What showed up	KB Audit alone	What the monitor added
The stale archive	Scored it 88 — a mild "add an archival tag" note	It was actually served 4 times. The mild note should have been a top-severity finding.
The draft procedure	Scored it 63 — the audit's #1 fix-first item	Confirmed it answered a live call. The audit's instinct was right, and now it is proven.
Wrong-state answers	Scored the documents 90 — clean	7 findings on those same clean docs: a 90-rated policy served to two wrong states. A blind spot the audit cannot close on its own.
Conflict pairs	Flagged 4 document pairs as conflicting	Only 1 ever surfaced in real traffic. The monitor tells the audit which of its conflicts to prioritize.

The headline: a document can score 90 on the audit and still be wrong in production, because the audit grades the document and the monitor grades the retrieval. Neither is enough alone.

Closing the loop: three ways the monitor re-tunes the audit

Each is a concrete change to how the audit scores, driven by what the monitor saw happen for real.

Re-weight

Escalate by real exposure

A document the monitor has actually caught serving stale or out-of-state gets its audit severity raised. Being served wrong 4 times is not a style nit; it is a priority-one fix, and the score should say so.

Re-scope

Grade retrieval, not just prose

Add the dimension the audit is missing: not "is this document well written" but "does this well-written document get pulled out of its jurisdiction." That signal only exists in the monitor.

Re-rank

Prioritize by what traffic hits

The audit's backlog is sorted by score. The monitor lets it sort by exposure instead: fix the conflict that fired in production first, not the three that are only theoretical.

Why this is the honest version

A static audit that never hears back from production slowly drifts into grading a rubric instead of a risk. A monitor with no audit only ever reacts to failures after a customer already saw them. The loop is what keeps each one tied to the thing that matters.

The audit grades documents; the monitor tells it which grades the real world cares about. Before ship, the audit is your best guess at what will go wrong. After ship, the monitor is the record of what actually did. Feed the second back into the first, and the audit stops being a checklist and starts being calibrated to production — which is the only calibration that counts.