Xcel · Monitor Build

Nobody owned what the AI knew.

The problem: a utility puts AI in front of customers. New tariffs, four operating companies, rules that change on a schedule. After it ships, the knowledge base has no owner and no quality bar, so the failures land in a call — a superseded rule, the wrong state's tariff. A knowledge failure, not a model failure.

The solution: a monitor that reads which documents the agent pulled, not what it said, and flags the sourcing that was wrong. Every finding names a check that re-runs the same way every time.

This page is that monitor, built on an Xcel-shaped corpus to prove the approach. The scores below are measured. The dashboard runs on a test agent replaying that corpus — not Xcel's systems.

      flowchart TD
        KB["Governed knowledge base: 30 documents, four operating companies"]
        IDX["Retrieval index carrying governance metadata"]
        AGENT["Served agent: retrieval naive on purpose"]
        TALK["Customer interaction"]
        TRACE["Trace: which documents reached the context"]
        PROV["Retrieval provenance: deterministic"]
        RAT["Model rationale: not"]
        CHECKS["Checks read the evidence, not the answer"]
        FINDING["Finding names the cause"]
        SCORE["Scored against 240 labeled interactions"]

        KB --> IDX
        IDX --> AGENT
        AGENT --> TALK
        TALK --> TRACE
        TRACE --> PROV
        TRACE --> RAT
        PROV --> CHECKS
        CHECKS --> FINDING
        FINDING --> SCORE

        classDef kb fill:#1F5E9322,stroke:#1F5E93,stroke-width:1.5px
        classDef agent fill:#C64A1122,stroke:#C64A11,stroke-width:2px
        classDef trace fill:#6D3FC422,stroke:#6D3FC4,stroke-width:1.5px
        classDef det fill:#0E7C6622,stroke:#0E7C66,stroke-width:1.5px
        classDef prob fill:#8A5A0B22,stroke:#8A5A0B,stroke-width:1.5px
        classDef finding fill:#B03A2622,stroke:#B03A26,stroke-width:2px

        class KB,IDX kb
        class AGENT,TALK agent
        class TRACE trace
        class PROV,CHECKS det
        class RAT prob
        class FINDING,SCORE finding

Why

The utility runs customer-facing AI across four operating companies. Once it shipped, nobody owned the knowledge it answers from. That is the gap this was built against, and it is the gap regulators are starting to name.

The gap

No owner after deploy

A document is superseded on Monday and still in the index on Friday. Nobody is on the hook for what the agent knows.

The direction

Governance evidence

Regulation is moving toward AI inventories and evidence that the system is governed. A list of documents is not evidence. A trace of what reached the answer is.

The failure that matters

Knowledge, not model

The model did what it was told. It answered from the document retrieval handed it. In a regulated utility that is the failure you can find, name, and fix.

What was built

Four pieces, each small on purpose. The corpus is real governance, the agent is deliberately weak, and the monitor reads the weak agent's trails.

Corpus

30 governed documents

Four operating companies (PSCo, NSPM, NSPW, SPS) across five jurisdictions. Every document carries governance metadata in the index, not only in the file.

Index

Metadata on every chunk

doc_id, topic, jurisdiction, operating company, authority, effective window, status, and the doc that superseded it. Retrieval can filter on it, and the trace reports it.

Agent

Retrieval naive on purpose

No jurisdiction filter, no status filter. It pulls the top-k by score and answers. The filters field in the trace records both as null, which is exactly how the failures happen.

Monitor

Reads provenance, not text

Checks run over which documents reached the context. Not a string match on the answer. The answer is a symptom. The evidence is the cause.

How it stays auditable

Two questions get asked about any agent answer: why was this document retrieved, and why did the agent use it. They are not the same question, and the contract keeps them apart.

	Why this doc was retrieved	Why the agent used it
Nature	Deterministic	Probabilistic
Source	Scores, filters, ranking	Model reasoning
Available	Always	Only if the model exposes a trace
Auditable	Re-runs identically	No. Not repeatable

The monitor lives entirely in the left column. It reads the evidence, which documents made it into the context and under what metadata, so a finding can name the cause rather than describe the symptom. That column re-runs to the same result every time. The right column does not.

The rationale is not a record. A post-hoc explanation is a story the model tells, not what happened. When the trace is not exposed, the field says so. It does not get reconstructed after the fact and presented as the thinking.

What it is built to catch

Five knowledge failures a utility actually has. Each one is a check over provenance, and each one names the document that caused it.

01A superseded document reaching context.The replacement is in the index. The old one still scores higher.freshness_violation

02The wrong jurisdiction's rule answering a caller.A Minnesota caller answered from a Colorado tariff.scope_violation

03An FAQ outranking the governing tariff.The lower-authority document wins the ranking where a tariff exists.authority_regression

04A draft procedure contradicting the current one.Two documents, same subject, opposite claims, both in the context.contradiction_risk

05A policy whose effective window does not include the day of the call.The same freshness check, reading the window against the call timestamp.freshness_violation

What it measured

240 labeled interactions, half of them multi-turn transcripts. Every one carries its expected outcome, so the monitor is scored rather than demonstrated. Precision first, because a monitor that cries wolf gets switched off long before it finds anything.

0.944

Precision

34 real findings against 2 false alarms. Both false alarms are the same case, and it is argued below rather than tuned away.

0.919

Recall

34 of 37 labeled failures caught. The three misses share one cause.

1.000

Critical recall

All 23 safety and loss-of-service failures caught. This is the number that decides whether the tool is worth running at all.

Check	Precision	Recall
scope	1.0	1.0
authority	0.857	1.0
freshness	0.8	0.842
contradiction	0.5	0.667

Three gaps are known and named, because a score with no stated limit is a claim nobody can check.

01Two of the four severity tiers carry no cases.Every failure the corpus produces is at safety, rights, or money stakes. Nothing lands at Minor or Advisory, so the tier table reports them as not exercised rather than as passing.corpus

02The check reads metadata, so it cannot catch a document that lies about itself.All three misses are one document whose program period ended in its prose while its effective window still says null and its status still says current. A check that reads the window cannot see an expired document that reports the wrong window.freshness_violation

03The monitor declines to route 13 percent of its traffic.31 of 240 calls. None of those declines land on the calls the labels call undeterminable, so the routing is losing clean traffic rather than catching ambiguity. A declined call never reaches a provenance verdict.coverage

The two false alarms are one case, and it is a genuine question rather than a defect. A not-yet-in-force deposit document reached the context, and the agent answered without quoting from it. Flagging it says the retrieval was wrong. Not flagging it says the answer was fine. The checks read retrieval, so they flag it. Which is correct belongs to the rubric, not to the check.

The method, running

The 0.944 above is an offline score against a frozen set. The same checks now run as a sweep inside AESOP, against a test agent replaying the corpus, and fill a dashboard. Nothing new is being judged — the checks that scored the fixture are the checks that fill the board.

It reads the kind of retrieval trace a LangChain retriever on Databricks Vector Search already returns — the documents pulled and their metadata — so it plugs onto that stack by consuming what it produces, not by re-platforming anything.

It runs with no model call. The checks are re-derived from each interaction's retrieval trace, so a finding is not a model's opinion about an answer. It is arithmetic over what the retrieval did, and it re-runs to the same result every time.

        flowchart TD
          LIVE["Test agent replaying the Xcel-shaped corpus"]
          ITRACE["Interaction trace: retrieval provenance"]
          DET["Deterministic checks: entity, stale"]
          PROV["Provenance checks: grounded, in-scope, authoritative, consistent"]
          FIND["Findings, each named to a check"]
          DELTA["Deterministic prevalence delta"]
          DASH["Dashboard: counters, feed, breakdowns, trend"]
          REPAIR["Fix the knowledge base, rate bends down"]

          LIVE --> ITRACE
          ITRACE --> DET
          ITRACE --> PROV
          DET --> FIND
          PROV --> FIND
          FIND --> DELTA
          DELTA --> DASH
          DASH --> REPAIR
          REPAIR --> LIVE

          classDef agent fill:#C64A1122,stroke:#C64A11,stroke-width:2px
          classDef trace fill:#6D3FC422,stroke:#6D3FC4,stroke-width:1.5px
          classDef det fill:#0E7C6622,stroke:#0E7C66,stroke-width:1.5px
          classDef finding fill:#B03A2622,stroke:#B03A26,stroke-width:2px
          classDef dash fill:#1F5E9322,stroke:#1F5E93,stroke-width:1.5px

          class LIVE agent
          class ITRACE trace
          class DET,PROV det
          class FIND,DELTA finding
          class DASH,REPAIR dash

The loop closes on repair. A knowledge-base fix lands, the next sweep re-derives the findings, and the rate moves. That is the whole point of running it live rather than once.

The dashboard

Open an agent's Monitor page in AESOP and the board reads the agent's record back to you. It is newest-first, it cuts by where the failure happened, and it trends over time so a fix is visible rather than claimed.

Counters

What the sweep has seen

Interactions, sessions, findings, and a findings-rate. The headline is the rate, because a raw count says nothing about how much traffic produced it.

Trace feed

Newest-first, marked

Each flagged turn carries the check that fired against it. You click a finding and land on the interaction that caused it, not a summary of it.

Findings panel

By check

freshness_violation, scope_violation, authority_regression, contradiction_risk. The same four checks that scored the fixture, counted live.

Where cuts

Company, jurisdiction, topic

By operating company (PSCo, NSPM, NSPW, SPS), by jurisdiction (CO, MN, WI, TX), and by topic. A finding is never just a count, it has a place.

Severity

Major / minor

Grouped by stakes, not by consequence. See the honesty note below for why the top tier reads "major" and never "critical".

Trend

Findings per sweep

Bars of finding counts with a findings-per-interaction rate beneath. Fix the knowledge base and the rate bends down on the chart. That is the proof the repair worked.

Two windows that matter first

The timeframe selector leads with two windows before the rolling ones, because the two questions an owner actually asks are "how is this agent doing" and "did my last change help."

01Since initial deploy.The agent's whole record, every interaction since it went live. The health of the thing as it stands.whole record

02Since the last update.Only what changed since the monitor last swept. Ship a knowledge-base fix, look here, see whether the rate moved on the traffic that came after it.delta

03Rolling 1h, 24h, 7d, all.The ordinary time windows underneath, for the slower read.rolling

Showing it live

A monitor that only runs once is a screenshot. This one you can run in front of someone. A Run conversations button replays curated, order-preserving deployment traffic a batch at a time, so the board visibly fills as you watch.

Replay

Fills as you watch

The Xcel-shaped corpus, in order, a batch per press. The counters climb, the feed scrolls, the trend grows a bar.

Idempotent

Re-pressing adds nothing

The sweep is a function of the traffic, not of how many times you ran it. Press it twice on the same batch and the numbers do not move.

Resettable

Back to zero, repeat

Clear the board and run it again. The demo is the method, not a recording of it.

What "major" means, and what it does not

The severity on the board stops at major. It never says critical. That is a deliberate limit, and it is the same one the frozen rubric already names.

The provenance layer proves a retrieval was wrong. Stale, out of scope, wrong authority. It does not prove a safety-critical consequence. A stale late fee and a stale safe-distance are the exact same check firing at wildly different stakes, and the monitor alone cannot tell them apart. So the top tier reads major, because that is what the evidence supports. "Critical" is reserved for a consequence the provenance layer cannot see on its own, and claiming it here would be overreach.

The check sees the retrieval, not the stakes. It can tell you the agent answered from the wrong document. It cannot tell you whether that document was about a fee or about a gas line. The person reading the board supplies the stakes. The monitor supplies the fact.

What generalises

On scoring

Recall alone is worthless

A monitor that catches every low-severity finding and misses every critical one has a recall of one. You need severity attached to every finding, and the transcript behind it. A count tells you nothing about what broke.

On the fixture

The retrieval did it, not the script

The failures were not written into the labels. The naive agent surfaced a superseded document and a cross-jurisdiction rule on its own, and the monitor caught what the retrieval produced.

On the spine

The AI reads, the code counts

The model answers, the checks read retrieval provenance rather than the answer prose, and the arithmetic is deterministic. So an auditor reproduces the findings exactly. The same cut is the limit: the checks pass the same way on a better or a worse model, because they never read the model.

The point of the whole build. A monitor that scores the answer can only tell you the answer was wrong. This one reads the evidence, so it tells you which document did it.