Monitor · Production

Catch the drift before the customer does.

An agent ships, then it drifts. Last week it was right. This week a stale article wins and it gives the wrong answer. Monitoring is how you spot that and fix it.

Thumbs up and thumbs down can't do it. You need a baseline, a way to see what changed, and a way to trace the problem back to the document that broke.

      flowchart TD
        CAPTURE["Save each interaction and the documents it pulled"]
        TMAN["Trigger: dashboard button"]
        TCAD["Trigger: on a schedule"]
        TEVT["Trigger: a model, KB, or instruction change"]
        SWEEP["Sweep: one run over a batch"]

        TMAN --> SWEEP
        TCAD --> SWEEP
        TEVT --> SWEEP
        CAPTURE --> SWEEP

        SWEEP --> DET["Fixed checks, no LLM"]
        SWEEP --> PROV["Source checks, read the trace"]
        SWEEP --> JUDGE["Drift judge, optional"]

        PROV --> FRESH["Stale document won"]
        PROV --> SCOPE["Wrong jurisdiction answered"]
        PROV --> AUTH["Governing document skipped"]
        PROV --> CONTRA["Sources disagree"]

        DET --> FIND["Findings, each tied to its interaction"]
        FRESH --> FIND
        SCOPE --> FIND
        AUTH --> FIND
        CONTRA --> FIND
        JUDGE --> FIND

        FIND --> PREV["Rate: how often each mode shows up"]
        PREV --> DELTA["Change vs the last sweep"]
        DELTA --> GATE{"Did a mode rise past the set threshold?"}

        GATE -- "yes" --> ALERT["Drift alert: names the mode and the rise"]
        GATE -- "no" --> DASH["Dashboard"]
        ALERT --> DASH

        DASH --> TRIAGE["Open the flagged trace"]
        TRIAGE --> REPAIR["Fix the source or the instructions"]
        REPAIR --> REMEASURE["Re-measure on the next sweep"]
        REMEASURE --> CAPTURE

        classDef capture fill:#C64A1122,stroke:#C64A11,stroke-width:2px
        classDef trig fill:#6B625915,stroke:#6B6259,stroke-width:1.5px
        classDef det fill:#8A5A0B22,stroke:#8A5A0B,stroke-width:1.5px
        classDef prov fill:#1F5E9322,stroke:#1F5E93,stroke-width:1.5px
        classDef check fill:#1F5E9315,stroke:#1F5E93,stroke-width:1.2px
        classDef judge fill:#0E7C6622,stroke:#0E7C66,stroke-width:1.5px
        classDef find fill:#6D3FC422,stroke:#6D3FC4,stroke-width:1.5px
        classDef calc fill:#1F5E9322,stroke:#1F5E93,stroke-width:1.5px
        classDef gate fill:#8A5A0B22,stroke:#8A5A0B,stroke-width:2px
        classDef alert fill:#B03A2622,stroke:#B03A26,stroke-width:2px
        classDef dash fill:#C64A1122,stroke:#C64A11,stroke-width:2px
        classDef repair fill:#0E7C6622,stroke:#0E7C66,stroke-width:1.5px

        class CAPTURE capture
        class TMAN,TCAD,TEVT trig
        class SWEEP capture
        class DET det
        class PROV prov
        class FRESH,SCOPE,AUTH,CONTRA check
        class JUDGE judge
        class FIND find
        class PREV,DELTA calc
        class GATE gate
        class ALERT alert
        class DASH dash
        class TRIAGE,REPAIR,REMEASURE repair

Five failure modes to watch

Hallucination

No grounding

A confident answer with no source under it at all.

Retrieval failure

Wrong document

The wrong doc pulled, or the right one never surfaced.

Attribution error

Wrong source

A right-looking answer built on the wrong source. The wrong phone number.

Recency

Stale article wins

A superseded document favored over a newer one.

Guardrail breach

Broke the rules

Break character, leak system instructions, or output something unsafe or biased.

Two layers, one signal

Reads the trace · no LLM

Fixed checks + source checks

It checks the structure and the trace of what the agent retrieved: what it pulled, where it came from, and whose authority it carried. It reads the sourcing, not the wording, so the same trace scores the same every sweep.

Judges the meaning · optional

Drift judge (optional)

An LLM pass over meaning a trace can't show. Off unless a key is set, because the fixed checks already carry the drift signal on their own.

The checks it runs

freshnessA superseded document wonThe retrieval pulled a doc that a newer version has already replaced.

scopeOut of the caller's jurisdictionA document answered for the wrong operating company or state. The signal that fires on a loosened filter.

authorityA governing document was skippedA weaker source answered when a stronger one already covered the topic.

contradictionThe sources disagreeThe retrieved chunks conflict with each other, so the answer rests on a coin flip.

entityThe wrong value came outFixed checks catch a transposed number or a stale figure by its shape, not its meaning.

Inside one sweep

Step 1

Fixed + source checks

Run both check families over the batch with no LLM. Every finding is tied to the interaction that produced it.

Step 2

Drift judge, optional

An LLM pass scores how much the meaning eroded. Skipped without a key; the sweep still produces a full signal.

Step 3

Rate

How often each mode shows up per interaction. A rate you can compare sweep to sweep, not a raw count.

Step 4

Change

Compare each mode's rate to the last sweep. Drift fires when a mode rises past the threshold on a sweep big enough to trust.

Step 5

Alert

The alert names the mode, the direction, and the old-to-new rate. Re-run the sweep and it adds nothing new.

Step 6

Dashboard + repair

The operator reads the delta, opens the flagged trace, fixes the source or the instructions, and the next sweep re-measures.

The demo: two sweeps, one change

Wave 1 · baseline

The agent, running clean

Sweep 22 live-shaped interactions. Scope violations land at 4.5% — 1 of 22. This is the baseline the next sweep is measured against.

Wave 2 · after a change

A jurisdiction filter loosens upstream

Sweep again. Scope violations rise to 22.7% — 5 of 22. The delta fires a drift alert naming scope_violation, up 18 points. The rate rose past the threshold; the regression caught without anyone reading a transcript.

Honesty, not magic. This runs live on a deployed agent, but it is periodic re-evaluation and drift comparison, not continuous stream ingestion — you can't live-instrument a vendor agent from outside. The waves are a governed synthetic corpus built to reproduce real failure modes on demand. The claim is about what the structure catches and how reliably the alert fires, not that every mode has been seen in the wild.