Monitor · Same Answer Every Run

Same input. Same finding. Every time.

This is the part of the monitor that decides whether an agent gave a bad answer, and it is not an AI model. It reads the documents the agent pulled to build its answer (the retrieval trace), plus the details attached to each one. Same documents in, same finding out, every time. No guessing, no randomness.

That is what makes a finding hold up: you can re-run it and get the exact same result, it tells you why it flagged something, and we have measured how often it is right.

Storage — where the answer comes from Compute — the engine and its five checks Consumer — the finding and its count

It reads the record your existing setup already produces

LangChain

What the retriever hands back

A LangChain retriever hands back the documents it pulled, each with its details attached. That list, which LangChain can hand over through its callbacks, is what the engine reads. Nothing gets rebuilt.

Databricks

Vector Search results

Databricks Vector Search returns each document chunk with its details: source, jurisdiction, the dates it is valid, who has authority over it. The engine compares exactly those details. Your setup already produces them.

The one requirement

It needs the fetch recorded

The honest ask: your system logs what it retrieved. LangChain can expose that. If the record is too thin to judge, the engine tells you instead of guessing.

What it checks, and why

scopeWrong jurisdiction or companyThe document is from a different jurisdiction or company than the one asking. A Colorado doc answering a Minnesota question.

freshnessOut of dateThe document has passed the dates it was valid for, or has been replaced. A stale rate beating the current one.

authorityIt used the weaker sourceA stronger document on the topic exists in the index, but the agent pulled a weaker one.

contradictionThe sources disagreeTwo pulled documents say different things about the same topic, so the answer depends on which one won.

entityThe wrong numberA transposed digit or an old figure, caught by a fixed rule against a known-good value, not by understanding.

Why you can trust a finding

Repeatable

Same answer every run

Same record in, same findings out, every time. No randomness, no AI call. A finding someone disputes can be re-run, and it comes out the same.

Explainable

It names the rule

Every finding points at the document, the field, and the rule it broke, like "this chunk is from NSPM, the caller is PSCo." A person can check each one by hand.

Measured

A known error rate

Checked against 240 calls graded by hand. Out of those 240, it called it right about 226 times (0.944 precision). Knowing its own error rate is the first thing an auditor asks for.

Reading the record is the whole point. A judge that reads the AI's written answer is just a second AI, and a second place for things to drift. This engine reads where the answer came from: what was pulled, from where, whose rules apply, and as of when. Then it compares the details. That is arithmetic, and arithmetic comes out the same every time. The AI judge is still there for the softer failures a record cannot show, but the monitor does not lean on it. It will not catch everything. What you get is a written list of failure types it does catch, short enough to check and open enough to extend as you find more.