The search records which documents it used to answer each question. This is what the monitor does with that record: it runs five fixed checks over it, and writes down anything that looks wrong.
There is no AI grading the answer here. Each check is a plain rule comparing the documents' labels. Same record in, same findings out, every single time. That is what makes it something you can audit.
The search hands over one record per answer: the documents it used, each carrying its labels, plus a note of who was asking and from which state. The engine reads that record and nothing else. It never reads the answer the model wrote. Here is what it does with it, in four steps.
One answer, one record: the documents behind it with their labels, and the caller's state and company. No model output, no live database, no network call. Just the facts of what was retrieved.
Each check is a plain comparison of labels, listed below. They do not learn, drift, or disagree with themselves. The same record always produces the same findings, which is the whole reason they can be trusted in an audit.
Every problem becomes one short sentence naming the document and what went wrong: “used a Minnesota document for a Texas caller.” A clean record produces no findings at all.
Each finding gets a severity, duplicates are merged, and the ones that matter become alerts, routed to the owner of the document at fault. No one has to be watching a dashboard.
Each rule asks one question of the retrieved documents. The first four catch a specific kind of wrong answer. The fifth catches the case where the question could not be placed at all.
| Rule | Fires when | Severity |
|---|---|---|
| Out of date | A retrieved document is marked expired, draft, or superseded, or its dates do not cover the day it was used. A document that was never cleared to answer is answering. | high |
| Wrong state | A retrieved document covers a different state or company than the caller. A Texas caller served a Minnesota policy. Documents from a state the caller did not ask about do not belong in the answer. | high |
| Lower authority | An FAQ or procedure answered a question that a tariff or policy also covers. The official source should have governed; a weaker one answered over it. | high |
| Conflicting documents | Two retrieved documents on the same subject, for the same state, state different things. Different rules across states is correct; different rules for the same state is a conflict. | medium |
| Couldn't classify | The question could not be placed to a subject with enough confidence to check it. Not a wrong answer, but a blind spot worth surfacing rather than hiding. | medium |
There is a sixth outcome the engine refuses to skip: if a document is missing the label a check needs, that gap is itself recorded as a low-severity finding. A monitor that shrugs at a document it cannot read is not a monitor.
The obvious alternative is to have a second model read each answer and grade it. That means one AI checking another, and the checker has its own ways to be wrong, to change its mind between runs, and to drift over time.
A rule does not do any of that. It compares labels against a fixed standard and returns the same verdict every time. You can read the rule, you can predict what it will say, and you can defend the finding to someone who was not in the room. For a monitor whose job is to be trusted, that is the point.
The rules engine is the fast, certain core of every sweep. It runs with no model and no network, so it finishes in a blink and produces the findings the board shows. A separate step then looks across sweeps to see whether any failure is becoming more common over time.