A customer asks a question. This layer searches the knowledge base, pulls back the documents that read closest to the question, and hands them to the model to answer from. It does the simplest possible version of that on purpose.
It does not filter by state, and does not filter out old documents. So an expired policy, or a policy from the wrong state, can end up in the answer. But the search writes down exactly which documents it used. That written record is what the monitor reads.
Storage — the index and what it returns Compute — the retriever and the trace Model — the answer writer Consumer — the caller and the monitor
What happens on one question
The path is short. Four steps from the customer's words to a written answer, and one more that saves the record for the monitor to check later.
Step 1 · Search
Take the question as-is
The question is passed straight through. The search does not try to work out which state the caller is in, and it does not narrow the results to current documents. It just asks the knowledge base for the closest matches and says how many to return.
Step 2 · Databricks
Find the closest matches
The knowledge base ranks every document section by how close it reads to the question and returns the top four. Whether a section is current, which state it covers, or how official it is makes no difference to the ranking. Closest in meaning wins.
Step 3 · Assemble
Hand the matches to the model
The matches go to the model along with the tags each one carries: whether it is current or expired, which state it is for, how official it is, and its dates. The model can see that a document is expired. The search still hands it over anyway.
Step 4 · Model
Write the answer
The model answers only from the passages it was given. If two of them disagree, it picks one and names the document it used. The answer is plain prose, and the prose is not what the monitor grades.
No filter is the point, not a mistake
The obvious fix is one line: tell the search to only return current, in-state documents. Add that line and there is nothing left to demo.
This layer is built to fail the way a real system fails. The knowledge base drifts out of date, nobody ever wrote the rule that keeps one state's documents away from another state's callers, and the agent keeps answering as if everything were fine. Cleaning that up here would hide the exact failures the monitor is meant to catch.
So instead, the search records what it did: it notes that no state filter and no date filter were applied. That note is not an oversight. It is the layer stating plainly that it filtered nothing.
"No state filter, no date filter" is the diagnosis. You do not have to guess why an expired document reached the answer. The record says no date filter was applied. You do not have to guess why a Texas caller got a Minnesota policy. The record says no state filter was applied. The cause is written down, every single run, in the same place.
What the record carries
Every document the search pulled comes back in full: its text, plus the tags the knowledge base attached to it. The monitor reads these tags. It never reads the answer the model wrote.
Field on each chunk
What it holds
Which check reads it
status
current, superseded, or draft
freshness
superseded_by
doc_id of the replacement, else null
freshness
jurisdiction
CO, MN, WI, TX, NM
scope
operating_company
PSCo, NSPM, NSPW, SPS
scope
topic
the subject two documents must share
contradiction
claim
the single comparable statement
contradiction
authority
tariff, policy, procedure, faq
authority
The record also saves which filters were applied, how many results were asked for, and the exact wording of the search. Together they say what was asked, what came back, and what was filtered out. On this agent, nothing was filtered out.
One layer of two the monitor reads
The checks run on the tags attached to the documents, not on the sentence the model wrote. Grading the answer would mean a second AI reading the first one, with its own ways to be wrong. This layer hands over plain facts instead.
The search produces the record. The knowledge base attaches the tags to every document. The monitor compares those tags against fixed rules and returns the same finding every time it sees the same situation.
How this layer feeds the AESOP monitor. Every answer comes with a record of the documents behind it, each tagged with whether it is current, which state it covers, how official it is, its dates, and its subject, plus a note of which filters were applied. The monitor's checks for out-of-date, wrong-state, lower-authority, and conflicting documents read exactly that record. Rename a tag here and the matching check goes blind. The search is the supply line; the monitor is the reader.