Xcel demo · Index layer

The Databricks Index Layer

Thirty test documents, each one labelled at the top with its state, its dates, how official it is, and whether it is still current. A script reads those labels, turns away any document that is missing one, and loads the rest into a searchable store.

That store is what the agent searches. The labels travel with each document into the store. They are the facts the monitor's checks depend on.

Storage — the docs, the table, the index Compute — the gate and the query Consumer — the monitor that reads

Labelled documents in, a searchable store out

Each document is plain text with a block of labels at the top. Those labels are the whole point. Without them a document is just words, and words alone cannot tell you whether a policy is still in force or which state it belongs to.

The search store is a copy of a table, so a label can only reach the search if it is a column in that table first. That is why every label has to be its own column before anything is searchable.

Step 1

Read the labels

Each file is split into its label block and its body text. A file with no labels at all is not treated as a knowledge-base document and is skipped. A file with only some of its labels is a real gap worth flagging.

Step 2

Check them against the rules

Thirteen labels are required. A document missing its state or its start date fails the check. So does a label with an unexpected value, like a state that is not one of Colorado, Minnesota, Wisconsin, Texas, or New Mexico.

Step 3

Split into sections and store

Each document is split into sections the same way every time. Every section becomes one row, and each label becomes its own column on that row, so it can be searched and read back later.

Step 4

Make it searchable

The store builds a meaning-based index over the body text so the agent can find the closest sections, and keeps the labels attached to each one so they can be filtered on and returned with every result.

The gate: an unlabelled document never gets in

The labels are what make a document checkable, so a document missing them must not slip in quietly. By default the script stops and complains. It names the document and the labels it is missing, and refuses to load anything until that is fixed.

There is a gentler mode for when you want to load anyway: it lets the document in but marks it as incompletely labelled, so the gap is visible in the store instead of buried in a file. Either way the gap gets recorded, never ignored.

A missing label is not a cosmetic problem. If a document has no state, the wrong-state check has nothing to compare against. If it has no start date, the out-of-date check cannot tell whether the document still applies today. Turning the document away at the door is what stops those checks from quietly passing everything later.

Why the labels have to live in the table

The free tier of Databricks only offers one kind of search store, and that kind works by copying a table. There is no way to attach extra information to the search directly.

That one constraint decides the whole shape. Since the search is just a copy of the table, it cannot carry any label the table does not already have. So every label has to be a column in the table, and the search copies those columns across.

The load-bearing part: which labels get copied across

Fourteen labels are copied onto the search, alongside the document text itself. Copying them across does two things: the search can filter on them, and every result comes back with them attached.

Column	What it holds	Check it lets fire
`status`	current, superseded, draft	freshness
`effective_from` / `effective_to`	the window the document governs	freshness
`superseded_by`	doc_id of the replacement	freshness
`jurisdiction`	the state the document is written for	scope
`operating_company`	PSCo, NSPM, NSPW, SPS	scope
`topic`	the subject two documents must share	contradiction
`claim`	the single comparable statement	contradiction
`authority`	tariff, policy, procedure, faq	authority
`doc_id`	the stable name of the source	every finding

The rest of the copied labels are the document's title, its owner, where it came from, and when it was last updated, plus the body text the search is built from. Subject and claim are the two people forget. Without a shared subject, the conflicting-document and lower-authority checks have nothing to group together, so they cannot run at all.

One layer of two the monitor reads

This layer attaches the facts. The search carries them into the record. The monitor compares them against fixed rules and returns the same finding every time it sees the same situation.

How this layer feeds the AESOP monitor. The labels are copied onto the search, so every document a search returns arrives tagged with whether it is current, which state it covers, how official it is, its dates, and its subject. The monitor's checks for out-of-date, wrong-state, lower-authority, and conflicting documents read exactly those labels. Leave one out of the copy and the matching check goes quiet. The labels are not documentation. They are the wiring.