KB Audit

Before AESOP builds an agent, it audits the knowledge base that agent would answer from. It grades every document, then searches the whole set for facts two files state differently. What comes back is a report a knowledge base manager can act on.

pre-build gate · 4 dimensions
per-doc + cohort-wide · full report

How the audit runs

Every document in the set, then the set as a whole.

Each document is scored independently and in parallel, then all of the scores are synthesized as one cohort into a single go / no-go verdict.

Inputs · KB audit Per-document scoring (parallel) Cohort-wide synthesis Verdict · report

Two passes: every document graded alone first, then the entire knowledge base checked against itself.

What you feed it

A target to judge against, and the material to judge.

The audit needs two things to mean anything: the agent’s PRD (so “complete” means something specific) and the documents that would become its knowledge. Pull either from a running project or an ad hoc upload.

Uploaded filesDocuments converted to text and scored as given — .md, .pdf, .docx, slides. Each document is capped at 12,000 characters for scoring.

Stored project docsDocuments already saved to the project are pulled back in by id and re-audited. No re-upload. Scoped to the project.

The PRDUp to 8,000 characters are stored as context; the first 3,000 go into each scoring pass so “completeness” is judged against the actual build target.

Pass one — each document, four dimensions

This pass looks inside one document at a time.

Every document is scored alone against four dimensions. This is where a file catches itself contradicting itself. Each dimension gets a model judgment at temperature zero; the weights below are code.

Readability 30Clean prose with context. Penalizes “see screenshot,” “as shown above,” and content that only lives in an image.

Clarity 30Is the meaning unambiguous? Punishes vague terms an agent could read two ways.

Consistency 25Internal agreement — does the file contradict itself on names, numbers, or policy? Within one document.

Completeness 15Coverage against what the PRD actually needs. Lightest weight: a gap signal, not the whole story.

doc_score = round( readability × 0.30 + clarity × 0.30 + consistency × 0.25 + completeness × 0.15 )

Weights are deterministic. Two runs over the same document agree, so the score is something you can stand behind.

Pass two — it searches the whole knowledge base

Disagreements across files are the ones that poison an agent.

Scoring one document at a time can only catch a file disagreeing with itself. The gaps that hurt are between documents: two files that state a price, a threshold, an SLA, or an owner differently. So after every document is scored, the audit reviews the entire set as one cohort and flags any two documents that make conflicting factual claims — numbers, dates, policy — wherever those files sit in the knowledge base.

Only real contradictions count. The synthesis is told to ignore style differences and flag facts that would produce a genuinely different answer.

Conflicting factsPairs of documents (A and B) that disagree on a fact, each with its quoted excerpt. Tagged high when the conflict is likely to cause a wrong answer, warning when lower-stakes.

Cross-doc terminologyThe same concept called different names across the files — so the agent switches vocabulary mid-conversation and looks broken to the user.

Results stream in as it runs

Not a cold wait.

Each document writes its score to the audit the moment it finishes, so a long run fills the report document by document instead of going silent. The audit is a background job with a real status — running, completed, failed.

Live partial resultsScores stream into the report as they land.

Graceful failureIf one document’s model call fails it scores a neutral 50 and the run continues — a failed file never blanks the whole audit. Unreadable uploads are skipped and reported. If the run itself dies, it flips to failed with the error stored.

What comes back — a report a KB manager can act on

Built to be handed over, not stared at.

The audit is not a single score dropped in a vacuum. It returns a fully detailed, stored report: every issue found, tagged, with its location and a suggestion. On top of that sits a short prioritized list of action items — the concrete fixes that resolve the highest-risk problems first. A knowledge base manager opens it and has a clear answer to two questions: what is broken, and what do I do about it, in what order.

Per-document findings for every file

Its four dimension scores, every issue it raised with the exact tag, up to three quoted snippets, and up to three per-issue suggestions for fixing it.

The labels it uses so issues are parseable, not prose

Each issue a model finds is tagged with one exact label — no freeform description to sort later.

`possible-image-content`	Data an agent needs (a table, a matrix) lives only in an image it can’t read.
`format-issues`	Structure that breaks reading or extraction.
`vague-language`	Terms an agent could interpret several ways.
`underspecified-references`	“See above” pointers with no content behind them.
`terminology-inconsistency`	One concept, several names inside a file.
`low-completeness`	Coverage gaps against the PRD.
`conflicting-numbers-or-dates`	A file’s own values or dates disagree.

Cohort findings across the whole set

Every cross-document conflict (with both excerpts and its severity) and the terminology drift across files.

Action items prioritized fixes

A short, ranked list of what to repair first — the things that most raise hallucination risk. Each line maps to something the manager can change in a document.

The verdict — a gate, not a grade

Overall score maps to a go / no-go.

The overall score is the mean of every document’s score, computed in code. That number maps to a risk tier with a plain-language verdict — the gate that decides whether the build proceeds.

The four tiers, with what the audit tells you at each.
Overall	Tier	Verdict — go or no-go
≥ 80	good	KB is in good shape. Proceed to build with normal diligence.
60–79	fair	Minor issues. Proceed with caution; address flagged items before evaluation.
40–59	needs work	Significant issues. Proceeding now raises hallucination risk. Fix high-severity items first.
< 40	at risk	Critical issues. Building on this KB will likely produce an agent that answers incorrectly or inconsistently.

The thresholds, tier names, and verdict text are code — the model supplies judgment on each document and the cohort; the system supplies the line. That split is what makes two runs agree and a go / no-go question answerable.

Why it sits before Build

Cheap fixes happen on the workbench, not in production.

An agent is only as truthful as the material it answers from. Fix a contradiction in a document and you fix every answer that would have repeated it. Fix it after the agent ships and you are patching production — slower, embarrassing, and it already told someone the wrong thing. The audit makes the fix cheap by making it early.

The honest framing: the audit raises the bar for the KB you feed it.

It cannot invent information that was never written down. Garbage in still fails — it just fails on the workbench, where a fix costs an edit, not an incident. And because it checks every document against every other document, the report a knowledge base manager acts on is built from the whole knowledge base, not a file at a time.