Legal AI OS

Legal AI Storage Architecture

Three options. One production baseline. A tiered path to full graph-grounded retrieval.

Option 1

Vector DB / RAG Only

Pure semantic search

What it is

Chunk documents, embed them, query by cosine similarity. One vector store. No hybrid retrieval.

Accuracy

~70% ceiling (Lettria benchmark). Insufficient for regulated legal use.

Why it fails

Embeddings fuzz out exact clause numbers, statute citations, party names. No structured filtering. No entity awareness.

Who uses it

No production legal AI system. It’s the starting point everyone moves past.

Fast to prototype. Not production-ready for legal.

Option 2 — Recommended

Hybrid Vector + BM25 + Rerank

The production baseline

What it is

Dense vector embeddings + BM25 sparse/keyword index + cross-encoder reranker. Two retrieval paths fused before the LLM sees results.

Accuracy

Significantly above vector-only. BM25 catches exact matches — statute § numbers, party names, dates. Cross-encoder reranks for relevance.

Who uses it

Harvey AI (LanceDB Enterprise + BM25 + cross-encoder). Brazilian legal platform (184,895 production answers). HyPA-RAG (NAACL 2025, dense + BM25 + KG triplets via RRF).

Security

Per-matter database-level isolation. Deployed in customer VPC. Nothing leaves the client’s domain.

This is the production floor. Non-negotiable baseline.

Option 3

Full Hybrid with Knowledge Graph

Premium / long-term target

What it is

Option 2 plus a knowledge graph layer providing two new capabilities:

Capability A — Structured pre-filtering

KG narrows candidates by dates, parties, monetary amounts, jurisdictions before vector search. Neo4j contract agent pattern. Catches queries like “all contracts signed after June 2023 involving Delaware parties with consideration > $5M.”

Capability B — Hallucination verification

Entity grounding + relation preservation scoring against the KG as a guardrail. HalluGraph (arXiv Dec 2025): AUC 0.94 for entity error detection vs BERTScore 0.60. But: preprint only, synthetic data, not production-validated.

Production reality check

Legislation citations verify at 81.7%. Case law citations at only 47.1%. KG helps most where the gap is widest.

Powerful but partially research-grade. Worth building toward, not starting with.

Tiered Adoption Path

Ship the baseline now. Layer on graph capabilities as retrieval quality demands it.

Non-negotiable baseline — Option 2

LanceDB Enterprise (VPC deployment) + BM25 sparse index + cross-encoder reranker. Per-matter database-level isolation. VPC deployment for client data. This is what Harvey AI runs in production.

Knowledge graph + reference auditing

Add a knowledge graph for structured pre-filtering:

Entities: cases, statutes, clients, matters, dates, parties, monetary amounts
Relations: cites, reverses, supersedes, involves, governs

Add post-generation reference auditing: extract every citation from generated output, resolve against a structured reference store (court, year, docket, holding), flag unresolvable citations for human review. This directly addresses the 81.7% / 47.1% verification gap.

Full entity grounding guardrails

Full HalluGraph-style entity grounding: composite entity + relation preservation scoring against the KG as a guardrail. Triggers re-retrieval or human escalation on failure.

Still research-grade — budget for it, don’t build until Tiers 1–2 are solid.

Production Metrics

Numbers that drive the architecture decision

~70%

Vector-only accuracy ceiling

Lettria benchmark across regulated verticals

Below threshold

81.7%

Legislation citation verification rate

Brazilian legal platform — 184,895 production answers

Good

47.1%

Case law citation verification rate

Same platform. Identified as “the primary bottleneck for practitioners.”

Primary bottleneck

AUC 0.94

HalluGraph entity error detection vs BERTScore 0.60

arXiv Dec 2025 — 550 synthetic contract QA pairs

Research-stage

Sources

Harvey AI Engineering Blog (2026) HyPA-RAG / NAACL 2025 Figueiredo et al. / PROPOR 2026 HalluGraph / arXiv 2512.01659 Lettria / Qdrant Case Study Neo4j GraphRAG Contract Agent