Index/ Legal/ Storage Architectures
Legal AI OS — storage, in flow

How Each Architecture Works

Data flow through the three options — from document to answer.

1

Vector DB / RAG Only

Option 1 — Not recommended

Pure semantic search with embeddings

Documents Chunk & Embed split + embed chunks Vector Store LanceDB Enterprise User Query Embed Query vectorize the question Similarity Search cosine distance Retrieved Chunks top-k LLM Answer

~70% accuracy ceiling. No production legal AI system uses this alone. Embeddings fuzz out exact clause numbers, statute citations, and party names.

2

Hybrid Vector + BM25 + Cross-encoder Rerank

Option 2 — Production Baseline

Two parallel retrieval paths fused before the LLM

Documents User Query Chunk & Embed Dense Path Vector Store embeddings index Embed Query vectorize the question Vector Results cosine top candidates BM25 Index Sparse Path Keyword Index exact-match tokens Keyword Query statute § party names Keyword Results lexical matches Fusion RRF Reciprocal Rank Fusion Cross-encoder Rerank relevance scoring LLM Answer

The production floor. Harvey AI, Brazilian legal platform (184,895 answers), and HyPA-RAG (NAACL 2025) all use this pattern. BM25 catches exact matches (statute § numbers, party names, dates). Cross-encoder reranks for relevance. Non-negotiable baseline.

3

Full Hybrid with Knowledge Graph

Option 3 — Long-term Target

KG pre-filtering + retrieval + hallucination verification

Documents User Query Knowledge Graph entities & relations Chunk & Embed Dense Path Vector Store embeddings index Filtered Vector Results BM25 Index Sparse Path Keyword Index exact-match tokens Filtered Keyword Results KG Pre-filter dates, parties, jurisdictions, amounts Fusion RRF merge both result sets Cross-encoder Rerank relevance scoring LLM Answer KG Hallucination Verification entity grounding Verified Answer Human Escalation passes fails

Adds two KG capabilities: (A) structured pre-filtering narrows candidates by metadata before vector search, and (B) post-generation entity grounding verifies citations against the KG. HalluGraph: AUC 0.94 for entity error detection vs BERTScore 0.60. But partially research-grade — build toward it, don’t start here.

Σ

Summary

Which architecture fits where
Option 1
Fastest to prototype. Not production-ready for legal. Pure semantic search misses exact citations and structured metadata.
Option 2
Production baseline. Ship this now. Dual retrieval paths + reranking. Per-matter database-level isolation (ABA Model Rule 1.6). Deployed in customer VPC.
Option 3
Long-term target. Build toward it as retrieval quality demands. KG adds structured filtering and hallucination guardrails.