← Index
Legal AI OS

How Each Architecture Works

Data flow through the three options — from document to answer.

Option 1 — Not recommended

Vector DB / RAG Only

Pure semantic search with embeddings
      flowchart TD
        docs["Documents"]
        chunk["Chunk & Embed"]
        vector[("Vector Store
        LanceDB Enterprise")]
        query["User Query"]
        embed["Embed Query"]
        similarity["Similarity Search
        cosine distance"]
        retrieved["Retrieved Chunks
        top-k"]
        llm["LLM"]
        answer["Answer"]

        docs --> chunk
        chunk --> vector
        query --> embed
        embed --> similarity
        vector --> similarity
        similarity --> retrieved
        retrieved --> llm
        llm --> answer

        class docs,chunk,embed,similarity,retrieved roseNode
        class vector roseStore
        class query roseNode
        class llm llmNode
        class answer answerNode
    
~70% accuracy ceiling — No production legal AI system uses this alone. Embeddings fuzz out exact clause numbers, statute citations, and party names.
Option 2 — Production Baseline

Hybrid Vector + BM25 + Cross-encoder Rerank

Two parallel retrieval paths fused before the LLM
      flowchart TD
        docs["Documents"]

        subgraph dense["Dense Path"]
          chunk["Chunk & Embed"]
          vector[("Vector Store")]
        end

        subgraph sparse["Sparse Path"]
          bm25["BM25 Index"]
          keyword[("Keyword Index")]
        end

        query["User Query"]
        embed["Embed Query"]
        kwquery["Keyword Query"]
        vresults["Vector Results"]
        kresults["Keyword Results"]
        fusion["Fusion RRF
        Reciprocal Rank Fusion"]
        rerank["Cross-encoder Rerank
        relevance scoring"]
        llm["LLM"]
        answer["Answer"]

        docs --> chunk & bm25
        chunk --> vector
        bm25 --> keyword
        query --> embed & kwquery
        embed --> vresults
        kwquery --> kresults
        vector --> vresults
        keyword --> kresults
        vresults --> fusion
        kresults --> fusion
        fusion --> rerank
        rerank --> llm
        llm --> answer

        class docs,chunk,embed,kwquery,vresults,kresults blueNode
        class vector,keyword blueStore
        class bm25,query blueNode
        class fusion fusionNode
        class rerank blueNode
        class llm llmNode
        class answer answerNode
    
The production floor. Harvey AI, Brazilian legal platform (184,895 answers), and HyPA-RAG (NAACL 2025) all use this pattern. BM25 catches exact matches (statute § numbers, party names, dates). Cross-encoder reranks for relevance. Non-negotiable baseline.
Option 3 — Long-term Target

Full Hybrid with Knowledge Graph

KG pre-filtering + retrieval + hallucination verification
      flowchart TD
        docs["Documents"]
        kg[("Knowledge Graph
        entities & relations")]

        subgraph dense2["Dense Path"]
          chunk2["Chunk & Embed"]
          vector2[("Vector Store")]
        end

        subgraph sparse2["Sparse Path"]
          bm252["BM25 Index"]
          keyword2[("Keyword Index")]
        end

        query2["User Query"]
        prefilter{"KG Pre-filter
        dates, parties,
        jurisdictions, amounts"}
        vfiltered["Filtered Vector
        Results"]
        kfiltered["Filtered Keyword
        Results"]
        fusion2["Fusion RRF"]
        rerank2["Cross-encoder Rerank"]
        llm2["LLM"]
        answer2["Answer"]
        verify{"KG Hallucination
        Verification
        entity grounding"}
        verified["Verified Answer"]
        escalate["Human Escalation"]

        docs --> chunk2 & bm252
        chunk2 --> vector2
        bm252 --> keyword2
        query2 --> prefilter
        kg --> prefilter
        prefilter --> vfiltered & kfiltered
        vector2 --> vfiltered
        keyword2 --> kfiltered
        vfiltered --> fusion2
        kfiltered --> fusion2
        fusion2 --> rerank2
        rerank2 --> llm2
        llm2 --> answer2
        answer2 --> verify
        kg --> verify
        verify -->|"passes"| verified
        verify -->|"fails"| escalate

        class docs,chunk2,bm252,query2,vfiltered,kfiltered,rerank2 violetNode
        class vector2,keyword2 violetStore
        class prefilter,verify violetDiamond
        class kg kgNode
        class fusion2 fusionNode
        class llm2 llmNode
        class verified answerNode
        class escalate roseNode
    
Adds two KG capabilities: (A) structured pre-filtering narrows candidates by metadata before vector search, and (B) post-generation entity grounding verifies citations against the KG. HalluGraph: AUC 0.94 for entity error detection vs BERTScore 0.60. But partially research-grade — build toward it, don’t start here.

Summary

Which architecture fits where

Option 1
Fastest to prototype. Not production-ready for legal. Pure semantic search misses exact citations and structured metadata.
Option 2
Production baseline. Ship this now. Dual retrieval paths + reranking. Per-matter database-level isolation. Deployed in customer VPC.
Option 3
Long-term target. Build toward it as retrieval quality demands. KG adds structured filtering and hallucination guardrails.