Index/ AESOP/ Evaluation Wizard & Architecture
AESOP — Enterprise AI Agent Governance

Evaluation Wizard & Architecture

One unified wizard takes a system’s instructions, optional governance documents, and an optional live endpoint and routes them into the evaluation modes that fit. Each mode is a deterministic, agent-run pipeline with a human gate at the end. This page maps the wizard and the stack underneath it.

01

Four-step wizard flowMaterials → Governance → Endpoint → Capabilities

The wizard narrows from “what are you evaluating” to “how are you evaluating it.” Governance is optional; Endpoint appears only when a live-testing mode is selected.

01Materials
Provide the inputs for the evaluation.
  • Project name — identifies the run.
  • System instructions — upload or paste.
  • Governance documents — attach the policy layer to evaluate against.
  • Vital context — anything the evaluators need to read the agent fairly.
02Governance
Optionally attach governance documents.
  • Select from the document library.
  • Optional — skip when no policy layer is in scope.
03Endpoint
Only shown when live-testing phases are selected.
  • Chatbot API URL — optional.
  • Hidden for Quality-Analysis-only runs.
  • Providing it unlocks the live-testing phases.
04Capabilities
Toggle evaluation phases on or off.
  • Quality Analysis pre-checked — the default gate.
  • Each phase can be switched off for narrower runs.
  • Seven evaluation modes available on the platform.
02

Evaluation modesthe five wizard pipelines

Each mode carries its own requirement: an instruction set, a live endpoint, or both. Static-analysis modes never touch the endpoint; live modes are the only ones that do.

Quality
AnalysisNeeds instructions
Quality Analysis — Full 5-agent scored pipeline evaluating Instructions, Ethos, Bias, and Safety with weighted Synthesis scoring and AESOP certification.
Pipeline Instructions → Ethos → Bias → Safety → Synthesis
Custom
SuiteNeeds instructions
Custom Test Suite — Generates a tailored evaluation rubric from your instructions, then creates targeted test scenarios from that rubric.
Pipeline Instructions → Custom Rubric → Test Scenarios → Results
Black Box
InvestigationNeeds endpointNo instructions
Black Box Investigation — Interview-based discovery of an unknown chatbot. Probes the endpoint with strategic questions to map its capabilities, boundaries, and behavior with no prior knowledge.
Pipeline Endpoint → Interview → Discovery → Report
Adversarial
Red-TeamNeeds both
Adversarial Red-Team — Cave of Shadows attack scenarios. Generates adversarial prompts from analysis reports, then executes them against your live chatbot to surface exploitable weaknesses.
Pipeline Reports → Attack Scenarios → Execute → Results
Functional
TestingNeeds endpoint
Functional Testing — Executes test scenarios (from Custom Test Suite or Red-Team) against a live chatbot endpoint. Sends real requests, captures responses, and scores pass/fail outcomes.
Pipeline Test Scenarios → Live Requests → Response Capture → Pass/Fail Scoring
03

Tech stackfrontend → backend → data → AI → infra

Enterprise-grade by construction: SOC 2 Type II platforms, TLS + AES-256 encryption, JWT + row-level-security auth, user-controlled data, no AI training on your inputs, and a 99.9% SLA.

Frontend
Next.js 16 · React 19 · TypeScript
  • Next.js 16 App Router with server and client components.
  • React 19 with functional components and hooks; TypeScript strict mode.
  • Tailwind CSS v4 for utility-first styling.
  • Libraries: Recharts (KPI dashboards), Lucide React (icons), Supabase JS (auth + realtime), Server-Sent Events (streaming analysis progress).
Backend
FastAPI · Python · Uvicorn
  • FastAPI with async route handlers and Pydantic models; Uvicorn ASGI serving; SSE streaming via Starlette.
  • JWT middleware for Supabase token validation on every request.
  • Agent architecture: specialized agents inheriting a BaseAgent — deterministic pipelines, probabilistic evaluation, human-in-the-loop at every gate.
  • Prompt files stored as plain .txt for easy iteration; sequential pipeline with report chaining; httpx for async chatbot testing.
Database
& Auth
Supabase · PostgreSQL · Row Level Security
  • Supabase-hosted PostgreSQL with RLS policies; tables for analyses, evaluations, users, documents, help.
  • Voyage AI vector embeddings (512-dim) for the RAG help system.
  • SQL migrations tracked in /database/migrations/.
  • Auth: email/password + magic links; JWT validated server-side; roles org_admin / evaluator; RLS isolates tenant data.
AI &
Evaluation
Claude Sonnet 4.5 · 21+ agents · seven modes
  • Claude Sonnet 4.5 (claude-sonnet-4-5-20250915) for primary agent evaluations; DeepSeek for batch / summary tasks.
  • Temperature 0.1 for consistent, reproducible scoring; 8K max output tokens per agent (16K for Custom Rubric).
  • Cost tracking per analysis with token-level granularity.
  • Pipelines: 5-agent Quality Analysis; Custom Rubric + scenario generation; interview-based Black Box Investigation; Cave of Shadows Red-Team + execution; Functional Testing request/response scoring.
Infra
Vercel · Fly.io · GitHub Actions
  • Frontend on Vercel with automatic preview deploys; backend on Fly.io via GitHub integration.
  • Supabase managed cloud for database + auth; custom domains with SSL on frontend and API.
  • CI/CD: GitHub Actions help-doc reindexing on push; auto-deploy on merge to main; pytest suite for backend validation; per-service environment variables.