← Index

AESOP

Unified evaluation wizard for AI system instructions, governance, and live endpoint testing

• Four-Step Wizard Flow
1
Materials
Provide the inputs for your evaluation
  • Project name
  • System instructions (upload or paste)
  • Governance documents
  • Vital context
→
2
Governance
Optionally attach governance documents
  • Select from document library
  • Optional — skip if not needed
→
3
Endpoint
Only shown when live-testing phases are selected
  • Chatbot API URL (optional)
  • Hidden for Quality Analysis-only runs
  • Unlocks live testing phases
→
4
Capabilities
Toggle evaluation phases — Quality Analysis pre-checked
  • Quality Analysis on by default
  • Toggle phases on/off
  • 7 evaluation modes
• Seven Evaluation Modes
QA
Quality Analysis
Needs: Instructions
Full 5-agent scored pipeline evaluating Instructions, Ethos, Bias, and Safety dimensions with weighted Synthesis scoring and AESOP certification.
Instructions → Ethos → Bias → Safety → Synthesis
CTS
Custom Test Suite
Needs: Instructions
Generates a tailored evaluation rubric from your instructions, then creates targeted test scenarios from that rubric.
Instructions → Custom Rubric → Test Scenarios → Results
BBI
Black Box Investigation
Needs: Endpoint NO Instructions
Interview-based discovery of an unknown chatbot. Probes the endpoint with strategic questions to map its capabilities, boundaries, and behavior without any prior knowledge.
Endpoint → Interview → Discovery → Report
CoS
Adversarial Red-Team
Needs: Instructions Needs: Endpoint
Cave of Shadows attack scenarios. Generates adversarial prompts from analysis reports, then executes them against your live chatbot to find exploitable weaknesses.
Reports → Attack Scenarios → Execute → Results
FT
Functional Testing
Needs: Endpoint
Execute test scenarios (from Custom Test Suite or Red-Team) against a live chatbot endpoint. Sends real requests, captures responses, and scores pass/fail outcomes.
Test Scenarios → Live Requests → Response Capture → Pass/Fail Scoring
Enterprise-Grade Infrastructure
Built on SOC 2 Type II certified platforms with end-to-end encryption, role-based access control, and full audit trails.
Compliance
SOC 2 Type II
Encryption
TLS + AES-256
Auth
JWT + RLS
Data
User-Controlled
AI Privacy
No AI Training
Uptime
99.9% SLA
FE
Frontend
Next.js 16 · React 19 · TypeScript

Core Framework

  • Next.js 16 App Router with server and client components
  • React 19 with functional components and hooks
  • TypeScript in strict mode for type safety
  • Tailwind CSS v4 for utility-first styling

Key Libraries

  • Recharts for KPI dashboards and analytics charts
  • Lucide React for consistent iconography
  • Supabase JS client for auth and real-time
  • Server-Sent Events (SSE) for streaming analysis progress
Next.js 16 React 19 TypeScript Tailwind v4 Recharts Lucide Supabase JS EventSource
BE
Backend
FastAPI · Python · Uvicorn

API Framework

  • FastAPI with async route handlers and Pydantic models
  • Uvicorn ASGI server for high-concurrency serving
  • SSE streaming via starlette for real-time progress updates
  • JWT middleware for Supabase token validation

Agent Architecture

  • Specialized agents inheriting from BaseAgent — deterministic pipelines, probabilistic evaluation, human-in-the-loop at every gate
  • Prompt files stored as .txt for easy iteration
  • Sequential pipeline with report chaining between agents
  • httpx for async chatbot endpoint testing
FastAPI Uvicorn Pydantic Anthropic SDK httpx SSE PyJWT slowapi
DB
Database & Auth
Supabase · PostgreSQL · Row Level Security

Data Layer

  • Supabase-hosted PostgreSQL with RLS policies
  • Tables for analyses, evaluations, users, documents, and help
  • Voyage AI vector embeddings for RAG help system (512 dims)
  • SQL migrations tracked in /database/migrations/

Authentication

  • Supabase Auth with email/password and magic links
  • JWT tokens validated server-side on every API request
  • Role-based access: org_admin, evaluator
  • Row Level Security ensures tenant data isolation
Supabase PostgreSQL JWT RLS Voyage AI pgvector
AI
AI & Evaluation
Claude Sonnet 4.5 · 21+ Agents · 7 Evaluation Modes

Model & SDK

  • Anthropic Claude Sonnet 4.5 (claude-sonnet-4-5-20250915) for primary agent evaluations; DeepSeek for batch/summary tasks
  • Temperature 0.1 for consistent, reproducible scoring
  • 8K max output tokens per agent (16K for Custom Rubric)
  • Cost tracking per-analysis with token-level granularity

Evaluation Pipeline

  • Quality Analysis: 5-agent sequential pipeline with report chaining
  • Custom Test Suite: custom rubric + scenario generation pipeline
  • Black Box Investigation: interview-based endpoint discovery
  • Adversarial Red-Team: Cave of Shadows attack scenarios + execution
  • Functional Testing: live chatbot request/response scoring
Claude Sonnet 4.5 Anthropic SDK 21+ Agents 7 Evaluation Modes Voyage AI
OPS
Infrastructure
Vercel · Fly.io · GitHub Actions

Hosting

  • Frontend deployed to Vercel with automatic preview deploys
  • Backend deployed to Fly.io with GitHub integration
  • Supabase managed cloud for database and auth
  • Custom domains with SSL on both frontend and API

CI / CD

  • GitHub Actions for help doc reindexing on push
  • Automatic deploys on merge to main
  • pytest test suite for backend validation
  • Environment variables managed per-service
Vercel Fly.io GitHub Actions pytest