← Index

AESOP Studio

Testing Methodology & CI/CD Architecture

Multi-layered testing strategy for the six-stage agent development pipeline (DiscoverDesignBuildEvaluateRepairTrack). Backend on Fly.io, frontend on Vercel, database on Supabase.

1 — Current Test Suite at a Glance
847
Passing Tests
Backend · pytest
5
Failing Tests
Pre-existing
20
Skipped Tests
Conditional skips
6.6s
Test Runtime
847 tests
0
CI/CD Pipelines
No GitHub Actions
2 — Testing Pyramid
E2E
End-to-End
9 spec files · Playwright · ~3 min
Target: Vercel preview deployments via PR bot
Auth: Session-based (E2E_AUTH=true)
Coverage: Full user journeys, agent CRUD, eval wizard
Specs: evaluations.spec.ts, agents.spec.ts, auth-flows.spec.ts, dashboard.spec.ts, repair.spec.ts, hydra.spec.ts, evaluation-execution.spec.ts, evaluation-wizard.spec.ts, user-journey.spec.ts
Integration
API Contract + Service Integration
Target: ~40 tests · pytest · ~2s
API Contract Tests: Verify frontend API assumptions against backend schema
Supabase Integration: Real DB tests with seeded data
Auth Flow: JWT validation, RLS policies, role-based access
Status: Not yet built — all backend tests mock Supabase
Tool: httpx.AsyncClient + pytest-asyncio
Gate: Must pass before E2E runs
Unit
Backend Unit Tests + Frontend Component Tests
847+ tests · pytest -v · ~6.6s
Backend (59 files): Routes, agents, scoring, connectors, build pipeline, auth, middleware, org isolation, versioning, analytics
Key fixtures: mock_supabase, mock_anthropic, authenticated_client, valid_jwt_token
Frontend (4 files): ErrorBoundary.test.tsx, LoadingSpinner.test.tsx, EmptyState.test.tsx, api.test.ts
Status: Severely under-tested — 4 components vs. full Next.js app with complex multi-stage flows
3 — Target CI/CD Pipeline
        graph TD
          A["Push / PR Open"] --> B["Pre-commit Hooks
lint · type-check · format"] B --> C{"Branch?"} C -->|main| D["GitHub Actions Triggered"] C -->|feature| E["GitHub Actions Triggered"] D --> F{"Parallel Stage 1"} E --> F F --> G["Backend Unit
pytest · 847+ tests · ~7s"] F --> H["Frontend Unit
Jest · ~50 tests (target) · ~5s"] G --> I{"Parallel: all pass?"} H --> I I -->|Yes| J["Integration Tests
API contracts + Supabase seed · ~8s"] I -->|No| K["Fail Fast — notify, block merge"] J --> L{"Pass?"} L -->|Yes| M["Preview Deploy
Vercel (frontend) + Fly.io (backend)"] L -->|No| K M --> N["E2E Tests
Playwright vs preview URL · ~3 min"] N --> O{"Pass?"} O -->|Yes| P["Coverage Report
pytest-cov + Jest coverage"] O -->|No| Q["Alert + Block Merge
Playwright HTML report attached"] P --> R{"Thresholds Met?
Backend ≥80% · Frontend ≥70%"} R -->|Yes| S["Deploy Gate Open
Merge allowed · auto-deploy to production"] R -->|No| T["Warning + Allow Merge
Coverage regression noted"] S --> U["Production Deploy
Fly.io + Vercel"] U --> V["Health Check
GET / · GET /health · smoke test"] V --> W["Done — Slack Notification"]
Pass / Merge Allowed
Fail / Block Merge
Warning / Conditional
Automated Step
Decision Gate
4 — Gap Analysis & Remediation Plan
# Gap Current State Target State Severity Effort
1 No CI/CD Pipeline Tests run manually via CLI. No automation, no PR gating, no alerting. GitHub Actions workflow with parallel stages, preview deploys, and Slack notifications. Critical 1 day — scaffold workflow YAML
2 No PR/Branch Protection Code can merge without any test gate. E2E targets production only. Branch protection rules requiring passing CI. E2E runs against Vercel preview deploy URL. Critical 2 hr — GitHub branch rules
3 No JSONL Logging Test Harness Build standard requires structured JSONL logging + fixture-based test harness; not implemented. conftest.py fixture that captures structured logs, asserts log shape per request. Critical 2 days — log capture fixture + assertions
4 Frontend Unit Tests Anemic 4 component tests vs. full Next.js app with complex Discover→Design→Build→Evaluate flows. Jest tests for all pages, forms, hooks, and state management. Target: ≥50 test files. High 1-2 weeks — write tests per page/component
5 No Supabase Integration Tests Every backend test mocks supabase. RLS, migrations, and DB constraints untested. Dedicated test Supabase project. Seed scripts. Integration tests with httpx.AsyncClient. High 2 days — test DB + seed + suite
6 No Coverage Thresholds Unknown test coverage. No pytest-cov or Jest coverage config. pytest-cov with ≥80% branch threshold. Jest coverage with ≥70% threshold. CI enforcement. High 3 hr — config + CI enforcement
7 No Pre-commit Hooks Lint/type-check not enforced. No .pre-commit-config.yaml. pre-commit hooks: ruff format, ruff check, mypy, eslint, prettier. Medium 2 hr — config + install
8 E2E Only Targets Production Playwright connects to aesop-studio.vercel.app. No preview URL testing. CI passes PLAYWRIGHT_BASE_URL from Vercel preview deploy. Tests run before merge. Medium 3 hr — CI env var + deploy hook
9 No Failure Alerting 5 pre-existing failures sit unfixed with no notification. Slack webhook on CI failure. GitHub issue auto-created for persistent failures. Medium 1 hr — Slack webhook + CI step
10 No API Contract Tests Frontend API assumptions against backend schema unverified. Type drift possible. OpenAPI schema snapshot tests. CI fails if backend schema changes without frontend update. Medium 1 day — schema export + snapshot
11 Single Python Version Only tests against Python 3.14. No version matrix. GitHub Actions matrix: Python 3.12, 3.13, 3.14. Low 1 hr — matrix config
12 No Visual Regression Playwright can do visual snapshots; not configured. toHaveScreenshot() on critical pages. Stored as CI artifacts. Low 4 hr — snapshot config + baselines
12 gaps identified · 3 Critical · 3 High · 4 Medium · 2 Low
5 — Current vs Target State
Current State
Target State
  • No CI/CD — manual test runs only
  • No pre-commit hooks
  • No coverage tracking
  • No PR gating / branch protection
  • No integration tests (all Supabase mocked)
  • No contract tests (frontend↔backend)
  • No failure alerting
  • No JSONL logging test harness
  • 847 backend pytest tests · 6.6s
  • 9 Playwright E2E specs
  • 4 frontend component tests (anemic)
  • E2E targets production only
  • GitHub Actions CI/CD with parallel stages
  • Pre-commit hooks: ruff + mypy + eslint
  • pytest-cov ≥80% · Jest ≥70% · enforced in CI
  • Branch protection — CI must pass before merge
  • Supabase test project + seeded integration suite
  • OpenAPI schema snapshot + contract tests
  • Slack notifications on failure
  • Structured JSONL log capture fixture
  • 847+ backend tests (maintained + growing)
  • 9 E2E specs vs. preview deploys
  • 50+ frontend unit tests (pages, hooks, forms)
  • Visual regression snapshots on critical paths