AESOP Studio
Testing Methodology & CI/CD Architecture
Multi-layered testing strategy for the six-stage agent development pipeline
(Discover → Design → Build → Evaluate → Repair → Track).
Backend on Fly.io, frontend on Vercel, database on Supabase.
1 — Current Test Suite at a Glance
847
Passing Tests
Backend · pytest
5
Failing Tests
Pre-existing
20
Skipped Tests
Conditional skips
6.6s
Test Runtime
847 tests
0
CI/CD Pipelines
No GitHub Actions
2 — Testing Pyramid
E2E
End-to-End
9 spec files · Playwright · ~3 min
Target: Vercel preview deployments via PR bot
Auth: Session-based (
Coverage: Full user journeys, agent CRUD, eval wizard
Auth: Session-based (
E2E_AUTH=true)Coverage: Full user journeys, agent CRUD, eval wizard
Specs:
evaluations.spec.ts, agents.spec.ts, auth-flows.spec.ts, dashboard.spec.ts, repair.spec.ts, hydra.spec.ts, evaluation-execution.spec.ts, evaluation-wizard.spec.ts, user-journey.spec.ts
Integration
API Contract + Service Integration
Target: ~40 tests · pytest · ~2s
API Contract Tests: Verify frontend API assumptions against backend schema
Supabase Integration: Real DB tests with seeded data
Auth Flow: JWT validation, RLS policies, role-based access
Supabase Integration: Real DB tests with seeded data
Auth Flow: JWT validation, RLS policies, role-based access
Status: Not yet built — all backend tests mock Supabase
Tool:
Gate: Must pass before E2E runs
Tool:
httpx.AsyncClient + pytest-asyncioGate: Must pass before E2E runs
Unit
Backend Unit Tests + Frontend Component Tests
847+ tests ·
pytest -v · ~6.6s
Backend (59 files): Routes, agents, scoring, connectors, build pipeline, auth, middleware, org isolation, versioning, analytics
Key fixtures:
Key fixtures:
mock_supabase, mock_anthropic, authenticated_client, valid_jwt_token
Frontend (4 files):
Status: Severely under-tested — 4 components vs. full Next.js app with complex multi-stage flows
ErrorBoundary.test.tsx, LoadingSpinner.test.tsx, EmptyState.test.tsx, api.test.tsStatus: Severely under-tested — 4 components vs. full Next.js app with complex multi-stage flows
3 — Target CI/CD Pipeline
graph TD
A["Push / PR Open"] --> B["Pre-commit Hooks
lint · type-check · format"]
B --> C{"Branch?"}
C -->|main| D["GitHub Actions Triggered"]
C -->|feature| E["GitHub Actions Triggered"]
D --> F{"Parallel Stage 1"}
E --> F
F --> G["Backend Unit
pytest · 847+ tests · ~7s"]
F --> H["Frontend Unit
Jest · ~50 tests (target) · ~5s"]
G --> I{"Parallel: all pass?"}
H --> I
I -->|Yes| J["Integration Tests
API contracts + Supabase seed · ~8s"]
I -->|No| K["Fail Fast — notify, block merge"]
J --> L{"Pass?"}
L -->|Yes| M["Preview Deploy
Vercel (frontend) + Fly.io (backend)"]
L -->|No| K
M --> N["E2E Tests
Playwright vs preview URL · ~3 min"]
N --> O{"Pass?"}
O -->|Yes| P["Coverage Report
pytest-cov + Jest coverage"]
O -->|No| Q["Alert + Block Merge
Playwright HTML report attached"]
P --> R{"Thresholds Met?
Backend ≥80% · Frontend ≥70%"}
R -->|Yes| S["Deploy Gate Open
Merge allowed · auto-deploy to production"]
R -->|No| T["Warning + Allow Merge
Coverage regression noted"]
S --> U["Production Deploy
Fly.io + Vercel"]
U --> V["Health Check
GET / · GET /health · smoke test"]
V --> W["Done — Slack Notification"]
Pass / Merge Allowed
Fail / Block Merge
Warning / Conditional
Automated Step
Decision Gate
4 — Gap Analysis & Remediation Plan
| # | Gap | Current State | Target State | Severity | Effort |
|---|---|---|---|---|---|
| 1 | No CI/CD Pipeline | Tests run manually via CLI. No automation, no PR gating, no alerting. | GitHub Actions workflow with parallel stages, preview deploys, and Slack notifications. | Critical | 1 day — scaffold workflow YAML |
| 2 | No PR/Branch Protection | Code can merge without any test gate. E2E targets production only. | Branch protection rules requiring passing CI. E2E runs against Vercel preview deploy URL. | Critical | 2 hr — GitHub branch rules |
| 3 | No JSONL Logging Test Harness | Build standard requires structured JSONL logging + fixture-based test harness; not implemented. | conftest.py fixture that captures structured logs, asserts log shape per request. |
Critical | 2 days — log capture fixture + assertions |
| 4 | Frontend Unit Tests Anemic | 4 component tests vs. full Next.js app with complex Discover→Design→Build→Evaluate flows. | Jest tests for all pages, forms, hooks, and state management. Target: ≥50 test files. | High | 1-2 weeks — write tests per page/component |
| 5 | No Supabase Integration Tests | Every backend test mocks supabase. RLS, migrations, and DB constraints untested. |
Dedicated test Supabase project. Seed scripts. Integration tests with httpx.AsyncClient. |
High | 2 days — test DB + seed + suite |
| 6 | No Coverage Thresholds | Unknown test coverage. No pytest-cov or Jest coverage config. |
pytest-cov with ≥80% branch threshold. Jest coverage with ≥70% threshold. CI enforcement. |
High | 3 hr — config + CI enforcement |
| 7 | No Pre-commit Hooks | Lint/type-check not enforced. No .pre-commit-config.yaml. |
pre-commit hooks: ruff format, ruff check, mypy, eslint, prettier. |
Medium | 2 hr — config + install |
| 8 | E2E Only Targets Production | Playwright connects to aesop-studio.vercel.app. No preview URL testing. |
CI passes PLAYWRIGHT_BASE_URL from Vercel preview deploy. Tests run before merge. |
Medium | 3 hr — CI env var + deploy hook |
| 9 | No Failure Alerting | 5 pre-existing failures sit unfixed with no notification. | Slack webhook on CI failure. GitHub issue auto-created for persistent failures. | Medium | 1 hr — Slack webhook + CI step |
| 10 | No API Contract Tests | Frontend API assumptions against backend schema unverified. Type drift possible. | OpenAPI schema snapshot tests. CI fails if backend schema changes without frontend update. | Medium | 1 day — schema export + snapshot |
| 11 | Single Python Version | Only tests against Python 3.14. No version matrix. | GitHub Actions matrix: Python 3.12, 3.13, 3.14. | Low | 1 hr — matrix config |
| 12 | No Visual Regression | Playwright can do visual snapshots; not configured. | toHaveScreenshot() on critical pages. Stored as CI artifacts. |
Low | 4 hr — snapshot config + baselines |
| 12 gaps identified · 3 Critical · 3 High · 4 Medium · 2 Low | |||||
5 — Current vs Target State
Current State
Target State
- No CI/CD — manual test runs only
- No pre-commit hooks
- No coverage tracking
- No PR gating / branch protection
- No integration tests (all Supabase mocked)
- No contract tests (frontend↔backend)
- No failure alerting
- No JSONL logging test harness
- 847 backend pytest tests · 6.6s
- 9 Playwright E2E specs
- 4 frontend component tests (anemic)
- E2E targets production only
- GitHub Actions CI/CD with parallel stages
- Pre-commit hooks: ruff + mypy + eslint
- pytest-cov ≥80% · Jest ≥70% · enforced in CI
- Branch protection — CI must pass before merge
- Supabase test project + seeded integration suite
- OpenAPI schema snapshot + contract tests
- Slack notifications on failure
- Structured JSONL log capture fixture
- 847+ backend tests (maintained + growing)
- 9 E2E specs vs. preview deploys
- 50+ frontend unit tests (pages, hooks, forms)
- Visual regression snapshots on critical paths