Testing & CI/CD

Testing Methodology & CI/CD Architecture

A multi-layered testing strategy for the six-stage agent development pipeline — Discover, Design, Build, Evaluate, Repair, Track. Backend on Fly.io, frontend on Vercel, database on Supabase.

847

Passing tests

backend · pytest

Failing tests

pre-existing

Skipped tests

conditional skips

6.6s

Test runtime

847 tests

CI/CD pipelines

no GitHub Actions

Testing pyramid

E2E

End-to-End

9 spec files · Playwright · ~3 min

Target: Vercel preview deployments via a PR bot. Auth: session-based (E2E_AUTH=true). Coverage: full user journeys, agent CRUD, eval wizard. Specs evaluations.spec.ts · agents.spec.ts · auth-flows.spec.ts · dashboard.spec.ts · repair.spec.ts · hydra.spec.ts · evaluation-execution.spec.ts · evaluation-wizard.spec.ts · user-journey.spec.ts

Integration

API Contract + Service Integration

target ~40 tests · pytest · ~2s

API contract tests verify frontend assumptions against the backend schema. Supabase integration uses real DB tests with seeded data. Auth flow covers JWT validation, RLS policies, role-based access. not yet builtall backend tests mock Supabase Tool: httpx.AsyncClient + pytest-asyncio. Gate: must pass before E2E runs.

Unit

Backend Unit + Frontend Component

847+ tests · pytest -v · ~6.6s

Backend (59 files): routes, agents, scoring, connectors, build pipeline, auth, middleware, org isolation, versioning, analytics. Key fixtures: mock_supabase, mock_anthropic, authenticated_client, valid_jwt_token. frontend severely under-tested4 component tests vs a full Next.js app Frontend (4 files): ErrorBoundary.test.tsx, LoadingSpinner.test.tsx, EmptyState.test.tsx, api.test.ts.

Target CI/CD pipeline

Push / PR open

Every change enters here.

Pre-commit

Lint, type-check, format — before anything reaches the pipeline.

Branch gate

mainGitHub Actions triggered.featureGitHub Actions triggered.

Stage 1 — parallel

backend unitpytest · 847+ tests · ~7sfrontend unitJest · ~50 tests (target) · ~5s

Gate — all pass?

yesProceed to integration tests.noFail fast — notify, block merge.

Integration

API contracts + Supabase seed · ~8s.

Gate — pass?

yesProceed to preview deploy.noBlock merge.

Preview deploy

Vercel (frontend) + Fly.io (backend).

E2E

Playwright against the preview URL · ~3 min.

Gate — pass?

yesProceed to coverage report.noAlert + block merge; Playwright HTML report attached.

Coverage

pytest-cov + Jest coverage report.

Gate — thresholds met?

backend ≥80% · frontend ≥70%Targets.yesDeploy gate open — merge allowed, auto-deploy to production.noWarning + allow merge — coverage regression noted.

Deploy

Production deploy to Fly.io + Vercel.

Health check

GET /, GET /health, smoke test.

Done

Slack notification.

Gap analysis & remediation

#	Gap	Current state	Target state	Severity	Effort
1	No CI/CD pipeline	Tests run manually via CLI. No automation, no PR gating, no alerting.	GitHub Actions workflow with parallel stages, preview deploys, and Slack notifications.	Critical	1 day — scaffold workflow YAML
2	No PR / branch protection	Code can merge without any test gate. E2E targets production only.	Branch protection requiring passing CI. E2E runs against the Vercel preview deploy URL.	Critical	2 hr — GitHub branch rules
3	No JSONL logging test harness	Build standard requires structured JSONL logging + a fixture-based test harness; not implemented.	`conftest.py` fixture that captures structured logs and asserts log shape per request.	Critical	2 days — log capture fixture + assertions
4	Frontend unit tests anemic	4 component tests vs a full Next.js app with complex Discover→Design→Build→Evaluate flows.	Jest tests for all pages, forms, hooks, and state. Target ≥50 test files.	High	1–2 weeks — tests per page/component
5	No Supabase integration tests	Every backend test mocks `supabase`. RLS, migrations, and DB constraints untested.	Dedicated test Supabase project, seed scripts, integration tests with `httpx.AsyncClient`.	High	2 days — test DB + seed + suite
6	No coverage thresholds	Unknown test coverage. No `pytest-cov` or Jest coverage config.	`pytest-cov` ≥80% branch. Jest ≥70%. Enforced in CI.	High	3 hr — config + CI enforcement
7	No pre-commit hooks	Lint and type-check not enforced. No `.pre-commit-config.yaml`.	Hooks: ruff format, ruff check, mypy, eslint, prettier.	Medium	2 hr — config + install
8	E2E only targets production	Playwright connects to `aesop-studio.vercel.app`. No preview URL testing.	CI passes `PLAYWRIGHT_BASE_URL` from the Vercel preview deploy. Tests run before merge.	Medium	3 hr — CI env var + deploy hook
9	No failure alerting	5 pre-existing failures sit unfixed with no notification.	Slack webhook on CI failure. Auto-created GitHub issue for persistent failures.	Medium	1 hr — Slack webhook + CI step
10	No API contract tests	Frontend API assumptions against backend schema unverified. Type drift possible.	OpenAPI schema snapshot tests. CI fails if the backend schema changes without a frontend update.	Medium	1 day — schema export + snapshot
11	Single Python version	Only tested against Python 3.14. No version matrix.	GitHub Actions matrix: Python 3.12, 3.13, 3.14.	Low	1 hr — matrix config
12	No visual regression	Playwright can do visual snapshots; not configured.	`toHaveScreenshot()` on critical pages, stored as CI artifacts.	Low	4 hr — snapshot config + baselines

12 gaps identified · 3 Critical · 3 High · 4 Medium · 2 Low

Current vs target state

Current state

No CI/CD — manual test runs only
No pre-commit hooks
No coverage tracking
No PR gating / branch protection
No integration tests (all Supabase mocked)
No contract tests (frontend ↔ backend)
No failure alerting
No JSONL logging test harness
847 backend pytest tests · 6.6s
9 Playwright E2E specs
4 frontend component tests (anemic)
E2E targets production only

Target state

GitHub Actions CI/CD with parallel stages
Pre-commit hooks: ruff + mypy + eslint
pytest-cov ≥80% · Jest ≥70% · enforced in CI
Branch protection — CI must pass before merge
Supabase test project + seeded integration suite
OpenAPI schema snapshot + contract tests
Slack notifications on failure
Structured JSONL log capture fixture
847+ backend tests (maintained + growing)
9 E2E specs vs. preview deploys
50+ frontend unit tests (pages, hooks, forms)
Visual regression snapshots on critical paths

The pyramid exists; the automation around it does not. A strong 847-test backend and nine Playwright specs are running by hand. The entire value — PR gating, preview-deploy E2E, coverage enforcement, failure alerting — sits behind a single missing GitHub Actions workflow. Close the twelve gaps and the pyramid starts enforcing itself.