08 · Operations

Run the system

Day-2 reality. Who owns what, what’s deployed where, and where to look when something breaks at 02:00.

#hv-alerts

CloudWatch → SNS → Slack

Every failure in the Maestro/Hermes estate relays to #hv-alerts + email. AWS alarms (DLQ, ingest-age, consumer errors) and n8n Sentinel V2 both feed it. Live since 2026-06-11.

Hub v2

tasks-v2 review queue

hubv2.apphappenventures.com — reviewers act on the DynamoDB queue with the 6 actions (Approve&Send, Edit, Reject, Snooze, Comment, Escalate). A free-text reason on Approve&Send / Snooze / Reject is optional / non-blocking (DEV-471); reject & escalate still validate a reason code.

ECS Fargate

us-east-2 services

Cluster hv-hub-v2-development: services hv-hub-v2-api + hv-hub-v2-web, plus maestro (desired_count 0 in dev). Deploys wait for a healthy rollout before going green.

HV Hub v2 AWS architecture (us-east-2)
Animated · interactive — the AWS estate (us-east-2, account 891377221201): Route 53 → ALB → ECS Fargate (web/api/maestro), the Maestro producer→SQS→consumer Lambdas, Bedrock, DynamoDB + S3, and the CloudWatch → SNS → Slack observability path. Open full ↗
CI/CD

GitHub Actions on AWS CodeBuild runners

hv-hub-v2 CI runs on AWS CodeBuild self-hosted GitHub Actions runners (no GitHub minutes). One ephemeral build per queued job; app deploys go to ECS Fargate, infra via Terragrunt CD. All in us-east-2.

CodeBuild runner
Runner
CodeBuild project hv-hub-v2-development-gha-runner, image standard:7.0 (Ubuntu), privileged for buildx. Webhook on WORKFLOW_JOB_QUEUED.
module ci_runner
test.yml
PR gate
On every PR + push to main: JS build + vitest, Python pytest (6-module matrix: task_consumer / maestro_sender / governor_resurfacer / outbound_sender / acp_relay / acp_schedule_ticker), turbo lint. Tests mock everything — no secrets.
9 required checks
Branch protection
main
9 required status checks, enforce_admins=true, 1 review, merge queue enabled. No merge to main with any red check. Push restricted to danielr-art.
gh api .../protection
deploy-api / deploy-web
ECS
Build amd64 image → push ECR → ecs update-service → wait services-stable. Green = serving, not just triggered.
hv-hub-v2-{api,web}
deploy-maestro
ECS
Maestro orchestrator image → ECS service hv-hub-v2-maestro (desired_count 0 in dev). Pairs with Terragrunt task_maestro.
Bedrock orchestrator
deploy-mcp-airtable
AgentCore
arm64 image (buildx + QEMU) → update-agent-runtime → poll until READY (fail-loud on *_FAILED/timeout). Hosts the governance MCP tools.
read-modify-write
Terragrunt CD
Infra
Push to main on infra/** → run-all apply to dev. ci_runner excluded from run-all (bootstrapped out-of-band). Parallelism 4 + retry to dodge registry 423s.
infra/live/development
OIDC, not keys
Auth
Jobs that touch AWS assume role hv-hub-v2-github-actions-cd-development via OIDC (works in self-hosted). The runner's own role stays minimal (logs only).
configure-aws-credentials
Deploy-on-PR
UAT
deploy-* also fire on pull_request (intentional preview): each PR auto-deploys to the shared dev/UAT — last write wins. Merge to main stays gated.
last-write-wins
Safety controls

Operational guardrails · live pipeline

🛑
SSOT kill-switch
The consumer skips a playbook by its SSOT Status (Paused / Deprecated / Cancelled / Back to Drawing Board…). Now Status-only — decoupled from the old Enabled field (#268).
🚦
Contact Governor (GOV-01)
Central ALLOW / DEFER / BLOCK / DROP gate in the consumer: touch log + frequency caps + families. GOVERNOR_ENABLED=true in dev, fail-closed by default.
✋
Human-in-the-loop
Nothing sends until a reviewer approves in the Hub tasks-v2 queue. A free-text reason on Approve&Send / Snooze / Reject is optional / non-blocking (DEV-471); reject & escalate still validate a reason code, and escalation history is shown.
🚀
Send mode · live
The real sender — the message-dispatcher Lambda (Unified Sender) — runs with per-channel Governor flags. As last verified 2026-08-07 the email adapter’s sendDevelopmentMode=false, so email reaches the real client; setting sendDevelopmentMode=true in /admin/governor would redirect every email to the configured test addresses instead. The legacy maestro_sender Lambda is inert (SEND_DRY_RUN=true and its EventBridge tick disabled).
🔒
Suppression fail-closed
If Suppression_List is unavailable (no fresh data, no L2 cache) the consumer raises → DLQ + alarm instead of drafting un-gated (#228).
⚠️
Dual-worker hazard
A legacy n8n “AWS Send Worker (Maestro) — DRY-RUN” still polls the same approved queue as the live message-dispatcher. The dispatcher delivers for real (email is live to clients — Governor sendDevelopmentMode=false); the n8n worker is in dry-run and today just corrupts task status, marking approved tasks status=failed while the dispatcher delivers. Retire the n8n worker to avoid a double delivery to the client.
Owners

Who to ping

AreaOwnerBackupEscalation
CI/CD · CodeBuild runnersDaniel—Jess (business impact)
ECS Fargate · Hub v2DanielDaríoJess
Terragrunt / infra (us-east-2)Daniel—Jess
Maestro pipeline · consumer / senderFaridDanielJess
Contact Governor (GOV-01)FaridDanielJess
n8n workflows + Sentinel V2FaridDanielJess
Airtable SSOTFaridJessDaniel
Hub PR reviewDarío—Daniel
Runbook

Where to look when it breaks at 02:00

First stop is always #hv-alerts — both feeds (AWS alarms + n8n Sentinel V2) land there. From the alarm name, jump to the right surface below.

live DLQ not empty

tasks-dlq-not-empty

A generation batch died after all retries. Check the task-consumer Lambda logs in CloudWatch (us-east-2), fix the cause, then redrive the DLQ. Alarm hv-hub-v2-development-tasks-dlq-not-empty.

live Ingest stuck

tasks-ingest-age > 30 min

SQS oldest-message age is climbing — consumer is stuck or disabled. Verify the Lambda isn’t throttled/disabled and the ingest queue is draining.

live Deploy didn’t stick

ECS rollout

“Deploy success” means the image was pushed and services-stable returned. If new code isn’t serving, check the ECS service events for hv-hub-v2-{api,web} in cluster hv-hub-v2-development.

gotcha New module 403

RBAC registration

A new code module returns a silent 403 for everyone until it’s registered in Airtable Hub Modules + granted via /admin/rbac. Run POST /api/admin/rbac/modules/sync after deploy.

Rollback levers: stop a send instantly with SEND_DRY_RUN=true on the message-dispatcher Lambda, set the Governor sendDevelopmentMode=true (email back to test inboxes) or globalSendEnabled OFF (all channels dry-run), or turn off a per-channel send flag in the Governor config. Toggle reads back to Airtable via SALES_QUEUE_FROM_DYNAMO=false / ROBIN_QUEUE_FROM_DYNAMO=false. Pause any playbook by setting its SSOT Status to a stop state. Owner on call: Daniel for infra/CI, Farid for pipeline.