Traffic Spike Backpressure Gap Mining

Traffic Spike Backpressure Gap Mining

Scope: migration plan for traffic spikes that must not take down booking, payment, document, settlement, notification, matching, or webhook paths. This is an operator/backend decision page only; active payment, settlement, and branch-resource workers stay untouched.

Current foundation: Compose now applies Traefik rateLimit and inFlightReq middleware to public backend, frontend, and Supabase routers; self-hosted Supabase already creates pgmq; backend has a Supabase pgmq adapter; stage env has queue/worker/backpressure SSOT; backend now centralizes workload queue names, priorities, visibility timeout, max attempts, and dead-letter names in application.platform.supabase_queue_contract; backend worker dispatch now uses application.platform.supabase_queue_worker plus the stdlib SupabasePgmqQueueAdapter to read, archive, retry by visibility timeout, and dead-letter messages; Supabase CLI migrations create every queue/dead-letter queue and service-role-only pgmq_read/pgmq_archive RPC wrappers; and otel-collector receives backend OTLP telemetry on the internal network.

OSS Stack Decision

Layer	OSS choice	Use	Migration rule
Edge load balancing	Traefik	Host/path routing, service load balancing, HTTP rateLimit, inFlightReq, retry, and circuitBreaker middlewares where route-safe	Configure through Docker labels and env SSOT. No host ports. Return 429 or stable degraded response before backend pool exhaustion.
Telemetry collector	OpenTelemetry Collector	Vendor-neutral OTLP traces, metrics, logs, collector internal metrics, later exporter fanout	Run internally only. Backend emits OTLP gRPC. Prometheus/Grafana/Tempo/Loki remain exporter slice, not feature code dependencies.
Durable orchestration	Restate	Long-running payment, document, settlement, webhook, and batch workflows with persisted progress, retry, and replay	Restate owns workflow progress. Supabase remains business ledger, read model, RLS, and audit store.
Durable queue	Supabase Queues / pgmq	Postgres-native work queues for idempotent background jobs and burst smoothing	Default queue layer because self-hosted Supabase already owns Postgres. Enable pgmq only through migration/config proof and expose pgmq_public intentionally if Data API access is required.
Cache/rate counters	Valkey or Redis	Hot counters, short TTL locks, and request-shaping state when Postgres queue reads become too hot	Justified only if shared counters exceed Postgres-friendly write rate. Existing Novu Redis is Novu-owned; do not reuse it for Sanmopia domain state without a separate service/env contract.
Stream/broker	NATS or RabbitMQ	High fan-out event streams or competing consumers beyond pgmq visibility-timeout needs	Defer. Add only with measured need: cross-service event fanout, high-throughput notifications, or queue latency that pgmq cannot meet.

External baseline checked: Traefik rate limit, Traefik in-flight requests, Supabase Queues, and Restate key concepts. OpenTelemetry Collector architecture, configuration, and internal telemetry are tracked in Observability and Backpressure.

Backpressure Model

Flow	Spike risk	Ingress behavior	Queue/workflow	Worker cap	Idempotency key	Retry / dead letter	Priority
Payment pre-registration	Gateway timeout, duplicate click, payment id collision	Traefik rate limit per user/IP and low inFlightReq; backend rejects when payment queue lag is above threshold except status reads	Restate payment workflow plus Supabase payment ledger; pgmq only for follow-up reconciliation tasks	Small fixed pool per provider and per payment method	payment_id plus reservation id plus purpose	Provider 5xx/timeouts retry with exponential backoff; duplicate provider success is replay; nonretryable amount/status mismatch goes operator review queue	P0
Document generation	XLSX/PDF render CPU spike, bundle fanout, storage pressure	Accept request, return queued status, never render synchronously from operator click	pgmq render queue per artifact kind; Restate for bundle/build workflows needing multi-step recovery	Separate render workers by format; one large bundle per worker	render request code plus template version plus reservation/document snapshot hash	Retry renderer/storage transient failures; dead letter after capped attempts with repair command and stored failure reason	P1
Settlement exports	Month-end HQ/branch export burst, stale totals, long filesystem writes	Dashboard/export reads stay fast; export command freezes dataset then queues build	Restate export workflow using frozen Supabase snapshot; pgmq optional renderer queue	One export per branch plus small global cap for HQ bulk	export request id plus dataset revision plus format	Retry using frozen snapshot only; dead letter to settlement repair history, not recompute current totals	P1
Notification sends	Birth-info reminders, payment deadlines, campaign bursts	API creates dispatch request and returns accepted; no direct provider send on user path	Supabase task/queue rows hand off to Novu; Restate only for business workflows that wait on notification outcome	Channel-specific caps and provider-specific rate buckets	communication dispatch id plus business event id	Provider retry/backoff in Novu plus Sanmopia dispatch event audit; dead letter requires operator replay or suppression reason	P2
Matching batch	Branch-wide assignment optimizer CPU and provider travel-time lookup burst	User actions enqueue optimization; UI polls snapshot/projection	pgmq batch queue; Restate only if assignment workflow spans reservation mutation and notification	One optimizer per branch/profile segment; travel lookup cap per provider	optimization request id plus policy version plus reservation/candidate snapshot revision	Retry deterministic compute; external travel lookup failures become partial evidence with rerun action	P1
Webhook processing	Kill Bill duplicate/out-of-order callbacks, provider retry storm	Restricted internal Traefik route injects a callback-admission token; backend validates that token before enqueue. Kill Bill does not sign push callbacks.	Append webhook inbox row in Supabase, then Restate/payment workflow consumes ordered facts	Small ordered worker pool per provider/payment id	provider event identity plus payment id, transaction id, operation, and event type	Duplicate inbox rows replay to same result; missing/invalid admission token is rejected; unrecognized state moves to dead-letter review	P0

Queue Contract

Every queued command or workflow start must store:

• Stable idempotency key and request hash.
• Actor id, actor type, branch/HQ scope, and authorization decision reference.
• Business priority, queue name, visibility timeout, next attempt time, attempt count, max attempts, and retry class.
• Frozen input snapshot ids, not mutable source rows.
• Expected aggregate/read-model revision where command mutates business state.
• Result read-model ids, public status, blocked reason, and failure reason.
• Dead-letter row with replay policy: replay_same_input, repair_then_replay, suppress, or manual_reconcile.

Current backend workload queues:

Workload	Queue	Dead-letter	Priority	Worker env
SpiceDB relationship sync	sanmopia_spicedb_relationships	sanmopia_spicedb_relationships_dead	High	SANMOPIA_WORKER_AUTHORIZATION_SYNC_CONCURRENCY
Payment reconciliation	sanmopia_payment_reconciliation	sanmopia_payment_reconciliation_dead	Critical	SANMOPIA_WORKER_PAYMENT_CONCURRENCY
Kill Bill webhook processing	sanmopia_payment_webhook_processing	sanmopia_payment_webhook_processing_dead	Critical	SANMOPIA_WORKER_WEBHOOK_CONCURRENCY
Document rendering	sanmopia_document_rendering	sanmopia_document_rendering_dead	High	SANMOPIA_WORKER_DOCUMENT_RENDER_CONCURRENCY
Settlement export	sanmopia_settlement_export	sanmopia_settlement_export_dead	High	SANMOPIA_WORKER_SETTLEMENT_EXPORT_CONCURRENCY
Notification dispatch	sanmopia_notification_dispatch	sanmopia_notification_dispatch_dead	Normal	SANMOPIA_WORKER_NOTIFICATION_CONCURRENCY
Caregiver matching optimization	sanmopia_caregiver_matching_optimization	sanmopia_caregiver_matching_optimization_dead	High	SANMOPIA_WORKER_MATCHING_CONCURRENCY

Worker Rules

• Workers scale by queue and capability, not by one generic background worker.
• Default concurrency is explicit env SSOT per queue: SANMOPIA_WORKER_AUTHORIZATION_SYNC_CONCURRENCY, SANMOPIA_WORKER_PAYMENT_CONCURRENCY, SANMOPIA_WORKER_DOCUMENT_RENDER_CONCURRENCY, SANMOPIA_WORKER_SETTLEMENT_EXPORT_CONCURRENCY, SANMOPIA_WORKER_NOTIFICATION_CONCURRENCY, SANMOPIA_WORKER_MATCHING_CONCURRENCY, and SANMOPIA_WORKER_WEBHOOK_CONCURRENCY.
• Worker claims must use visibility timeout or equivalent lease. Long jobs heartbeat progress through Supabase ledger/workflow rows.
• Worker acknowledgement must archive successful source messages through pgmq; retryable failures must leave the source message leased for visibility-timeout retry; terminal or max-attempt failures must enqueue a dead-letter payload with replay policies before archiving the source message.
• Worker startup must fail closed when queue schema, env SSOT, Restate endpoint, or Supabase service role is missing.
• User-facing reads must prefer status projection over queue internals.

Traefik Label Pattern

Compose already routes through Traefik labels and container expose without host ports. Next safe config slice is label-only middleware on public routers:

• sanmopia-backend-rate-limit: per-source average/burst caps for /api.
• sanmopia-backend-inflight: concurrent request cap before Uvicorn pool starvation.
• sanmopia-backend-retry: only for idempotent GET status/read routes; do not retry payment mutation routes at the edge.
• sanmopia-backend-circuit-breaker: only where current Traefik image supports the OSS HTTP middleware and the fallback behavior is tested.

Do not add proprietary API gateways, cloud queue products, or host-port shortcuts. Docker label values must come from .env.example or generated env SSOT, not inline magic numbers.

Observability

Required dashboard/read-model fields:

• Queue depth by queue, priority, and oldest age.
• In-flight count and worker concurrency by queue.
• Retry count, dead-letter count, and replay count.
• Time to accept, time to first claim, time to complete, and p95/p99 latency.
• Traefik 429 rate, backend 503 shedding rate, provider timeout rate.
• Restate workflow status and blocked reason projected through backend APIs.
• Supabase ledger drift: queued item without workflow, workflow without ledger, and ready artifact without signed grant audit.
• Trace correlation id across Traefik request, backend command, pgmq message, Restate workflow id, external provider request id, and operator repair action.

Migration Checklist

• Add Supabase migration for pgmq availability. Current core access migration creates pgmq and sanmopia_spicedb_relationships; next migrations should add queue names per workload, not a second queue engine.
• Define queue names, priorities, visibility timeouts, max attempts, and dead-letter queues in backend contracts and Supabase CLI migration.
• Add backend queue worker dispatcher and Supabase pgmq read/archive adapter contract so workload processors do not reimplement ack/retry/dead-letter behavior.
• Bind SpiceDB relationship sync processor so branch membership and branch hierarchy relationship payloads are validated, written through SpiceDBAuthorizationPort, archived on success, and dead-lettered when the payload no longer matches the relationship tuple.
• Expose the internal service-token worker dispatch endpoint POST /internal/queue-workloads/{workloadKind}/dispatches so Cron/worker runners call one backend dispatcher instead of bespoke per-workload scripts.
• Add worker/env backpressure SSOT to .env.example.
• Add Traefik middleware labels for backend, frontend, and Supabase public routers with rateLimit and inFlightReq values enforced by pnpm stage:check.
• Add internal OpenTelemetry Collector and backend OTLP env wiring.
• Bind Kill Bill webhook processing to the shared Supabase queue worker so callbacks pass internal admission-token validation, atomically register inbox plus PGMQ handoff, and return bounded 202 outcomes: queued, duplicate, or quarantined. Worker validates/dead-letters corrupt payloads before starting payment workflow ingestion; duplicate/conflicting deliveries create no second queue message.
• Split payment/webhook mutation routes from idempotent status/read routes before enabling edge retry.
• Bind remaining workload-specific processors for payment reconciliation, document rendering, settlement export, notification dispatch, and caregiver matching optimization.
• Add Prometheus/Grafana/Tempo/Loki exporters after collector smoke and retention policy are decided.
• Add operator queue dashboard and repair/replay commands before production traffic spike testing.
• Run spike smoke: payment duplicate clicks, document bulk render, settlement month-end export, notification burst, matching batch, and webhook replay storm.