Observability and Backpressure

Sanmopia uses an OSS observability plane so traffic spikes are measured before operators see stale pages, duplicate payments, failed exports, or silent worker lag.

Runtime Plane

flowchart LR
  Browser[Mother, caregiver, BranchOwner, HeadOfficeManager portals]
  Traefik[Traefik Docker provider]
  Backend[FastAPI backend]
  Collector[OpenTelemetry Collector]
  Supabase[Self-hosted Supabase Postgres, Storage, Queues]
  Restate[Restate workflow state]
  Provider[Kill Bill, Novu, external providers]
  CollectorLog[Collector stdout]
  Future[Prometheus, Grafana, Tempo, Loki]

  Browser --> Traefik
  Traefik -->|rateLimit, inFlightReq| Backend
  Backend --> Supabase
  Backend --> Restate
  Backend --> Provider
  Backend -->|OTLP gRPC traces, metrics, logs| Collector
  Collector -->|debug exporter in dev| CollectorLog
  Collector -. later exporters .-> Future

Current stage runs otel-collector on the internal Compose network only. It has OTLP gRPC/HTTP receivers, internal health, internal Prometheus-format collector metrics, memory limiter, batch processor, and debug exporter. It is not routed through Traefik and does not publish host ports.

Backend FastAPI instrumentation is enabled by env:

OTEL_SERVICE_NAME
OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_RESOURCE_ATTRIBUTES

The backend composition root wires OpenTelemetry at the platform edge. Domain and application modules must not import OpenTelemetry packages.

Signal Rules

Traces: HTTP request spans, workflow starts, payment/webhook/provider calls, document render jobs, settlement export jobs, queue claim/complete/retry.
Metrics: Traefik 429, backend 503, queue depth, oldest queued age, in-flight workers, retry count, dead-letter count, workflow blocked count.
Logs: operational audit and incident correlation only. Operator-facing business history stays in domain journals and read models.

Spike Policy

Traefik protects public HTTP routers with env-driven rateLimit and inFlightReq labels. Edge retry is not enabled on mutation routes because payment, webhook, settlement, and document commands need idempotency-aware backend handling.

Supabase Queues / pgmq is the first durable queue because self-hosted Supabase already owns the Postgres control plane. Restate owns long workflows where replay crosses multiple domain commands or external providers. Backend queue workers must use the shared Supabase queue dispatcher and pgmq adapter contract for read, archive, visibility-timeout retry, and dead-letter handoff. Feature slices provide processors; they do not implement separate queue acknowledgement rules.

The family payment-delegation relationship outbox uses sanmopia_spicedb_relationships. Supabase Cron invokes the internal bounded dispatcher every minute through pg_net; the backend claims at most the configured batch, applies SpiceDB touch or delete, then archives, retries, or dead-letters through the shared dispatcher. pnpm supabase:cron:smoke verifies both the privacy trigger and this queue trigger return HTTP 202. The authenticated family-delegation E2E additionally proves permission convergence and queue cleanup instead of treating Cron registration as runtime proof.

Queue And Load Notes

Queue workloads are domain/application contracts, not ad hoc background route helpers. Each workload declares:

queue name and dead-letter queue name
owning bounded context and feature slice
idempotency key and command fingerprint
visibility timeout, retry limit, and repair command
maximum worker concurrency and backpressure response
trace attributes that link HTTP request, workflow request, queue message, and provider call

Restate owns long-running reservation, payment/refund, settlement export, document render, and privacy workflows when replay spans multiple domain commands or external providers. Supabase Queues / pgmq owns fanout and worker dispatch where a message can be retried or repaired independently. A workflow can enqueue delayed render, notification, or export work, but the queue message must carry the workflow request id and idempotency key so duplicate delivery does not create duplicate payment, settlement, or document facts.

Required OpenTelemetry labels:

actor_kind, actor_id, and branch_profile_id when available
reservation_id, settlement_statement_id, workflow_request_id, and queue_message_id when the command owns those facts
idempotency_key, processor_name, provider_name, and provider_action for payment, refund, notification, document, and export calls
retry_count, dead_letter_reason, and blocked_reason for worker and workflow failures

Operational alerts must watch queue depth, oldest queued age, in-flight worker count, retry rate, dead-letter rate, workflow-blocked count, payment-provider latency, webhook lag, settlement-export lag, and signed-artifact handoff failure. Traffic spike tests should prefer targeted load on mutation start routes, workflow progress reads, queue claiming, and export/download handoffs; full test-suite or browser-wide load runs are not the default migration check.

Mutation routes should return an explicit backend overload response, such as 503 with Retry-After, only before a command is accepted. After acceptance, progress is read from Supabase-backed workflow or queue projections. Traefik rate limiting and inFlightReq protect the edge, but provider webhooks, payment/refund commands, settlement approvals, document renders, and repair commands rely on backend idempotency instead of edge retry.

Prometheus, Grafana, Tempo, and Loki remain the next runtime slice. The current collector config keeps exporter wiring centralized so those backends can be added without rewriting backend instrumentation.

Frontend Rule

No browser RUM endpoint is enabled yet. Browser telemetry requires separate CSP, privacy, sampling, and PII filtering review. Astro frontend work must route through a dedicated frontend task, not backend migration patches.