Skip to content

Observability Runbook

Phase: 01-active-continuation-tracing Refs: TM-9, TM-10, TM-26

Repo-Side Setup

  • API OTLP endpoint is configured with OTEL_EXPORTER_OTLP_ENDPOINT.
  • API spans include service.namespace=trademate.
  • HTTPX instrumentation propagates trace context to journal and broker-adjacent calls.
  • Agent LLM instrumentation is opt-in through OPENINFERENCE_ENABLED=true.
  • Agent spans export to PHOENIX_COLLECTOR_ENDPOINT first, falling back to OTEL_EXPORTER_OTLP_ENDPOINT when Phoenix is not set.
  • ADR 0017 uses native OpenAI and Anthropic OpenInference instrumentation for customer-facing AI.

Deployment Checkpoints

  • Create SigNoz service from infra/signoz/compose.coolify.yml.
  • Create Phoenix service from infra/phoenix/compose.coolify.yml.
  • Wire private service DNS or internal Coolify hostnames.
  • Set production environment variables without committing secrets.
  • Import the starter SigNoz dashboard.

SigNoz Owner Login Setup

If SigNoz asks for an email on first open, create the owner account with the owner/admin email. This is a SigNoz application account; it is not controlled by TradeMate Better Auth.

  1. Open https://signoz.yourtrademate.io.
  2. If the first-user screen appears, register the owner email and store the password in the project password manager.
  3. If SigNoz requires SMTP for invites or password reset, configure SMTP in the SigNoz deployment settings or compose env. Use a verified sender domain.
  4. After login, confirm the Services page is reachable.
  5. Invite any additional operators from inside SigNoz rather than sharing the owner credentials.

Production OTEL Variables

Set these on trademate-api:

Variable Value
OTEL_SERVICE_NAME trademate-api
OTEL_SERVICE_NAMESPACE trademate
OTEL_EXPORTER_OTLP_ENDPOINT SigNoz OTEL collector HTTP endpoint, usually http://<signoz-collector>:4318 inside Coolify
OTEL_EXPORTER_OTLP_HEADERS Empty unless SigNoz is protected by collector auth
OTEL_RESOURCE_ATTRIBUTES deployment.environment=production,service.version=<deployed-sha>

Set equivalent variables on Celery/ingestion workers:

Variable Value
OTEL_SERVICE_NAME trademate-celery
OTEL_SERVICE_NAMESPACE trademate
OTEL_EXPORTER_OTLP_ENDPOINT Same SigNoz collector HTTP endpoint
OTEL_RESOURCE_ATTRIBUTES deployment.environment=production,service.version=<deployed-sha>

Set these on agent workers when LLM traces are being tested:

Variable Value
OPENINFERENCE_ENABLED true
PHOENIX_COLLECTOR_ENDPOINT https://phoenix.yourtrademate.io/v1/traces or the internal Phoenix collector endpoint
OTEL_EXPORTER_OTLP_ENDPOINT SigNoz fallback endpoint for non-LLM spans
OTEL_RESOURCE_ATTRIBUTES deployment.environment=production,service.namespace=trademate,service.version=<deployed-sha>

Use distinct service names for each deployable unit:

  • trademate-api
  • trademate-celery
  • trademate-agents
  • trademate-auth if Node OTEL is enabled later

Proof Harness

Use scripts/verify_observability.py whenever changing observability env vars or proving TM-9/TM-10 in production. The script has two modes:

  • Dry-run mode validates local environment variables and prints a redacted JSON report. It does not require OpenTelemetry packages.
  • Emit mode writes one named proof span per selected target. Run emit mode from a container or shell that has the same collector reachability as production.

Dry-run both targets:

powershell python scripts/verify_observability.py --target both --strict

Emit a SigNoz proof span:

powershell $env:OTEL_EXPORTER_OTLP_ENDPOINT = "http://trademate-signoz-otel-collector:4318" $env:OTEL_RESOURCE_ATTRIBUTES = "deployment.environment=production,service.version=<deployed-sha>" python scripts/verify_observability.py --target signoz --emit --strict --proof-id "obs-signoz-<date>"

Emit a Phoenix proof span:

powershell $env:PHOENIX_COLLECTOR_ENDPOINT = "http://trademate-phoenix:6006/v1/traces" $env:OPENINFERENCE_ENABLED = "true" $env:OTEL_RESOURCE_ATTRIBUTES = "deployment.environment=production,service.version=<deployed-sha>" python scripts/verify_observability.py --target phoenix --emit --strict --proof-id "obs-phoenix-<date>"

After emitting, search the observability UI for the exact trademate.proof_id attribute printed by the script. Save a screenshot or trace link in the phase verification artifact before changing TM-9 or TM-10 from partial to done.

SigNoz Smoke Test

  1. Redeploy the API and worker services after env changes.
  2. In Coolify logs, confirm there is no OTEL exporter startup error.
  3. Hit API health:

powershell Invoke-WebRequest https://api.yourtrademate.io/health

  1. Hit one authenticated endpoint, for example /users/me/profile from the web UI or with a valid bearer token.
  2. Open SigNoz -> Services.
  3. Confirm trademate-api appears with recent spans.
  4. Run the proof harness from a shell/container that can reach the collector, then search SigNoz for the printed trademate.proof_id.
  5. Open a trace and verify these attributes:
  6. service.name=trademate-api
  7. service.namespace=trademate
  8. deployment.environment=production
  9. HTTP route/status attributes for the smoke request
  10. Repeat with a scheduled or manually-triggered Celery ingestion task and confirm trademate-celery appears.

Phoenix Smoke Test

Use the smoke procedure that matches the deployed runtime slice.

Transitional Legacy Flow Smoke

Use this until ADR 0017 PR6/PR7 move the tested flow to the native OpenAI/Anthropic compatibility layer.

  1. Set OPENINFERENCE_ENABLED=true and PHOENIX_COLLECTOR_ENDPOINT on the agent/API/Celery services that own the tested LLM call path.
  2. Redeploy the agent/API/Celery services that own the tested LLM call path.
  3. Trigger a small existing agent-backed flow that still reaches the transitional path.
  4. Run the proof harness with --target phoenix --emit --strict.
  5. Open https://phoenix.yourtrademate.io.
  6. Confirm a new TradeMate stage trace appears with the service name or the printed trademate.proof_id.
  7. Native provider spans are expected only for flows already migrated to the TradeMate compatibility layer.
  8. Save a screenshot or trace link in the phase verification artifacts.

Native OpenAI/Anthropic Smoke

Use this after the tested flow has been migrated to the TradeMate native compatibility layer.

  1. Set OPENINFERENCE_ENABLED=true and PHOENIX_COLLECTOR_ENDPOINT.
  2. Redeploy the agent/API/Celery services that own the tested native LLM call path.
  3. Trigger a small agent-backed flow that makes a real OpenAI or Anthropic native SDK call.
  4. Run the proof harness with --target phoenix --emit --strict.
  5. Open https://phoenix.yourtrademate.io.
  6. Confirm a new trace appears with the TradeMate service name or the printed trademate.proof_id.
  7. Open the trace and verify provider, requested/reported model, reasoning profile, route kind, latency, token/cache/cost metadata, and redacted correlation identifiers.
  8. Save a screenshot or trace link in the phase verification artifacts.

Failure Triage

  • No service in SigNoz: verify the collector hostname/port from inside the application container and check OTEL_EXPORTER_OTLP_ENDPOINT.
  • Service exists but no routes: confirm the API process is instrumented and the request is hitting the deployed container you changed.
  • Phoenix empty: confirm OPENINFERENCE_ENABLED=true, the Phoenix endpoint ends in /v1/traces, and the tested flow actually made an LLM call.
  • Wrong environment label: fix OTEL_RESOURCE_ATTRIBUTES and redeploy.
  • Wrong deployed SHA: fix /version metadata injection for that service before using trace evidence as release evidence.