Monitoring

Rack Gateway gives you four signals: a health endpoint, heartbeats it sends out, errors in Sentry, and logs. It doesn’t expose metrics; use your platform’s container metrics for CPU and memory.

Health Endpoint

curl https://gateway.example.com/api/v1/health

Healthy:

{"status": "ok", "service": "rack-gateway"}

For a minute after an audit log write fails, the gateway refuses every request it can’t record, and the health check answers 503:

{"status": "unhealthy", "service": "rack-gateway", "error": "..."}

It doesn’t query the database, so it stays healthy while the database is unreachable until a request fails to write its audit entry. Use it for container health checks and load balancers.

Convox

The convox.yml in the repository already points Convox’s health check at it:

services:
  gateway:
    health: /api/v1/health
    port: 8080

Heartbeats

A gateway usually runs on a private network (for example behind Tailscale) that an uptime service can’t reach, so it reports in instead: it requests heartbeat URLs on a schedule, and a heartbeat monitor (Cronitor, Healthchecks.io and the like) alerts when the requests stop. Set the URLs as secrets: they usually carry the monitoring service’s key, so the gateway never logs them.

Uptime (UPTIME_HEARTBEAT_URL): requested when the gateway starts and every 15 minutes (UPTIME_HEARTBEAT_INTERVAL_MINUTES) while the health check would pass. It’s a background job, so it stops when the gateway is down, can’t reach its database, has stopped running jobs, or can’t write its audit log. Give the monitor that schedule and some grace for restarts, for example 15 minutes.
Audit verification (AUDIT_VERIFY_HEARTBEAT_URL): requested after every hourly audit verification run, half past every hour (UTC), whatever it found: failures already alert through Sentry and Slack, and the monitor alerts when runs stop. A run takes about a minute for a year of anchors, so expect the ping a few minutes after :30 and give the monitor up to 30 minutes of grace, the run’s time limit.

A ping that fails is logged and not retried; the next one is due on schedule. Leave both unset on a gateway that is stopped on purpose (for example a staging rack shut down at night), or its monitors will alert.

Error Tracking with Sentry

Configuration

# Backend error tracking
SENTRY_DSN=https://abc123@sentry.io/123456

# Frontend error tracking
SENTRY_JS_DSN=https://def456@sentry.io/789012

# Environment tag
SENTRY_ENVIRONMENT=production

At startup the gateway logs Sentry enabled (environment=..., release=...). See Environment Variables for the release and sampling settings.

What Gets Tracked

Event Type	Details
Panics	Recovered panics with stack traces
HTTP Errors	Every 5xx response
Background jobs	Each failed job attempt (email, Slack, CircleCI, GitHub, audit anchors), tagged with the job kind
Audit failures	A failed audit write, and the hourly audit verification failing
Notifications	Slack messages that couldn’t be sent

Sensitive Data

The gateway turns off the SDK’s own collection of request data and user details. Before an event leaves, it removes every request header except Accept, Content-Type, User-Agent and X-Request-ID, and drops cookies, the query string and the request body, so session tokens, API tokens and env values never reach Sentry.

Logs

Log Format

Rack Gateway writes two kinds of lines to stdout:

Application logs in Go’s log format, with a level and optional topic, for example [ERROR] job email:user:welcome failed: ...
Request logs: one JSON line per request

2025/01/29 12:34:56 Sentry enabled (environment=production, release=RABCDEF)

{
  "ts": "2025-01-15T10:30:00Z",
  "user_email": "alice@example.com",
  "method": "POST",
  "path": "/apps/myapp/releases/RABC123/promote",
  "status": 200,
  "latency_ms": 1250,
  "rbac_decision": "allow",
  "request_id": "0b6c...",
  "client_ip": "100.64.0.12"
}

query_params and response_error are added when present. rbac_decision is deny for any request the gateway refused (401, 403 or 429, including failed sign-ins and rack proxy requests refused before they reached the rack), error when the permission check failed, allow when authorization let the request through, and not_evaluated for routes without authorization (login, health). These lines are for log search; the audit log in the database (with its hash chain and anchors) is the record. See Audit Logging.

Log Levels

Level	When Used
`error`	Failures requiring attention
`warn`	Potential issues, degraded functionality
`info`	Normal operations
`debug`	Detailed diagnostics, only for the topics in `DEBUG_TOPICS`

LOG_LEVEL=info  # Options: debug, info, warn, error

Where Logs Go

On Convox, convox logs -a rack-gateway follows them, and the rack ships them to CloudWatch Logs, kept for LOG_RETENTION_DAYS (the awsLogs setting in convox.yml). With Docker, use docker logs or your log driver.

What to Alert On

Signal	How
Gateway down or stuck	The uptime heartbeat stops
Audit writes failing	Health check returns 503; Sentry error
Audit chain or anchors wrong	Audit verification failure in Sentry and Slack (`audit_log.verification_*`), or its heartbeat stops
Errors	New Sentry issues, 5xx responses
Security events	Slack routing for `login._failed`, `mfa.verify_failed`, `api_token.` and the other default patterns
Capacity	Container CPU and memory from your platform (the default limit is 128 MB)