Operations

This section covers the operational aspects of running Rack Gateway in production, including monitoring, maintenance, upgrades, and troubleshooting.

Operational Areas

MonitoringHealth checks, metrics, and error tracking with Sentry.

Database MaintenanceMigrations, backups, and data management.

TroubleshootingCommon issues and how to resolve them.

UpgradesUpgrading Rack Gateway to new versions.

Quick Reference

Health Endpoint

Endpoint	Healthy	Unhealthy
/api/v1/health	200 {"status":"ok","service":"rack-gateway"}	503 {"status":"unhealthy",...,"error":"..."} for a minute after an audit write fails

It’s the only health endpoint. It doesn’t query the database: a request that fails to reach it shows up as errors in the logs and Sentry.

Common Commands

# Check gateway health
curl https://gateway.example.com/api/v1/health

# Check the audit log's hash chain and anchors (run on the gateway, e.g. via convox run)
./rack-gateway-api verify-audit

# Apply pending migrations
./rack-gateway-api migrate

Recent audit logs are on the web UI’s Audit Logs page, and GET /api/v1/audit-logs (admin).

Log Locations

Deployment	Log Location
Docker	docker logs <container>
Convox	convox logs -a rack-gateway

Operational Runbooks

Daily Checks

1. Verify the health endpoint responds
2. Check error tracking (Sentry) for new issues
3. Review audit logs for unusual activity, and Slack alerts for a failed audit verification

Weekly Tasks

1. Review user access patterns
2. Check for unused API tokens
3. Verify backup integrity

Monthly Tasks

1. Rotate API tokens as needed
2. Review RBAC role assignments
3. Check for available updates

Tip

Automate these checks where possible. Use monitoring tools to alert on health endpoint failures and error spikes.

Emergency Procedures

Gateway Unreachable

1. Check the health endpoint (a 503 names the failing audit write)
2. Verify database connectivity
3. Check container/pod status
4. Review recent deployments or changes

Suspected Security Incident

1. Do not modify audit logs (the database refuses updates and deletes, and the hash chain would show them)
2. Check audit log for suspicious activity
3. Lock affected users (ending their sessions), and revoke sessions and API tokens
4. Contact security team
5. Preserve evidence: export the audit log and run verify-audit

Database Issues

1. Check database connectivity from gateway
2. Verify database is accepting connections
3. Check for blocking queries or locks
4. Review recent migrations

See Troubleshooting for detailed procedures.