Upgrades

This guide covers upgrading Rack Gateway to new versions, including planning, execution, and rollback procedures.

Before Upgrading

Check Release Notes

Always review the release notes for breaking changes:

1. Visit the GitHub Releases
2. Read all releases between your current and target version
3. Note any breaking changes or migration requirements

Verify Current Version

# The CLI's version and the gateway's
rack-gateway version

rack-gateway version asks the gateway for its version (GET /api/v1/info), so it needs a login.

Backup Database

Create a backup before upgrading:

# RDS snapshot (AWS)
aws rds create-db-snapshot \
  --db-instance-identifier rack-gateway-prod \
  --db-snapshot-identifier rack-gateway-pre-upgrade-$(date +%Y%m%d)

# Or manual backup, with the admin connection
pg_dump "$ADMIN_DATABASE_URL" > backup_pre_upgrade.sql

Upgrade Procedures

Convox Deployment

1. Update the image tag

   Edit convox.yml:

   services:
     gateway:
       image: docker.io/docspringcom/rack-gateway:v1.2.0

2. Review environment changes

   Check if new environment variables are required:

   # Compare current env with docs
   convox env -a rack-gateway

3. Build, migrate, then promote

   The gateway doesn’t run migrations on startup in production, so run them against the new release before promoting it:

   convox build -a rack-gateway                 # note the release ID it prints
   convox run admin -a rack-gateway --release RELEASE_ID ./rack-gateway-api migrate
   convox releases promote RELEASE_ID -a rack-gateway --wait

   scripts/deploy.sh <rack> does these three steps through the gateway, and scripts/deploy_all.sh runs them for every rack in turn.

4. Verify deployment

   # Check health
   curl https://gateway.example.com/api/v1/health
   
   # Check logs for errors
   convox logs -a rack-gateway --since 5m

Docker Deployment

1. Pull new image

   docker pull docker.io/docspringcom/rack-gateway:v1.2.0

2. Stop current container

   docker compose down

3. Update compose.yaml

   services:
     gateway:
       image: docker.io/docspringcom/rack-gateway:v1.2.0

4. Run migrations (as the admin database user)

   docker compose run --rm gateway ./rack-gateway-api migrate

5. Start new version

   docker compose up -d

6. Verify

   curl http://localhost:8080/api/v1/health
   docker compose logs -f

Zero-Downtime Upgrades

On Convox, promoting a release is a rolling deploy: with the default deployment settings Convox starts the new process, waits for its health check, then stops the old one, so even a single instance upgrades without downtime. Migrations run before the promote, while the old release is still serving, so they must work with the old release; the release notes below say when one doesn’t.

Database Migrations

Migrations run automatically on startup only in development builds with DEV_MODE=true; the release image never runs them on startup. In production, run them against the new release before promoting it:

convox run admin -a rack-gateway --release RELEASE_ID ./rack-gateway-api migrate

Migrations are idempotent; running them again is safe.

Rollback Migrations

Danger

Rolling back migrations can cause data loss. Only do this if absolutely necessary and you have a backup.

1. Restore from backup if needed
2. Deploy the previous version
3. Migrations are forward-only. Most leave the schema usable by the previous version, but not all: a release note below says when the previous version can’t run (or can’t fully run) against the new schema.

Rollback Procedures

Quick Rollback (Convox)

# List previous releases
convox releases -a rack-gateway

# Rollback to previous release
convox releases rollback RELEASE_ID -a rack-gateway

Image Rollback

The image tag is pinned in convox.yml. To go back to an earlier version, deploy the convox.yml from that version’s tag (git checkout v1.1.0 -- convox.yml, then scripts/deploy.sh <rack>), or roll back to a release that already runs it. Migrations aren’t undone, so check the release notes below before rolling back past one.

Database Rollback

If a migration caused issues:

1. Stop the gateway
2. Restore database from backup
3. Deploy previous version

# Stop gateway
convox scale gateway --count 0 -a rack-gateway

# Restore database (example for RDS)
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier rack-gateway-restored \
  --db-snapshot-identifier rack-gateway-pre-upgrade-20240115

# Update DATABASE_URL and ADMIN_DATABASE_URL to point to restored instance
# Deploy previous version

A restore to before the upgrade loses the audit events written since; see Restore from Backup.

Release Notes That Need Action

Security audit fixes (v0.1.15)

This release fixes the findings of a security audit. Most changes need nothing from you; check these before you deploy.

Before deploying:

1. The gateway verifies its database’s certificate. convox.yml sets PGSSLMODE=verify-full with Amazon RDS’s CA bundle, which the image ships. DATABASE_URL (and ADMIN_DATABASE_URL) must name the database by its RDS hostname: an IP address or a CNAME fails the hostname check. For a database that isn’t on RDS, set PGSSLROOTCERT to its CA.
2. Service restarts need convox:app:restart (they took convox:process:start). Give it to any API token that restarts services.
3. CircleCI auto-approval needs the app’s vcs_repo. Without it, approving a request no longer queues the CircleCI approval job. The job also checks the pipeline number and the repository’s host, owner and name.
4. Deploys from the repository’s default branch are refused when allow_deploy_from_default_branch is off, whatever default_branch says: the gateway now reads the default branch from GitHub too.
5. Rate limits on failures and approvals. Each client IP gets AUTH_FAILURE_LIMIT_BURST (30) refused sign-ins, refilled at AUTH_FAILURE_LIMIT_PER_MINUTE (30). A session or API token that signed in within the last 24 hours, or a session just created, still gets through when its IP is refused, so clients sharing an address aren’t locked out by one of them. It has the same limit of its own, which caps how many requests with it are checked at once. Each user can create DEPLOY_APPROVAL_LIMIT_BURST (20) deploy approval requests, across all their API tokens, refilled at DEPLOY_APPROVAL_LIMIT_PER_HOUR (60). Raise that if CI creates more requests than that. A request that repeats an open one gets 409 with that request and doesn’t count, but every attempt, repeats and refusals included, counts against a limit three times as large.
6. Settings with a stored value of the wrong type are errors. The MFA settings refuse to load one, rather than use a default, and approving or extending a deploy approval request fails while deploy_approval_window_minutes can’t be read. If a settings error shows up after migrating, set that setting again.

What changes for users and CI:

• More env vars are secrets: names containing PASSWORD, PASSWD, PASSPHRASE or CREDENTIAL, or ending in _PASS. A value with credentials in it (a URL with a user or password, password=) is masked for callers without convox:secret:read, whatever its name.
• Creating a user whose email exists answers 409 instead of replacing their roles. Change roles with an update. An update whose roles were decided against roles that changed since it read them (another admin changed them) also gets 409: reload the user and make the change again.
• Changing a user’s email deletes their API tokens and ends their sessions. Issue new tokens.
• An archive’s object key can’t be uploaded twice, and a concurrent env change waits up to 15 seconds for the app’s env lock, then gets 409.
• A promote whose outcome the gateway couldn’t see keeps its approval’s claim: CI needs a new approval to promote again.
• A build, object or process the rack created but the approval couldn’t record gets 503, naming what to do again (create the build again, upload to a new key, start a new process). So does an upload the rack says it stored at a different key.
• Exec and log sessions are rechecked every minute and end when the user, session, token, permission or approval is gone, or when a recheck can’t finish within 30 seconds. A session lasts at most 12 hours, so a long convox logs -f reconnects. One a deploy approval allowed lasts at most an hour, whether or not the approval’s window ends first.
• Web logout is a POST with the session’s CSRF token; a GET no longer signs out.

Logs and audit:

• An exec’s command is audited with the credentials a command line commonly carries redacted: a URL’s password, the value of a secret-named assignment or flag (DB_PASSWORD=..., --token ...), the value of a credential header (Authorization, Cookie, X-API-Key: ...), a Bearer value, and the credentials common programs take with short flags (mysql -p..., redis-cli -a, sshpass -p, curl -u user:...). A deploy approval request’s record of the commands run under it, and the refusal of a convox run command it doesn’t approve, are redacted the same way. Redaction can’t recognize every credential, and the audit log is permanent: pass credentials to commands through the app’s env, never on the command line.
• A request’s query string is logged and audited with the value of every parameter named for a credential redacted (api_key, client_secret, password, any *token*, code, state, …), not only the login parameters.
• Request log lines say rbac_decision deny for every 401, 403 and 429, error when the permission check failed, and not_evaluated when no check ran. Update CloudWatch filters that relied on the old values.
• A deploy approval request’s approve, reject and extend entries use the resource type deploy_approval_request (it was deploy-approval-request), like its create and expire entries, and all of them set deploy_approval_request_id.
• With AUDIT_HASH_V2_FROM pinned, a chain that has no hash v2 events fails verification. An anchor over 16 KiB fails verification. When the chain has legacy (v1) events, verify-audit’s result says their content can’t be verified.

users.suspended dropped (v0.1.12)

This release’s migration drops the unused users.suspended column. v0.1.10 and earlier still read it, so they can’t load users once you’ve migrated.

1. Upgrade through v0.1.11 first. Migrations run before the promote, while the old release is still serving (see Zero-Downtime Upgrades), and only v0.1.11 or later keeps working against the new schema.
2. Don’t roll back past v0.1.11 after migrating.

Anchor bucket retention checked at startup (v0.1.11)

AUDIT_ANCHOR_RETENTION_DAYS is now checked: the gateway reads the anchor bucket’s Object Lock configuration at startup and in every audit verification, and refuses to start (verification fails and alerts) unless the bucket’s default retention is COMPLIANCE mode for at least that many days.

1. Give the gateway’s role s3:GetBucketObjectLockConfiguration on the anchor bucket before deploying (see IAM Role for the Gateway), and the role verify-audit runs with elsewhere. Without it the new release doesn’t start.
2. Check AUDIT_ANCHOR_RETENTION_DAYS matches the bucket’s retention (worm_retention_days). It defaults to 400.

Audit hash v2 and verify-audit

This release changes how audit events are hashed (hash v2: every stored column, recomputable from the row) and adds rack-gateway-api verify-audit, which checks the chain and the S3 anchors. The gateway also runs those checks itself every hour and alerts when they fail.

1. The previous version can’t write audit events to the database once you’ve migrated. The migration replaces audit.append_audit_event. From migrate until the new release is promoted, the running release’s audit writes fail; requests aren’t blocked, the failures are logged, and every event still goes to stdout (and so to CloudWatch). Promote straight after migrating (scripts/deploy.sh does). The same applies if you roll back: the earlier release can’t write audit events to the database until you roll forward again. Events from those windows are only in the logs, never in the chain.

2. Existing events become legacy (v1) events. verify-audit can’t recompute their HMAC (it covered values the database doesn’t store exactly), so it only checks their previous_hash links. Events written from now on are hash v2 and fully checked.

3. Pin where hash v2 starts. Right after promoting, run verify-audit (see below). It prints hash v2 from event N or, before the new release has written an event, a failure naming the event that will be the first. Set AUDIT_HASH_V2_FROM=N in the gateway’s environment and wherever else verify-audit runs. With the pin set, verification fails until that first hash v2 event exists (any request writes one). Without it, verification fails while legacy events exist, until an anchor written by this release (within the hour) confirms the start, and always without anchors. See Pinning where hash v2 starts.

4. Give the gateway’s role s3:ListBucketVersions and s3:GetObjectVersion on the anchor bucket (see IAM Role for the Gateway). The hourly verification reads every version of each anchor; without them every run fails and alerts. To run verify-audit elsewhere, it needs AUDIT_HMAC_SECRET, a database role that can read audit.audit_event (audit_reader is enough), and for the anchor checks AUDIT_ANCHOR_BUCKET, AUDIT_ANCHOR_CHAIN_ID and AWS credentials allowed s3:ListBucket, s3:ListBucketVersions, s3:GetObject and s3:GetObjectVersion on the anchor bucket and kms:Decrypt on its key.

5. Run it once, and route its alerts to Slack.

   convox run gateway -a rack-gateway ./rack-gateway-api verify-audit

   From then on the gateway runs the same checks every hour, sends failures to Sentry and Slack, and shows the latest result on the Audit Logs page (Hourly verification in the gateway). Slack integrations connected from this release on send the alerts (actions audit_log.verification_failed and audit_log.verification_recovered) to #security; for an existing integration, add audit_log.verification_* to a channel’s action patterns on the Integrations page. The same goes for failed MFA verifications (mfa.verify_failed). Verification also fails when no anchor has been written for 3 hours.

6. Expect findings in anchors from earlier versions. Anchors written before this release have no format. Their prev_anchor_hash stopped naming the previous anchor once the bucket held 1000 anchors; those are counted, not failures. Earlier releases didn’t serialise anchor writes, so concurrent runs could write an hour’s key twice, which verify-audit reports as an overwritten anchor; check the versions’ timestamps and contents to confirm they’re from the same hour’s job. Earlier releases also retried a failing anchor job for weeks; an anchor without a format stored more than 6 hours after its ts_utc gets a warning. Builds from before v0.0.1 wrote per-second keys (anchor-YYYYMMDDTHHMMSSZ.json, reported as not matching ts_utc) and .sha256 files (ignored, with a warning). Locked anchors can’t be changed, so these findings repeat on every run, and the hourly verification keeps failing (with a daily reminder), until the anchors’ retention ends and they’re deleted.

Column encryption key required (v0.1.7)

TOTP seeds, YubiKey public IDs and the Slack bot token are now stored encrypted with a dedicated key. See Column Encryption.

1. Set COLUMN_ENCRYPTION_KEY before deploying (openssl rand -base64 32) and back it up. Every command that opens the database, including migrate, refuses to run without it, so a deploy without the key fails before the new release is promoted.
2. Existing secrets are encrypted when the new release starts (v0.1.7 to v0.1.10 only), before it serves requests. Releases after v0.1.10 don’t encrypt existing secrets, so upgrade a gateway from before v0.1.7 to v0.1.10 first.
3. Expect a short gap for TOTP and YubiKey OTP. Once the new release has upgraded the data, any process still running the previous release can’t read the encrypted values, so its TOTP and YubiKey OTP checks and Slack notifications fail until it is replaced. WebAuthn and backup codes keep working.
4. Rolling back to a release from before column encryption has the same effect until you deploy this release again.

Test bypasses compiled out of the release image

The published image no longer contains the development and test bypasses. It refuses to start, migrate, reset-db or write-anchor while any of these is set, naming each one:

• DEV_MODE or E2E_TEST_MODE set to a true value (DEV_MODE=false is fine)
• COOKIE_SECURE=false (session cookies always have Secure, so serve the gateway over HTTPS)
• AWS_ENDPOINT_URL, AWS_ENDPOINT_URL_S3, AWS_ENDPOINT_URL_STS, AWS_ENDPOINT_URL_KMS or POSTMARK_API_BASE
• a GOOGLE_OAUTH_BASE_URL that isn’t https://

Before upgrading, remove them from the gateway’s environment. rack-gateway-api build-info shows whether an image has the bypasses compiled in.

A database the gateway first started with DEV_MODE on is marked development, and stays marked that way. Check SELECT environment FROM rgw_internal_metadata on production databases; a production gateway should report production.

Secrets and env var lists checked at startup (v0.1.7)

Check these before upgrading:

• Secrets: APP_SECRET_KEY and AUDIT_HMAC_SECRET must each be at least 32 bytes, and neither can be the development value from this repository. migrate refuses weak ones too, so a deploy stops before it promotes. Generate new ones with openssl rand -base64 32. Changing AUDIT_HMAC_SECRET breaks verification of earlier audit events, so only replace it if it’s actually weak.
• Env var list variables: CONVOX_SECRET_ENV_VARS and the protected_env_vars, secret_env_vars and audit_visible_env_vars setting variables must be a JSON array of names or a comma-separated list of names. A quoted value ('"A,B"', as jsonencode of a string writes) or an invalid name stops the gateway from starting.
• Stored lists and image patterns: a stored env var list with an invalid name (for example STRIPE_*, which never matched anything) makes that app’s env count as all secret and protected. Its env pages return 500 until you fix the list under Apps → {app} → Settings. A stored service_image_patterns value that isn’t an object of valid patterns refuses the app’s builds.
• Promotes are checked against image patterns: for an app with service_image_patterns, a promote now loads the release’s manifest with the release’s env, as the rack does, and is refused (403) if an init container image doesn’t match the patterns, with {{GIT_COMMIT}} as the commit the release was built from, or a resource sets its own image. Check this before rolling back to an older release. See Promotes are checked too.

Deploy approvals bound to commits (v0.1.3)

Approval-bound CI builds now have to use pre-built images tagged with the approved commit, from a repository pinned by service_image_patterns. Builds are refused until the patterns are set, so plan the rollout:

1. Set service_image_patterns right after the new version is live, not before. Earlier versions require the build’s git-sha once patterns are configured, and a CI build from a directory that isn’t a git checkout sends none, so setting them first breaks deploys on the old version. Avoid running CI deploys until the patterns are set: a build refused in that window has already used up its approval’s archive upload, so the request has to be rejected (or left to expire) and requested again.
2. Check approved_deploy_commands is a JSON array of strings for every app. An empty list (or a value of any other shape) now allows no commands, where it used to allow every command. The settings API refuses other shapes from this version on.
3. Check vcs_repo is owner/repo for apps using CircleCI auto-approval. Hold jobs are only approved when the workflow’s pipeline builds the approved commit of that repository.
4. CircleCI approval jobs queued by the previous version are cancelled rather than run; approve those holds in CircleCI by hand.
5. Earlier versions stored CIRCLECI_TOKEN and GITHUB_TOKEN in background job arguments. Once the old version is fully drained, remove them from river_job.args and rotate both tokens.

Version Compatibility

Rack Gateway is pre-1.0: any release can change the API, the CLI or the schema. Keep the CLI and every gateway on the same release, and read the release notes for each version you skip. rack-gateway version shows the CLI’s version and the gateway’s; a CLI that’s too old or too new for the gateway’s login flow says so when you log in.

Migrations are forward-only. Rolling back the gateway works when the previous version can use the new schema (the release notes say when it can’t); otherwise restore from backup.

Upgrade Checklist

Before upgrading:

• Read release notes for all versions
• Backup database
• Note any new required environment variables
• Schedule maintenance window (if not zero-downtime)
• Notify users of potential brief interruption

After upgrading:

• Verify the health endpoint
• Check application logs for errors
• Test critical functionality (login, deploy)
• Run verify-audit, or check the next hourly verification on the Audit Logs page
• Monitor error rates for 24 hours
• Upgrade the CLI to the same release

Troubleshooting Upgrades

Upgrade Stuck

# Check pod/container status
convox ps -a rack-gateway

# Check events
convox logs -a rack-gateway --since 5m

Health Check Failing

# Test health endpoint inside a running process (PID from convox ps)
convox exec <pid> -a rack-gateway wget -q -O - http://localhost:8080/api/v1/health

# Check startup logs
convox logs -a rack-gateway | grep -i "start\|error"

Database Migration Error

# Run the migrations again; the error names the failing migration
convox run admin ./rack-gateway-api migrate -a rack-gateway

-- Applied migrations
SELECT version FROM schema_migrations ORDER BY version DESC LIMIT 10;

Each migration runs in its own transaction, so a failed one leaves nothing behind.

Further Reading

• Database Maintenance - Backup and restore procedures
• Monitoring - Verify upgrade success
• Troubleshooting - Resolve upgrade issues