Skip to content
Rack Gateway

Troubleshooting

This guide covers common issues with Rack Gateway and how to diagnose and resolve them.

Quick Diagnostics

Section titled “Quick Diagnostics”
Terminal window
# Check gateway health
curl https://gateway.example.com/api/v1/health


# View recent logs (Convox)
convox logs -a rack-gateway --since 10m

Authentication Issues

Section titled “Authentication Issues”

“Login Failed” After OAuth

Section titled ““Login Failed” After OAuth”

Possible causes:

  • GOOGLE_ALLOWED_DOMAIN doesn’t match the user’s email domain
  • The user hasn’t been added to the gateway (signing in never creates a user)
  • The user’s account is locked
  • Redirect URI mismatch in Google Cloud Console
  • DOMAIN misconfigured (affects OAuth redirects)

The audit log records each refused login (login.*_failed, or login.user_not_authorized) with the reason.

Checks:

  1. Confirm redirect URIs include:
    • https://<DOMAIN>/api/v1/auth/web/callback
    • https://<DOMAIN>/api/v1/auth/cli/callback
  2. Verify GOOGLE_ALLOWED_DOMAIN matches your workspace domain.

Web Session Ends Quickly

Section titled “Web Session Ends Quickly”

Possible causes:

  • APP_SECRET_KEY changed between restarts
  • Web sessions end after session_timeout_minutes without activity (5 minutes by default)

Checks:

  • Ensure APP_SECRET_KEY is stable and stored in Convox env.
  • Raise the timeout on the Settings page, or with RGW_SETTING_SESSION_TIMEOUT_MINUTES.

CLI Login Hangs

Section titled “CLI Login Hangs”

The CLI listens on http://127.0.0.1:<random port>/callback and waits (up to 10 minutes) for the browser to be redirected there with a single-use login code.

  1. Check gateway reachability

    Terminal window
    curl https://gateway.example.com/api/v1/health

  2. Use a browser on the same machine as the CLI

    The login code is delivered to 127.0.0.1. For a remote host, forward the port from the CLI’s “Waiting for the browser to return to http://127.0.0.1:<port>/callback” line with ssh -L <port>:127.0.0.1:<port> <host>. Local firewalls must allow connections to 127.0.0.1.

  3. Use --no-open to capture the auth URL

    Terminal window
    rack-gateway login production https://gateway.example.com --no-open

  4. Complete the browser flow

    Ensure the browser can reach the gateway and Google OAuth endpoints.

Authorization Issues

Section titled “Authorization Issues”

“Forbidden” for Expected Users

Section titled ““Forbidden” for Expected Users”

Possible causes:

  • User role not updated in the gateway
  • Permission requires MFA step-up

Checks:

  • Verify roles in the web UI (Users page). A role change applies to the user’s next request.
  • See Permissions for what each role can do. The CLI asks for MFA itself when a command needs it; --mfa-code passes a code non-interactively.

API Token Rejected

Section titled “API Token Rejected”

Possible causes:

  • Token revoked or expired
  • Token not passed correctly

Checks:

  • Token must start with rgw_.
  • Use RACK_GATEWAY_API_TOKEN or --api-token, and set RACK_GATEWAY_URL when there’s no config file (CI).
  • Confirm token exists in the web UI (API Tokens) and hasn’t expired.
  • A token stops working while its owner is locked, and is deleted with its owner.
  • A token never gets more than its owner’s role, and never more than its own permissions.

Convox Proxy Issues

Section titled “Convox Proxy Issues”

“Bad Gateway” Errors

Section titled ““Bad Gateway” Errors”

Possible causes:

  • Convox API unreachable
  • RACK_TOKEN invalid or missing
  • RACK_HOST misconfigured

Checks:

  • Verify RACK_TOKEN is correct and set.
  • If running outside the rack network, set RACK_HOST to the Convox API URL.
  • Check gateway logs for upstream errors.

MFA Issues

Section titled “MFA Issues”

“Invalid TOTP Code”

Section titled ““Invalid TOTP Code””
  • Ensure device clock is in sync
  • Try the next code window (30 seconds)

“WebAuthn Not Available”

Section titled ““WebAuthn Not Available””
  • Ensure the CLI build has WebAuthn support (CGO + libfido2); the release build doesn’t
  • Fall back to TOTP, YubiKey OTP or a backup code with --mfa-method or --mfa-code

Account Locked After Failed MFA

Section titled “Account Locked After Failed MFA”

Five failed MFA attempts in five minutes lock the account. The admins are emailed, and the lock is in the audit log as user.lock by system. An admin unlocks it on the Users page once they’ve confirmed the attempts were the user’s.

Lost MFA Device

Section titled “Lost MFA Device”
  1. Sign in with a backup code, then remove the lost method and enroll a new one under Account Security
  2. Without any method or backup code, an admin chooses Reset MFA on the user’s page (after confirming who’s asking). It removes their methods, backup codes and trusted devices and signs them out; their API tokens are kept. They set up a new method when they next sign in. If the account was locked by failed attempts, unlock it too

Database Issues

Section titled “Database Issues”

If the gateway fails to start or reports DB errors:

  1. Verify DATABASE_URL / RGW_DATABASE_URL is set
  2. Run migrations (production never migrates on startup):
Terminal window
convox run admin ./rack-gateway-api migrate -a rack-gateway
  1. If requests fail with 503 and the health check names an audit write, see Database Maintenance: the gateway refuses requests it can’t audit.