Private Network Deployment

This guide covers deploying your Convox rack and Rack Gateway on a completely private network, with no public internet exposure.

Why Go Fully Private?

By default, the Convox API is exposed publicly via nginx ingress with authentication via the rack password. While protected by authentication, this presents security concerns:

Discoverable endpoint - The API is subject to brute force attempts
Token compromise risk - Any rack token compromise grants API access from anywhere
No network-level access control - Cannot enforce IP restrictions for sensitive operations
Compliance requirements - Some organizations require all internal tools to be on private networks

With a fully private deployment:

The Convox API is only accessible through your VPN/private network
Even if the rack token is compromised, attackers can’t reach the API
You gain defense-in-depth: network security + authentication + RBAC

Architecture Overview

The `private_api` Feature

There’s an open PR (#935) to add a private_api parameter to Convox that disables public access to the API entirely.

Using DocSpring’s Fork

DocSpring maintains a public fork of Convox that includes the private_api feature along with other improvements:

private_api - Disable public nginx ingress for the Convox API
CloudFlare DNS-01 resolver - Alternative certificate validation method
Various bug fixes and enhancements

Deployment with Tailscale

Set up Tailscale Operator in your cluster

Follow the Tailscale Kubernetes operator guide to install the operator in your EKS cluster.

Configure your Convox rack with private_api

Use DocSpring’s fork in your Terraform configuration:

module "system" {
  source  = "github.com/DocSpring/convox//terraform/system/aws?ref=docspring"
  image   = "docspringcom/convox"
  release = "docspring-004"

  name   = "production"
  region = "us-east-1"

  # Private EKS API access (existing Convox feature)
  private_eks_host      = "https://eks-production.your-tailnet.ts.net"
  disable_public_access = true

  # Private Convox API access (DocSpring fork feature)
  private_api = true

  # ... other configuration ...
}

Create Tailscale ingress for the Convox API

resource "kubernetes_ingress_v1" "convox_api_tailscale" {
  metadata {
    name      = "convox-api"
    namespace = "rack-system"  # or your rack namespace
    annotations = {
      "tailscale.com/expose"   = "true"
      "tailscale.com/hostname" = "convox-api-production"
      "tailscale.com/tags"     = "tag:convox-api"
    }
  }

  spec {
    ingress_class_name = "tailscale"
    rule {
      host = "convox-api-production.your-tailnet.ts.net"
      http {
        path {
          path      = "/"
          path_type = "Prefix"
          backend {
            service {
              name = "api"
              port { name = "https" }
            }
          }
        }
      }
    }
    tls {
      hosts = ["convox-api-production.your-tailnet.ts.net"]
    }
  }
}

Deploy Rack Gateway with Tailscale access

When the gateway runs on the rack it protects, leave RACK_HOST unset: it reaches the Convox API inside the cluster (https://api.<RACK>-system.svc.cluster.local:5443) and never needs the Tailscale ingress. Only a gateway running elsewhere sets it:

# In your Rack Gateway environment (gateway outside the rack only)
RACK_HOST=https://convox-api-production.your-tailnet.ts.net
RACK_TOKEN=your-rack-token

Expose Rack Gateway itself via Tailscale. Convox puts the app in the <rack>-rack-gateway namespace, with a Kubernetes service named after the gateway service:

resource "kubernetes_ingress_v1" "rack_gateway_tailscale" {
  metadata {
    name      = "rack-gateway"
    namespace = "production-rack-gateway"
    annotations = {
      "tailscale.com/tags" = "tag:rack-gateway"
    }
  }

  spec {
    ingress_class_name = "tailscale"
    rule {
      host = "gateway-production.your-tailnet.ts.net"
      http {
        path {
          path      = "/"
          path_type = "Prefix"
          backend {
            service {
              name = "gateway"
              port { name = "main" }   # the service's port (8080)
            }
          }
        }
      }
    }
    tls {
      hosts = ["gateway-production.your-tailnet.ts.net"]
    }
  }
}

The Tailscale ingress proxy connects to the gateway from inside the VPC, so set TRUSTED_PROXY_CIDRS to the rack’s VPC CIDR (for example 10.2.0.0/16). The gateway then takes the client’s tailnet address from the proxy’s X-Forwarded-For, and the audit log, rate limits and security emails name the real client.

Configure Tailscale ACLs

In your Tailscale admin console, set up ACLs to control who can access what:

{
  "acls": [
    {
      "action": "accept",
      "src": ["group:developers"],
      "dst": ["tag:rack-gateway:443"]
    },
    // Only for a gateway outside the rack, which reaches the Convox API over Tailscale
    {
      "action": "accept",
      "src": ["tag:rack-gateway"],
      "dst": ["tag:convox-api:443"]
    }
  ],
  "tagOwners": {
    "tag:convox-api": ["group:infrastructure"],
    "tag:rack-gateway": ["group:infrastructure"]
  }
}

Verifying the Setup

Check that public access is disabled

# Should timeout or return DNS error
curl https://api.your-rack-domain.convox.cloud
# Connection refused or timeout expected

Verify private access works

# Connect to Tailscale first
tailscale up

# Test Convox API via Tailscale
curl https://convox-api-production.your-tailnet.ts.net/system

# Test Rack Gateway via Tailscale
curl https://gateway-production.your-tailnet.ts.net/api/v1/health

Configure the rack-gateway CLI

# Login using the Tailscale hostname
rack-gateway login production https://gateway-production.your-tailnet.ts.net

# Verify commands work
rack-gateway apps

Alternative Private Network Options

While we recommend Tailscale for its simplicity, you can achieve private network access with other methods:

Pros:

Zero-config mesh VPN
Works across cloud providers
Built-in ACLs and audit logging
Easy to set up Kubernetes operator

Cons:

Requires Tailscale subscription for teams
External dependency

Security Benefits

With a fully private deployment, you achieve defense-in-depth:

Layer	Protection
Network	API only accessible via private network
Authentication	Google Workspace OAuth via Rack Gateway
Authorization	RBAC controls what each user can do
Audit	Every action logged with user attribution
MFA	Additional verification for sensitive operations

Even if an attacker obtains:

Rack token alone → Can’t reach the API (network blocked)
Tailscale access alone → Can’t authenticate (no OAuth session)
OAuth session alone → Limited by RBAC permissions
All of the above → Actions are logged and auditable

Next Steps

Terraform Infrastructure - Full Terraform setup guide
S3 WORM Storage - Immutable audit log storage
Production Checklist - Pre-launch security review