Audit Logging

Audit logging creates a permanent record of all actions taken in a system. These logs serve multiple purposes: security forensics, compliance evidence, and operational visibility. Effective audit logs answer the questions: Who did what, when, and what was the outcome?

Why Audit Logs Matter

Security Forensics

When a security incident occurs, audit logs are essential for:

• Determining scope: What was accessed?
• Identifying attackers: Who performed the actions?
• Understanding timeline: When did the breach begin and end?
• Assessing damage: What changes were made?

Compliance Requirements

Regulatory frameworks mandate audit logging:

Framework	Audit Requirements
SOC 2	Log access to customer data, retain for defined period
HIPAA	Log access to protected health information
PCI DSS	Log all access to cardholder data
GDPR	Document processing activities
ISO 27001	Event logging and monitoring

Operational Visibility

Beyond security, audit logs provide:

• Change tracking: Who modified configuration?
• Troubleshooting: What happened before the outage?
• Usage patterns: How is the system being used?
• Accountability: Who approved this deployment?

Anatomy of an Audit Log Entry

A complete audit log entry captures:

{
  "timestamp": "2024-01-15T14:30:45.123Z",
  "event_type": "convox.releases.promote",
  "actor": {
    "type": "user",
    "id": "user_123",
    "email": "alice@example.com",
    "ip_address": "192.168.1.100",
    "user_agent": "rack-gateway-cli/1.2.0"
  },
  "action": {
    "method": "POST",
    "path": "/apps/myapp/releases/R123/promote",
    "parameters": {
      "app": "myapp",
      "release_id": "R123"
    }
  },
  "outcome": {
    "status": "success",
    "response_code": 200
  },
  "context": {
    "rack": "production",
    "session_id": "sess_abc123",
    "request_id": "req_xyz789"
  }
}

Required Fields

Field	Purpose	Example
timestamp	When the event occurred	ISO 8601 format
event_type	What kind of event	convox.apps.create
actor	Who performed the action	User email, IP address
action	What was done	HTTP method, path, parameters
outcome	What happened	Success/failure, status code

Optional but Valuable

Field	Purpose
session_id	Links multiple actions
request_id	Traces through distributed systems
user_agent	Client identification
changes	Before/after values

Sensitive Data in Logs

The Problem

Logs often contain sensitive data:

{
  "action": "env.set",
  "parameters": {
    "DATABASE_URL": "postgres://user:password@host/db",
    "API_KEY": "sk_live_abc123secret"
  }
}

This creates risk:

• Logs may be accessed by support staff
• Log aggregation services may store data externally
• Breached logs expose credentials

The Solution: Redact by Default

Guessing which values are sensitive from their names misses credentials under ordinary names (DATABASE_URL, SMTP_PASSWORD). Rack Gateway logs one entry per changed env var and redacts every value:

{
  "action": "env.set",
  "resource": "myapp/DATABASE_URL",
  "details": {"old": "[REDACTED]", "new": "[REDACTED]"}
}

Note

An admin can list harmless env vars (PORT, RAILS_ENV) in the app’s audit_visible_env_vars setting to see their values in the audit log and Slack. Secret and protected env vars are redacted even when listed.

What Gets Redacted

Data	Logged as
Env var values	[REDACTED], unless listed in audit_visible_env_vars
Rack parameter values	[REDACTED]; the changed parameter names are logged
Login codes, OAuth values and tokens in query strings	[REDACTED]

See Secret Redaction for the details.

Immutability and Integrity

Why Immutability Matters

If attackers can modify logs, they can:

• Delete evidence of their access
• Alter timestamps to confuse investigations
• Remove their user from logged actions
• Make it appear another user was responsible

Achieving Immutability

Write-Once Storage (WORM)

Rack Gateway supports S3 Object Lock for tamper-evident logs:

Cryptographic Chaining

Every audit event is chained to the one before it. Its event_hash is an HMAC-SHA256, keyed with AUDIT_HMAC_SECRET, over every stored column of the event: its id, chain index, timestamp, the previous event’s hash and all of its content. Each value is length-prefixed, so content can’t move from one field to another without changing the hash. Changing, deleting or reordering an event breaks the chain at that event, except for deleting the newest events: the events before them still form a valid chain, and only an anchor shows they existed.

Values are normalized before they’re hashed: timestamps are kept to the microsecond, IP addresses in canonical form, and a value too long for its column is cut with a visible marker (…[truncated from N characters]), so an oversized value never makes the write fail.

Every hour the gateway also writes an anchor to S3 Object Lock: the latest event’s chain index and hash, plus the SHA-256 of the previous anchor:

Anchor 1: last_seq 100, last_hash h(event 100) | prev_anchor_hash: none
Anchor 2: last_seq 250, last_hash h(event 250) | prev_anchor_hash: sha256(anchor 1)
Anchor 3: last_seq 400, last_hash h(event 400) | prev_anchor_hash: sha256(anchor 2)

Anchors live outside the database and can’t be changed or deleted during their retention period, so they pin the chain: rewriting any event up to an anchor would change the hash the anchor recorded.

Events written after the latest anchor aren’t pinned yet. Anyone with write access to the database can delete them without the secret (and with the secret, rewrite them) until the next anchor records them. Anchors are written hourly, and verification fails when the latest one is too old, which keeps that window to about an hour.

Failing Closed

A tamper-evident log only helps if actions can’t happen without an entry. Rack Gateway records every request before it acts, reads, sign-ins and MFA checks included, and refuses it with 503 when the audit log can’t take that entry. There are no exceptions: during an audit outage the gateway refuses everything and reports itself unhealthy. See When the Audit Log Can’t Be Written.

Verifying Integrity

rack-gateway-api verify-audit

Audit chain: checked 48210 events (0..48209), latest event 48209
  hash v2 from event 31877: 16333 events checked, HMAC recomputed
  legacy (v1) events: 31877 checked (up to event 31876), previous_hash links only: their content is unverifiable (a v1 event_hash can't be recomputed)
  hash v2 start: pinned at event 31877, and recorded by 1210 format 2 anchors
Anchors: 1460 checked, latest production/2025/03/02/13/anchor-20250302T13.json (written 2025-03-02T13:00:02Z)
Warnings (1):
  - events 48190..48209 (20) were written after the latest anchor and aren't anchored yet
Result: OK, except that the content of 31877 legacy (v1) events can't be verified

The command checks every link in the chain, recomputes the HMAC of every hash v2 event, checks every anchor and exits non-zero on any integrity failure. Events written before hash v2 (legacy v1 events) are checked for their links only: a v1 hash can’t be recomputed, so a change to a v1 event’s content (by someone able to get past the append-only trigger) wouldn’t be found, and the result says so. After upgrading from a version before hash v2, pin where hash v2 starts with AUDIT_HASH_V2_FROM. See Verification for its options, the permissions it needs, and what it can and can’t detect.

The gateway runs the same check every hour and alerts through Sentry and Slack when it fails; see Hourly verification in the gateway.

Log Retention

Determining Retention Period

Factor	Consideration
Compliance	SOC 2 typically requires 1 year
Legal	Potential litigation may require 7+ years
Investigation	Incidents may not be discovered for months
Cost	Storage costs increase with retention

Rack Gateway never deletes audit events: the table is append-only. Its S3 WORM anchors are locked for the bucket’s default Object Lock retention (400 days in the example Terraform).

Implementing Retention

# The anchor bucket's default retention locks every anchor
resource "aws_s3_bucket_object_lock_configuration" "audit_anchors" {
  bucket = aws_s3_bucket.audit_anchors.id
  rule {
    default_retention {
      mode = "COMPLIANCE"
      days = 400
    }
  }
}

The events themselves stay in Postgres; keep its backups as long as your policies require.

Caution

Once data is written to WORM storage with a retention period, it cannot be deleted until that period expires—even by administrators. Set retention carefully.

Audit Log Architecture

Rack Gateway Implementation

Flow:

1. Audit middleware writes an attempted entry, and refuses the request (503) if it can’t
2. Request processed by handler (or forwarded to the rack)
3. The result entry is written with the same request ID
4. An hourly job writes an anchor of the chain head to S3 WORM
5. An hourly job verifies the chain and anchors, alerting Sentry and Slack on failure

What Gets Logged

Event Type	Details Captured
Authentication	Login attempts, OAuth callbacks
Session	Session creation, expiration, logout
MFA	Enrollment, verification, removal
Convox API	Every proxied request and its outcome (status, not the response body)
Admin Actions	User management, role changes
API Tokens	Creation, deletion, usage
Settings	Configuration changes

What Doesn’t Get Logged

• Health check requests (/api/v1/health)
• Static asset requests
• Actual response bodies (only status codes)
• Sensitive field values (redacted)

Querying Audit Logs

Web UI

Access audit logs through the web interface:

• Filter by user, status, action type, resource type, text and time range
• Export to CSV for external analysis
• View each entry’s details

API

# Requests refused in the last 30 days (with an admin's session)
curl -b "session_token=$TOKEN" \
  "https://gateway.example.com/api/v1/audit-logs?status=denied&range=30d"

See Using Audit Logs for the filters. API tokens can’t read the audit log, and there’s no CLI command for it.

Verifying Integrity

# Check the hash chain and S3 anchors (on the gateway's admin service)
rack-gateway-api verify-audit

Best Practices

1. Log everything

   It’s better to have logs you don’t need than to need logs you don’t have

2. Redact automatically

   Don’t rely on developers to remember—use pattern-based redaction

3. Use structured logging

   JSON format enables querying and analysis

4. Include context

   Request IDs, session IDs, and correlation IDs help trace actions

5. Store immutably

   Use WORM storage to prevent tampering

6. Retain adequately

   Meet compliance requirements with margin for safety

7. Monitor the logs

   Logs are useless if no one watches them

8. Test your logging

   Verify logs capture what you need before an incident

Audit Logs and Compliance

SOC 2 Mapping

SOC 2 Criteria	Audit Log Support
CC6.1	All access logged with user attribution
CC7.1	Configuration changes logged
CC7.2	Security events captured
A1.2	Log integrity via WORM anchoring

Demonstrating Compliance

Auditors typically request:

1. Evidence of logging: Screenshot of log entries
2. Retention policy: Documentation of retention settings
3. Access controls: Who can view/modify logs
4. Integrity: Evidence of WORM storage configuration

Rack Gateway provides:

• Audit log viewer in web UI
• WORM bucket configuration documentation
• RBAC for log access (admin only)
• An hourly verification of the hash chain and its anchors, with its latest result on the Audit Logs page, and rack-gateway-api verify-audit to run it by hand

Key Takeaways

1. Audit logs are essential for security, compliance, and operations
2. Automatic redaction protects sensitive data in logs
3. Immutable storage prevents evidence tampering
4. Cryptographic chaining enables integrity verification
5. Retention policies must meet compliance requirements
6. Structured logs enable effective querying and analysis

Further Reading

• S3 WORM Storage - Configure immutable log storage
• Audit Trail - Compliance documentation
• SOC 2 Compliance - Meeting audit requirements