Skip to content

S3 WORM Storage

This guide covers setting up S3 Object Lock (WORM - Write Once Read Many) storage for tamper-evident audit log anchoring.

The gateway’s audit log uses a cryptographic chain (HMAC-SHA256) to make logs tamper-evident. However, this chain lives in PostgreSQL, which could be compromised. S3 Object Lock anchors solve this by:

  • External immutability - Checkpoints written to S3 with COMPLIANCE mode cannot be modified
  • Non-repudiation - Once written, anchors cannot be deleted (even by AWS account owner)
  • Independent verification - Auditors can read the anchors with read-only credentials and check the database against them
  • Regulatory compliance - Meets SOC 2, financial records, and audit requirements

The gateway writes at most one anchor per hour. A periodic job runs every AUDIT_ANCHOR_INTERVAL_MINUTES (default 60) and when the gateway starts; anchors are keyed by hour, so a run that finds this hour’s anchor already written does nothing. rack-gateway-api write-anchor writes one by hand.

Anchor writes hold a database advisory lock from the check for this hour’s anchor until the write, so two of them (on different gateway instances, or the job and write-anchor) never overlap: an overlap could write an hour’s key twice or link two anchors to the same predecessor, and Object Lock would keep that forever. A job that finds the lock taken tries again a minute later; write-anchor fails and can be run again. The lock lives on one of the gateway’s database connections, which the write holds for the length of its S3 calls (well under a second normally, at most the job’s one-minute timeout).

The upload itself is sent once. S3 can store an object and still fail the request (a 5xx, or a connection dropped before the response), and the AWS SDK’s automatic retry would then add a second version of the key. Instead the job fails, and its retry finds the anchor if S3 stored it.

A failing job is retried, but not past the hour’s write window: a retry that starts more than 6 hours after the job was created gives up (it’s logged as an error and sent to Sentry, if configured), and the next hours’ anchors cover the events. verify-audit fails an anchor written later than that.

If a job finds its hour’s key already holds an object, it checks the anchor writer could have written it: S3 stored it no earlier than the hour began, it is an anchor for this chain and hour, and it records every event written before S3 stored it. Otherwise someone with write access to the bucket put it there (created ahead of time, or a stale copy of an earlier anchor), and while Object Lock keeps it that hour can’t be anchored. The job is cancelled and the error logged and sent to Sentry, and write-anchor fails; verify-audit reports the object too.

A job’s hour comes from its creation time in the database. An attempt that started before the job was created means someone moved that time forward to name a later hour, so the job refuses it the same way. write-anchor names the anchor by the database’s clock, not the clock of the host running it.

Each anchor is a JSON blob containing chain state:

{
"chain_id": "production",
"last_seq": 8723412,
"last_hash": "b7e9f82a3c1d4e5f6a7b8c9d0e1f2a3b...",
"ts_utc": "2025-01-15T14:00:03Z",
"db_snapshot_lsn": "",
"prev_anchor_hash": "1d4f90e2b3c4d5e6f7a8b9c0d1e2f3a4...",
"format": 2,
"hash_v2_from_seq": 8100000
}
FieldPurpose
chain_idEnvironment identifier (AUDIT_ANCHOR_CHAIN_ID)
last_seqLatest event’s chain index (0 with an empty last_hash when there are no events)
last_hashLatest event’s hash (hex-encoded)
ts_utcWhen the anchor job was created; the key is derived from its hour
db_snapshot_lsnReserved, always empty
prev_anchor_hashSHA-256 of the anchor written before this one (creates the anchor chain)
format2 for anchors written by current versions; absent in older anchors
hash_v2_from_seqChain index of the first hash v2 event, once there is one

Older anchors (no format) have a known bug: once the bucket held more than 1000 anchors, the writer only saw the first 1000, so their prev_anchor_hash stopped pointing at the previous anchor. Verification counts these instead of failing.

s3://bucket/chain_id/YYYY/MM/DD/HH/anchor-YYYYMMDDTHH.json

Example:

s3://audit-anchors-prod/production/2025/01/15/14/anchor-20250115T14.json

Object Lock buckets are always versioned. Writing an anchor’s key again doesn’t replace the locked version, it adds another one, and deleting it only adds a delete marker. Verification lists every version, so either shows up.

S3 Object Lock is extremely cost-effective:

ItemCalculationAnnual Cost
Storage (steady state)24 anchors/day × 400 days × 2 KB~$0.01
PUT requests24 × 365 = 8,760 @ $0.005/1k~$0.04
KMS encryption8,760 requests @ $0.03/10k~$0.03
GET requests (verification)8,760 @ $0.0004/1k~$0.00
Total~$0.08/year
resource "aws_s3_bucket" "audit_anchors" {
bucket = "audit-anchors-${var.environment}-${random_id.bucket_suffix.hex}"
# Object Lock must be enabled at creation
object_lock_enabled = true
tags = merge(var.tags, {
Name = "audit-anchors-${var.environment}"
Purpose = "WORM audit log anchoring"
})
}
resource "random_id" "bucket_suffix" {
byte_length = 4
}
resource "aws_s3_bucket_object_lock_configuration" "audit_anchors" {
bucket = aws_s3_bucket.audit_anchors.id
rule {
default_retention {
mode = "COMPLIANCE"
days = var.worm_retention_days # e.g., 400
}
}
}

COMPLIANCE mode means:

  • Objects cannot be deleted or modified by anyone
  • Even AWS account root cannot override
  • Only way to delete is wait for retention to expire
resource "aws_s3_bucket_versioning" "audit_anchors" {
bucket = aws_s3_bucket.audit_anchors.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_server_side_encryption_configuration" "audit_anchors" {
bucket = aws_s3_bucket.audit_anchors.id
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "aws:kms"
kms_master_key_id = aws_kms_key.audit_anchors.arn
}
bucket_key_enabled = true
}
}
resource "aws_kms_key" "audit_anchors" {
description = "Audit anchor encryption - ${var.environment}"
deletion_window_in_days = 30
enable_key_rotation = true
tags = var.tags
}
resource "aws_s3_bucket_public_access_block" "audit_anchors" {
bucket = aws_s3_bucket.audit_anchors.id
block_public_acls = true
block_public_policy = true
ignore_public_acls = true
restrict_public_buckets = true
}
resource "aws_s3_bucket_policy" "audit_anchors" {
bucket = aws_s3_bucket.audit_anchors.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "DenyUnencryptedUploads"
Effect = "Deny"
Principal = "*"
Action = "s3:PutObject"
Resource = "${aws_s3_bucket.audit_anchors.arn}/*"
Condition = {
StringNotEquals = {
"s3:x-amz-server-side-encryption" = "aws:kms"
}
}
},
{
Sid = "DenyDelete"
Effect = "Deny"
Principal = "*"
Action = [
"s3:DeleteObject",
"s3:DeleteObjectVersion"
]
Resource = "${aws_s3_bucket.audit_anchors.arn}/*"
}
]
})
}

Don’t add a statement that denies uploads without an x-amz-object-lock-mode header. The gateway doesn’t send Object Lock headers: the bucket’s default retention (above) locks every new object version in COMPLIANCE mode for worm_retention_days, so a policy requiring the header would deny every anchor write. Uploads to an Object Lock bucket must carry an integrity checksum; the gateway sends a SHA-256 checksum with each anchor.

The gateway writes the anchors and verifies them every hour, which reads every version of each anchor.

resource "aws_iam_role_policy" "anchor_writer" {
name = "audit-anchor-writer"
role = aws_iam_role.gateway.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = [
"s3:PutObject",
"s3:PutObjectRetention",
"s3:GetObject",
"s3:GetObjectVersion",
"s3:ListBucket",
"s3:ListBucketVersions",
# The gateway checks the bucket's default retention at startup and in every verification
"s3:GetBucketObjectLockConfiguration"
]
Resource = [
aws_s3_bucket.audit_anchors.arn,
"${aws_s3_bucket.audit_anchors.arn}/*"
]
},
{
Effect = "Allow"
Action = [
"kms:Encrypt",
"kms:Decrypt",
"kms:GenerateDataKey"
]
Resource = [aws_kms_key.audit_anchors.arn]
}
]
})
}

s3:GetObject also covers the HeadObject check for an existing anchor.

resource "aws_iam_role" "anchor_verifier" {
name = "audit-anchor-verifier-${var.environment}"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
AWS = var.auditor_account_ids
}
}
]
})
tags = var.tags
}
resource "aws_iam_role_policy" "anchor_verifier" {
name = "s3-read-only"
role = aws_iam_role.anchor_verifier.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = [
"s3:GetObject",
"s3:GetObjectVersion",
"s3:ListBucket",
"s3:ListBucketVersions",
"s3:GetObjectRetention",
"s3:GetBucketObjectLockConfiguration"
]
Resource = [
aws_s3_bucket.audit_anchors.arn,
"${aws_s3_bucket.audit_anchors.arn}/*"
]
},
{
Effect = "Allow"
Action = ["kms:Decrypt"]
Resource = [aws_kms_key.audit_anchors.arn]
}
]
})
}

Enable anchoring with these environment variables:

Terminal window
# Required
AUDIT_ANCHOR_BUCKET=audit-anchors-prod-abc123
AUDIT_ANCHOR_CHAIN_ID=production
# Optional
AUDIT_ANCHOR_INTERVAL_MINUTES=60 # Default: 60
# AWS credentials (if not using IAM roles)
AWS_ACCESS_KEY_ID=xxx
AWS_SECRET_ACCESS_KEY=xxx
AWS_REGION=us-east-1

rack-gateway-api verify-audit checks the audit chain in the database and every anchor in the bucket. It only reads. The gateway also runs these checks itself every hour and alerts when they fail (see Hourly Verification in the Gateway).

Terminal window
rack-gateway-api verify-audit [--from N] [--to N] [--hash-v2-from N] [--max-anchor-age 3h] [--no-anchors]
OptionMeaning
--from NFirst chain index to check (default 0). The event before it is still read to check the link.
--to NLast chain index to check (default: the latest event)
--hash-v2-from NWhere hash v2 starts, recorded after upgrading (default: AUDIT_HASH_V2_FROM). See Pinning where hash v2 starts.
--max-anchor-age DFail when the latest anchor was written longer ago than this (default 3h, or three anchor intervals when AUDIT_ANCHOR_INTERVAL_MINUTES is longer than an hour)
--no-anchorsSkip the S3 checks and only check the database

It prints a summary and exits non-zero if it finds any integrity failure. Warnings don’t change the exit code.

  • AUDIT_HMAC_SECRET: the secret the gateway writes events with. There’s no fallback; without it the command fails.
  • RGW_DATABASE_URL, GATEWAY_DATABASE_URL or DATABASE_URL, for a role with SELECT on audit.audit_event (the audit_reader role is enough).
  • Unless --no-anchors is given: AUDIT_ANCHOR_BUCKET, AUDIT_ANCHOR_CHAIN_ID and AWS credentials allowed s3:ListBucket, s3:ListBucketVersions, s3:GetObject, s3:GetObjectVersion and s3:GetBucketObjectLockConfiguration on the bucket, and kms:Decrypt on its key (the verifier role above). It fails unless the bucket’s default retention is COMPLIANCE mode for at least AUDIT_ANCHOR_RETENTION_DAYS (default 400).
  • On a gateway that was upgraded from a version before hash v2: AUDIT_HASH_V2_FROM (or --hash-v2-from), see below.

The anchor checks read every anchor, one request per hour of history (16 at a time), so a full run gets slower as the bucket grows: about a minute for a year of anchors. --from only limits the database checks.

Events, in chain order:

  • The chain starts with the genesis event (index 0, no previous hash).
  • Each event’s previous_hash is the hash of the event before it.
  • Hash v2 events have consecutive indexes, and their HMAC is recomputed from the stored row and compared with event_hash.
  • Legacy (v1) events only come before hash v2 starts, and something outside the database confirms where it starts (a pinned start or a format 2 anchor; see below).
  • On a full run (no --from or --to): the number of events in the table matches the number checked, so no event hides outside the chain (a NULL or negative chain index, or two events at one index).

Anchors, in the order they were written:

  • Each anchor key has exactly one version and no delete marker.
  • chain_id matches, and the key matches ts_utc.
  • S3 stored the anchor (its LastModified) no earlier than its ts_utc and no more than 6 hours (the write window) after it, give or take 5 minutes of clock skew. An anchor stored earlier was created ahead of time by someone with write access to the bucket, and stops the gateway anchoring that hour; one stored later was backfilled. Anchors without a format written late only get a warning: earlier versions retried a failing job for weeks.
  • An anchor without a format was stored no more than 24 hours after the first format 2 anchor (an instance of the earlier version can still write one during the rolling deploy). Later ones weren’t written by the anchor writer, and the allowances for earlier versions’ anchors mustn’t excuse them.
  • The event at last_seq exists and its hash equals last_hash.
  • last_seq never goes down (events removed from the end of the chain).
  • prev_anchor_hash is the SHA-256 of the anchor written before it (failure for format 2 anchors; older anchors are counted, see above).
  • Every format 2 anchor agrees with the database on where hash v2 starts: hash_v2_from_seq once there are hash v2 events, or, in an anchor written before the first one, the event after last_seq. This catches hash v2 events relabeled as legacy events to skip their HMAC check.
  • There is at least one anchor once there are events, and the latest anchor was written within --max-anchor-age. Either failing means anchors aren’t being written, so nothing outside the database protects new events.

It warns about hours without an anchor and about events written after the latest anchor. Unanchored events fail verification once the oldest was written more than --max-anchor-age ago: anchors are being written but don’t cover new events.

Events written before hash v2 (by versions before the hash v2 migration) are legacy (v1) events: only their links are checked. Which events are legacy is recorded in the database itself, so someone with write access to the database (no secret needed) could relabel hash v2 events as legacy and then edit them. To catch that, the start of hash v2 has to be confirmed outside the database, by either:

  • A pinned start. Right after upgrading, run verify-audit: it prints hash v2 from event N (before the new version has written an event, its failure names the event that will be the first). Record N and set AUDIT_HASH_V2_FROM=N wherever verify-audit runs, or pass --hash-v2-from N. A different start in the database is a failure, and so is every legacy event from N on. So is a database with no hash v2 events at all, which is what removing every one of them from the end leaves: with a pin, verification fails until the new version has written its first event (any request does).
  • A format 2 anchor. Every anchor written since the upgrade pins where hash v2 starts. Anchors are written hourly, so this is available about an hour after upgrading, but not with --no-anchors.

Pin the start even when anchors are configured: it also covers --no-anchors runs and doesn’t depend on the bucket. While legacy events exist and neither confirms the start, verify-audit fails. A gateway that never ran a version before hash v2 has no legacy events and needs no pin.

  • Events that were never written, for example when an audit write failed.
  • Events deleted from the end of the chain before an anchor records them. This needs only write access to the database, not the secret: the events that are left still form a valid chain. Anchors bound the window to about an hour, and verify-audit fails when the latest anchor is older than --max-anchor-age.
  • Changes after the latest anchor by someone who has AUDIT_HMAC_SECRET. With the secret and write access to the database, they can rewrite the unanchored tail and recompute every hash. Only history up to the latest anchor is protected from them, so keep the secret away from people who administer the database.
  • Forgery by anyone who can verify. The HMAC is symmetric: whoever holds the secret to verify can also compute valid hashes. Treat every copy of the secret (including where verify-audit runs) as able to write the chain.
  • Edits to legacy (v1) events. Events written before hash v2 can’t have their HMAC recomputed from the stored row: it covered values the database doesn’t keep exactly (nanosecond timestamps, some fields before they were normalized for storage, and a chain index that could differ from the stored one). Only their previous_hash links are checked. An anchor still pins the hash of the event it recorded, but not the content of legacy events.
  • Hash v2 events relabeled as legacy, without a pinned start or format 2 anchors. Nothing outside the database then says which events are hash v2. verify-audit reports this as a failure rather than passing, but it can’t tell whether anything was relabeled.

The gateway runs these checks itself at half past every hour (UTC), with the configuration verify-audit reads: AUDIT_HMAC_SECRET, AUDIT_HASH_V2_FROM and the anchor settings. Its role needs s3:ListBucketVersions, s3:GetObjectVersion and s3:GetBucketObjectLockConfiguration on the bucket (IAM Role for the Gateway); without them every run fails and alerts. The gateway also reads the bucket’s Object Lock configuration before it starts, and refuses to start unless the default retention is COMPLIANCE mode for at least AUDIT_ANCHOR_RETENTION_DAYS.

  • What a run checks. The first run after the gateway starts, every run while verification is failing, and one run a day check every event. The hourly runs in between check the events written since the last passing run, every anchor against the database, and that the last event that run checked is still there, unchanged, which catches events removed from the end even after new ones take their place. A change to an earlier event is found by the next daily check unless an anchor records that event. Anchor versions can’t change, so the gateway fetches each one once and lists the bucket on every run.
  • What it remembers. How far it has checked and what it has alerted about stay in the gateway’s memory, never in the database, so someone who can write to the database can’t use them to skip checks or silence alerts. A restart starts over: the next run checks every event, and alerts again if verification is still failing.
  • Without anchors (no AUDIT_ANCHOR_BUCKET), it checks the hash chain only and says so in its log. Without AUDIT_HMAC_SECRET, which only DEV_MODE allows, it doesn’t run.
  • Alerts. Every failing run (an integrity failure, or a run that couldn’t read the database or the bucket) is sent to Sentry, where runs that keep failing group into one issue. The gateway also records an audit_log.verification_failed audit event with status failed or error and the first failures, which goes to every Slack channel whose action patterns match it (the default #security channel’s audit_log.verification_* does): on the first failing run, when a run finds a failure the last alert didn’t report (even with the same number of failures), and once a day while it keeps failing. The first passing run after that records audit_log.verification_recovered. An alert whose Slack notification can’t be queued is sent again by the next run, recovery included. A run that stops partway (an anchor it can’t read, say) still reports the failures it found before it stopped. A passing run otherwise records nothing, and warnings don’t alert. Findings that can’t be fixed, such as a locked anchor that an earlier version wrote twice, keep verification failing until the anchor’s retention ends. To be alerted when runs stop, set AUDIT_VERIFY_HEARTBEAT_URL to a heartbeat monitor’s URL; the gateway requests it after every run (see Heartbeats).
  • Results. Each run logs a line saying what it checked and found, and the Audit Logs page shows the latest run: when it ran, its result, how long it has been failing and the full report. The page reads that from the database, so treat the alerts and the log as the record.

Run verify-audit to look into a failure. For a check that doesn’t depend on the gateway’s own credentials, also run it on a schedule with the read-only verifier role and a database role that can only read the audit schema.

For disaster recovery, replicate anchors to another region:

# Destination bucket (in different region)
resource "aws_s3_bucket" "audit_anchors_dr" {
provider = aws.dr_region
bucket = "audit-anchors-dr-${var.environment}-${random_id.bucket_suffix.hex}"
object_lock_enabled = true
tags = var.tags
}
# Replication configuration
resource "aws_s3_bucket_replication_configuration" "audit_anchors" {
bucket = aws_s3_bucket.audit_anchors.id
role = aws_iam_role.replication.arn
rule {
id = "replicate-anchors"
status = "Enabled"
destination {
bucket = aws_s3_bucket.audit_anchors_dr.arn
storage_class = "STANDARD"
replication_time {
status = "Enabled"
time {
minutes = 15
}
}
metrics {
status = "Enabled"
event_threshold {
minutes = 15
}
}
}
delete_marker_replication {
status = "Disabled"
}
}
}

Transition old anchors to cheaper storage:

resource "aws_s3_bucket_lifecycle_configuration" "audit_anchors" {
bucket = aws_s3_bucket.audit_anchors.id
rule {
id = "archive-old-anchors"
status = "Enabled"
transition {
days = 30
storage_class = "GLACIER_IR" # Instant retrieval
}
}
}
# Alert if no anchor written in 2 hours
resource "aws_cloudwatch_metric_alarm" "missing_anchor" {
alarm_name = "audit-anchor-missing-${var.environment}"
comparison_operator = "LessThanThreshold"
evaluation_periods = 1
metric_name = "NumberOfObjects"
namespace = "AWS/S3"
period = 7200 # 2 hours
statistic = "Sum"
threshold = 1
dimensions = {
BucketName = aws_s3_bucket.audit_anchors.id
StorageType = "AllStorageTypes"
}
alarm_actions = var.alarm_sns_topic_arns
}
resource "aws_s3_bucket_logging" "audit_anchors" {
bucket = aws_s3_bucket.audit_anchors.id
target_bucket = aws_s3_bucket.access_logs.id
target_prefix = "audit-anchors/"
}

In COMPLIANCE mode, you cannot delete objects before retention expires. Options:

  1. Wait for retention to end - Retention only prevents deletion; nothing is deleted when it ends, and anchors stay in the bucket. Deleting them afterwards is a deliberate step: delete the old versions themselves (not with a delete marker, which verify-audit reports as a failure). The oldest anchor left then points at one that’s gone, which verify-audit only warns about.
  2. Delete entire bucket - Requires AWS Support ticket
  3. Legal Hold - Mark objects for investigation without deleting
Terminal window
# Apply legal hold (does not affect retention)
aws s3api put-object-legal-hold \
--bucket audit-anchors-prod \
--key "production/2025/01/15/14/anchor-20250115T14.json" \
--legal-hold Status=ON

Provide read-only credentials to external auditors:

Terminal window
# Create temporary credentials
aws sts assume-role \
--role-arn arn:aws:iam::123456789012:role/audit-anchor-verifier-production \
--role-session-name external-auditor \
--duration-seconds 43200

S3 Object Lock anchors provide evidence for:

FrameworkRequirementEvidence
SOC 2CC6.1 Logical AccessImmutable audit trail
SOC 2CC7.2 System MonitoringContinuous verification
SOC 2CC7.3 Incident ManagementTamper-evident logs
SOXFinancial Records7-year retention available
GDPRRight to ErasureLegal exceptions documented
variable "environment" {
description = "Environment name"
type = string
}
variable "worm_retention_days" {
description = "Object Lock retention period in days"
type = number
default = 400 # ~13 months (covers annual audit + buffer)
}
variable "enable_replication" {
description = "Enable cross-region replication"
type = bool
default = false
}
variable "dr_region" {
description = "DR region for replication"
type = string
default = "us-west-2"
}
variable "auditor_account_ids" {
description = "AWS account IDs allowed to assume verifier role"
type = list(string)
default = []
}
output "audit_anchor_bucket" {
description = "S3 bucket name for audit anchors"
value = aws_s3_bucket.audit_anchors.id
}
output "audit_anchor_bucket_arn" {
description = "S3 bucket ARN"
value = aws_s3_bucket.audit_anchors.arn
}
output "kms_key_arn" {
description = "KMS key ARN for anchor encryption"
value = aws_kms_key.audit_anchors.arn
}
output "verifier_role_arn" {
description = "IAM role ARN for external verification"
value = aws_iam_role.anchor_verifier.arn
}