Cache System

Cigen can inject restore/save cache steps when a job declares cache:. Cache definitions are extensible via YAML; key calculation is simple by default and can be customized.

Cache Key Structure

By default, injected keys include the cache name and the selected architecture, plus a checksum expression. You can override keys via cache_definitions if you need a specific pattern.

Built-in Cache Types

Cigen ships default definitions for common technologies that you can extend or override. These drive detection and provide sensible paths; key templates can be provided if desired.

Ruby Gems

Detects Ruby & Bundler versions, uses Gemfile.lock for checksums

Node Modules

Supports npm, yarn, pnpm, and bun with appropriate lock files

Python Packages

Handles pip, pipenv, and poetry with virtual environment detection

Go Modules

Uses go.mod and go.sum for dependency tracking

Ruby Gems

cache_definitions:
gems:
  versions: [ruby, bundler]
  checksum_sources: [Gemfile, Gemfile.lock]
  paths: [vendor/bundle, .bundle]

Node.js Modules

cache_definitions:
node_modules:
  versions: [node]
  checksum_sources:
    - package.json
    - detect: [package-lock.json, yarn.lock, bun.lockb, pnpm-lock.yaml]
  paths: [node_modules]

Python Packages

cache_definitions:
pip:
  versions: [python, pip]
  checksum_sources:
    - detect: [requirements.txt, Pipfile]
    - detect_optional: [requirements.lock, Pipfile.lock]
  paths:
    - detect: [.venv, venv]
    - ~/.cache/pip

Using Caches in Jobs

Cache Field vs Restore Cache

The cache and restore_cache fields serve different purposes:

cache: restores before steps and saves after steps for the named caches
restore_cache: restores only (read-only)

# Read and write cache
jobs:
  build:
    cache: gems  # Restores before steps, saves after steps

# Read-only cache
jobs:
  test:
    restore_cache: gems  # Only restores, doesn't save

The restore_cache field is a shorthand for:

cache:
  gems:
    restore: true # Always true by default anyway
    save: false # This is what restore_cache does

Simple Cache Usage

jobs:
test:
  image: cimg/ruby:3.3
  cache: gems  # Uses all defaults from gems cache definition
  steps:
    - run: bundle install
    - run: bundle exec rspec

This injects a restore_cache step before, and a save_cache step after, when the cache is declared on the job.

Multiple Caches

jobs:
full_stack_test:
  image: cimg/ruby:3.3
  cache: [gems, node_modules]  # Multiple cache types
  services: [postgres]
  steps:
    - run: bundle install
    - run: npm install
    - run: bundle exec rspec

Custom Cache Paths

jobs:
test:
  image: cimg/ruby:3.3
  cache:
    gems:
      paths: [vendor/rubygems, .bundle]  # Override paths
      # versions and checksum_sources from definition
  steps:
    - run: bundle install --path vendor/rubygems

Advanced Cache Configuration

Custom Cache Types

# In config.yml
cache_definitions:
ml_models:
  versions: [python]
  checksum_sources:
    - requirements-ml.txt
    - models/config.json
  paths:
    - models/cache
    - ~/.cache/torch

webpack*assets:
checksum_sources: # No versions - pure checksum cache - package.json - webpack.config.js - src/**/_.js
paths: - dist/assets - .webpack-cache

Cache Without Versions

For caches that don’t depend on runtime versions:

cache_definitions:
static_assets:
  checksum_sources: [assets/**/*.css, assets/**/*.js]
  paths: [public/compiled-assets]

Example key: linux-ubuntu22.04-amd64-static_assets-abc123def456

Path Detection

paths:
- node_modules  # Required - job fails if missing
- detect:       # At least one must exist
    - .venv
    - venv
    - virtualenv
- detect_optional:  # All are optional
    - .cache/pre-commit
    - ~/.cache/myapp

Manual Cache Steps

For fine-grained control, use manual cache steps:

steps:
- restore_cache:
    name: Restore webpack cache
    key: webpack-{{ checksum "webpack.config.js" }}

- name: Build assets
run: npm run build

- save_cache:
name: Save webpack cache
key: webpack-{{ checksum "webpack.config.js" }}
paths: [.webpack-cache, dist/]

These steps:

Use identical syntax across all CI providers
Automatically use the configured cache backend
Support template variables and functions

Cache Backends

Configure different storage backends for various scenarios:

cache_backends:
default: native  # Use CI provider's native cache

# Backend configurations

s3:
bucket: my-cache-bucket
region: us-east-1
prefix: cigen-cache/

redis:
url: redis://cache.example.com:6379
ttl: 604800 # 7 days

minio:
endpoint: minio.internal:9000
bucket: ci-cache
access_key: ${MINIO_ACCESS_KEY}
secret_key: ${MINIO_SECRET_KEY}

# Per-cache backend override

cache_definitions:
ml_models:
backend: s3 # Large models go to S3
paths: [models/]

Version Detection

Detection Priority

Version sources are checked in order until one succeeds:

Version files - Read directly from committed files
```
- file: .ruby-version # Contains: 3.3.0
```

Tool version files - Extract using patterns

- file: .tool-versions
  pattern: 'ruby (.+)' # From: ruby 3.3.0

Runtime commands - Query installed versions

- command: 'ruby --version' # Auto-parses version numbers

Raw commands - Use output directly

- command: "grep -A1 'BUNDLED WITH' Gemfile.lock | tail -n1"
  parse_version: false # Use raw output

Multi-Version Keys

When multiple versions are detected:

linux-ubuntu22.04-amd64-fullstack-ruby3.3.0-node20.11.0-abc123def

Multi-Architecture Support

Cache keys automatically include architecture information:

jobs:
build:
  architectures: [amd64, arm64]
  cache: gems

# Generates separate cache keys:

# linux-ubuntu22.04-amd64-gems-ruby3.3.0-abc123def

# linux-ubuntu22.04-arm64-gems-ruby3.3.0-abc123def

Best Practices

Use Built-in Types

Leverage built-in cache definitions when possible - they handle edge cases and evolve with best practices

Include Lock Files

Always include both manifest files (package.json) and lock files (package-lock.json) in checksum sources

Version Files in Git

Commit version files (.ruby-version, .node-version) to ensure consistent cache keys across environments

Descriptive Names

Use clear, descriptive cache names (gems, node_modules) rather than generic names (cache1, deps)

Troubleshooting

Cache Misses

If caches aren’t restoring when expected:

Check version files exist and are committed to git
Verify checksum sources haven’t changed unexpectedly
Use debug mode to inspect generated cache keys:
Terminal window
```
cigen generate --verbose
```

Cache Conflicts

For jobs needing different cache strategies:

# Different cache strategies for same dependency type
cache_definitions:
gems_production:
  type: gems  # Inherit from gems
  paths: [vendor/production-gems]

gems_development:
type: gems
paths: [vendor/development-gems, .bundle]

Implementation Details

Cache Key Generation Process

Detect platform (OS, version, architecture)
Resolve versions from configured sources
Calculate checksums of dependency files
Combine components into final cache key
Inject restore/save steps at appropriate points

Backend Selection

Cigen chooses cache backends based on:

Global cache backend configuration
Per-cache backend overrides
Runner type (cloud vs self-hosted)
Environment variables and context

The same cache configuration works across all backends - cigen handles the implementation details.