Package Management

The packages feature provides intelligent package manager detection, installation, and optimization that builds on top of cigen’s cache system. It automatically detects which package manager your project uses and generates the correct installation commands.

Architecture Overview

Note

Packages builds on Cache: The package system leverages the existing cache system for storage and retrieval, while adding package manager detection and installation logic on top.

The package management system provides:

1. Dynamic package manager detection - Detects npm vs yarn vs pnpm based on lock files
2. Configurable installation commands - YAML-based definitions for any package manager
3. Smart job deduplication - Optimizes workflows when multiple jobs use the same packages
4. Extensible definitions - Built-in support for common languages, fully customizable

Package Manager Definitions

Package managers are defined in YAML configuration with detection rules and installation commands. Cigen includes built-in definitions that you can extend or override.

Built-in Package Managers

Built-in package manager definitions

# Built into cigen's default configuration
package_managers:
node:
  versions: [node]
  detect:
    - npm:
        lockfile: package-lock.json
        command: npm ci
    - yarn:
        lockfile: yarn.lock
        command: yarn install --frozen-lockfile
    - pnpm:
        lockfile: pnpm-lock.yaml
        command: pnpm install --frozen-lockfile
    - bun:
        lockfile: bun.lockb
        command: bun install --frozen-lockfile
  checksum_sources:
    - package.json
    - detect: [package-lock.json, yarn.lock, pnpm-lock.yaml, bun.lockb]
  cache_paths: [node_modules]

ruby:
  versions: [ruby, bundler]
  detect:
    - bundler:
        lockfile: Gemfile.lock
        command: bundle install
  checksum_sources: [Gemfile, Gemfile.lock]
  cache_paths: [vendor/bundle, .bundle]

python:
  versions: [python]
  detect:
    - pip:
        lockfile: requirements.txt
        command: pip install -r requirements.txt
    - pipenv:
        lockfile: Pipfile.lock
        command: pipenv install --deploy
    - poetry:
        lockfile: poetry.lock
        command: poetry install
  checksum_sources:
    - detect: [requirements.txt, Pipfile, pyproject.toml]
    - detect_optional: [requirements.lock, Pipfile.lock, poetry.lock]
  cache_paths:
    - detect: [.venv, venv]
    - ~/.cache/pip

How Detection Works

When you use packages: node, cigen:

1. Detects the specific tool - Checks for package-lock.json (npm), yarn.lock (yarn), pnpm-lock.yaml (pnpm), or bun.lockb (bun)
2. Generates the install command - Uses the appropriate command (npm ci, yarn install --frozen-lockfile, etc.)
3. Creates cache configuration - Leverages the cache system with proper version detection and checksums

Basic Usage

Simple Package Installation

Simple package usage

jobs:
test:
  image: cimg/node:18.0
  packages: node  # Auto-detects npm/yarn/pnpm/bun
  steps:
    - run: npm test

This does the following:

1. Detects which package manager based on lock files
2. Restores the appropriate cache if declared on the job
3. Runs the detected install command (e.g., yarn install --frozen-lockfile)

Note: saving caches occurs when the job declares cache:. If you only declare packages: without a cache: entry, cigen will generate install steps but will not automatically add a save step.

Multiple Package Types

Multiple package types

jobs:
full_stack_test:
  image: cimg/ruby:3.3
  packages: [ruby, node]  # Multiple package managers
  services: [postgres]
  steps:
    - run: bundle exec rspec
    - run: npm test

Smart Job Deduplication

Note

Automatic Optimization: Cigen detects when multiple jobs use the same packages and automatically creates a shared installation job.

Single Job (Inline Installation)

When only one job uses a package manager, installation runs inline:

Single job - inline installation

jobs:
test:
  packages: node
  steps:
    - run: npm test

# Generated structure:
jobs:
test:
  cache: node_modules  # Uses cache system with detected package manager
  steps:
    - run: yarn install --frozen-lockfile  # Detected command
    - run: npm test

Multiple Jobs (Dedicated Installation Job)

When multiple jobs use the same packages, cigen can create a dedicated job with installation steps. To enable caching in that job, declare the relevant caches in your config.

Multiple jobs - dedicated installation

jobs:
test:
  packages: node
  steps:
    - run: npm test

lint:
  packages: node
  steps:
    - run: npm run lint

build:
  packages: node
  steps:
    - run: npm run build

# Generated structure:
jobs:
install_node_packages:
  steps:
    - run: yarn install --frozen-lockfile  # Detected based on yarn.lock

# Tip: add caching to the install job in your config if desired
# cache: node_modules

test:
  requires: [install_node_packages]
  restore_cache: node_modules  # Read-only cache access (when install job saves it)
  steps:
    - run: npm test

lint:
  requires: [install_node_packages]
  restore_cache: node_modules
  steps:
    - run: npm run lint

build:
  requires: [install_node_packages]
  restore_cache: node_modules
  steps:
    - run: npm run build

Benefits of Deduplication

The automatic job deduplication provides:

Faster CI

Install dependencies once, use everywhere

Better Parallelization

Installation runs in parallel with other setup tasks

Reduced Redundancy

No duplicate installation commands across jobs

Cache Efficiency

Single cache save operation reduces overhead

Package Detection

Lock File Detection

Cigen automatically detects which package manager to use:

1. Checks for lock files in the repository root
2. Identifies package manager from lock file type:
   • package-lock.json → npm
   • yarn.lock → yarn
   • pnpm-lock.yaml → pnpm
   • bun.lockb → bun
   • Gemfile.lock → bundler
   • requirements.lock or Pipfile.lock → pip/pipenv
3. Runs appropriate install command
4. Caches correct directories for that package manager

Install Commands

Based on detection, cigen runs the appropriate command:

Package Type	Lock File	Install Command
node (npm)	package-lock.json	npm ci
node (yarn)	yarn.lock	yarn install --frozen-lockfile
node (pnpm)	pnpm-lock.yaml	pnpm install --frozen-lockfile
node (bun)	bun.lockb	bun install --frozen-lockfile
ruby	Gemfile.lock	bundle install
python	requirements.txt	pip install -r requirements.txt
python	Pipfile.lock	pipenv install --deploy
go	go.mod	go mod download

How Default Configuration Works

Cigen includes production-ready package manager definitions that are compiled into the binary. These defaults are organized as separate YAML files for maintainability:

src/packages/config_templates/
├── package_managers/
│   ├── node.yml
│   ├── ruby.yml
│   ├── python.yml
│   ├── go.yml
│   ├── rust.yml
│   ├── java.yml
│   └── dotnet.yml
└── version_sources/
    ├── node.yml
    ├── ruby.yml
    ├── bundler.yml
    ├── python.yml
    ├── go.yml
    ├── rustc.yml
    ├── cargo.yml
    ├── java.yml
    └── dotnet.yml

When you define package_managers or version_sources in your .cigen/config.yml, your configuration intelligently merges with these defaults:

• Override: Completely replace a built-in definition
• Extend: Add new package managers alongside built-ins
• Inherit: Use built-in defaults when no custom config is provided

Customizing Package Managers

Overriding Built-in Definitions

You can customize the built-in package manager definitions in your configuration:

Custom package manager commands

# .cigen/config.yml
package_managers:
node:
  # Override the npm command
  detect:
    - npm:
        lockfile: package-lock.json
        command: npm ci --production  # Custom flags
    - yarn:
        lockfile: yarn.lock
        command: yarn install --frozen-lockfile --production
  # Keep other settings from built-in definition
  versions: [node]
  checksum_sources:
    - package.json
    - detect: [package-lock.json, yarn.lock, pnpm-lock.yaml, bun.lockb]
  cache_paths: [node_modules]

Customizing Version Sources

You can also customize how versions are detected for cache keys:

Custom version sources

# .cigen/config.yml
version_sources:
node:
  - file: .nvmrc
  - file: .node-version
  - file: package.json
    pattern: '"engines":\s*{\s*"node":\s*"([^"]+)"'
  - command: node --version
    parse_version: true

# Add custom version source
custom_tool:
  - file: .custom-version
  - command: custom-tool --version
    parse_version: false  # Don't parse, use raw output

Adding New Package Managers

Define completely new package managers:

Adding new package managers

# .cigen/config.yml
package_managers:
deno:
  versions: [deno]
  detect:
    - deno:
        lockfile: deno.lock
        command: deno cache deps.ts
  checksum_sources: [deps.ts, deno.lock]
  cache_paths: [~/.cache/deno]

swift:
  versions: [swift]
  detect:
    - spm:
        lockfile: Package.resolved
        command: swift package resolve
  checksum_sources: [Package.swift, Package.resolved]
  cache_paths: [.build]

Per-Job Customization

Override package behavior for specific jobs:

Per-job package customization

jobs:
test:
  packages:
    node:
      command: npm ci --ignore-optional  # Job-specific command
      cache_paths: [node_modules, .npm]   # Additional cache paths
  steps:
    - run: npm test

Packages vs Cache

Understanding when to use each:

Use packages when:	Use cache when:
Installing dependencies	Caching build artifacts
Standard package managers	Custom cache scenarios
Want automatic commands	Need fine control
Multiple jobs share deps	Single job optimization

Example: Packages for Dependencies, Cache for Artifacts

Combined packages and cache usage

jobs:
build:
  packages: [node]  # Install dependencies
  cache:
    webpack:  # Cache build artifacts
      paths: [dist/, .webpack-cache]
      checksum_sources: [webpack.config.js]
  steps:
    - run: npm run build

Migration Guide

From Manual Installation

Before (manual):

jobs:
  test:
    steps:
      - checkout
      - restore_cache: node_modules
      - run: npm install
      - save_cache: node_modules
      - run: npm test

After (with packages):

jobs:
  test:
    packages: [node]
    steps:
      - run: npm test

From Cache Field

Before (cache only):

jobs:
  test:
    cache: node_modules
    steps:
      - run: npm install  # Still needed
      - run: npm test

After (with packages):

jobs:
  test:
    packages: [node]  # Includes installation
    steps:
      - run: npm test

Best Practices

Use for Dependencies

Use packages for dependency installation, cache for build artifacts

Let Detection Work

Trust automatic lock file detection rather than specifying commands

Share When Possible

Let deduplication optimize when multiple jobs use the same packages

Version Lock Files

Always commit lock files to ensure reproducible builds

Troubleshooting

Package Manager Not Detected

If the wrong package manager is used:

1. Ensure lock file exists and is committed
2. Check detection order in the built-in definitions - earlier rules take precedence
3. Override detection rules if needed:

   package_managers:
     node:
       detect:
         - name: pnpm  # Force pnpm to be checked first
           lockfile: pnpm-lock.yaml
           command: pnpm install --frozen-lockfile
         - name: npm
           lockfile: package-lock.json
           command: npm ci

Custom Package Manager Not Working

If your custom package manager configuration isn’t being used:

1. Check YAML syntax - invalid YAML will be ignored
2. Verify field names match the schema (detect, checksum_sources, cache_paths, etc.)
3. Test with cigen validate to catch configuration errors early

Deduplication Not Happening

If jobs aren’t sharing installation:

1. Ensure identical packages values across jobs
2. Check job dependencies don’t prevent parallelization
3. Verify no custom commands that differ between jobs

Cache Misses

If packages are reinstalling unnecessarily:

1. Check lock files haven’t changed
2. Verify version files (.node-version, etc.) are consistent
3. Review cache keys with --verbose flag