Job Skipping

Cigen contains an initial scaffold for job skipping based on a hash of source files. This is experimental: the current implementation writes a local marker file and is not yet backed by a persistent cache, so it does not skip across separate CI runs.

How It Works

Calculate SHA-256 hash of configured source files 2. Check for a local skip marker file for that hash 3. If present, halt remaining steps (CircleCI) 4. Otherwise, run the job and write the marker file at the end

Source File Configuration

Named Source File Groups

Define reusable patterns in your configuration:

# In config.yml or config/source_file_groups.yml
source_file_groups:
ruby:
  - '**/*.rb'
  - '**/*.rake'
  - 'Gemfile*'
  - 'Rakefile'
  - '.ruby-version'

frontend: - 'client/**/*' - 'package.json' - 'pnpm-lock.yaml' - 'config/webpack/**/*' - 'babel.config.js'

docker: - 'Dockerfile*' - '.dockerignore' - 'docker-compose*.yml'

Using Source File Groups

jobs:
ruby_lint:
  image: cimg/ruby:3.3
  source_files: '@ruby'  # References the ruby group
  steps:
    - run: bundle exec rubocop

frontend_test:
image: cimg/node:18
source_files: - '@frontend' # Named group - 'jest.config.js' # Additional files - 'tsconfig.json'
steps: - run: npm test

Inline Source Files

For job-specific patterns:

jobs:
shellcheck:
  image: cimg/base:stable
  source_files:
    - 'scripts/**/*.sh'
    - '*.sh'
    - '.shellcheckrc'
  steps:
    - run: shellcheck scripts/*.sh

Automatic Template Inclusion

When processing workflows/test/jobs/ruby_lint.yml, cigen automatically includes:

The job definition file itself
Any referenced command templates
Provider-specific template files

This ensures that changes to CI configuration trigger appropriate job re-runs.

Multi-Architecture Support

Job skipping is architecture-aware:

jobs:
build_multiarch:
  image: cimg/base:stable
  architectures: [amd64, arm64]  # Each arch tracked separately
  source_files: '@docker'
  steps:
    - run: docker build -t myapp:latest .

Writes separate marker files per architecture under /tmp/cigen_skip_cache/, for example:

/tmp/cigen_skip_cache/job_{hash}_amd64
/tmp/cigen_skip_cache/job_{hash}_arm64

Generated Implementation

Hash Calculation

- run:
  name: Calculate source file hash
  command: |
    echo "Calculating hash for source files..."
    TEMP_HASH_FILE="/tmp/source_files_for_hash"
    rm -f "$TEMP_HASH_FILE"

    # Collect matching files for each pattern
    find . -name "*.rb" -type f >> "$TEMP_HASH_FILE" 2>/dev/null || true
    find . -name "Gemfile*" -type f >> "$TEMP_HASH_FILE" 2>/dev/null || true

    if [ -f "$TEMP_HASH_FILE" ]; then
      export JOB_HASH=$(sort "$TEMP_HASH_FILE" | xargs sha256sum | sha256sum | cut -d' ' -f1)
      echo "Source file hash: $JOB_HASH"
    else
      export JOB_HASH="empty"
      echo "No source files found"
    fi

Skip Logic

- run:
  name: Check if job should be skipped
  command: |
    SKIP_MARKER="/tmp/cigen_skip_cache/job_${JOB_HASH}_amd64"
    if [ -f "$SKIP_MARKER" ]; then
      echo "Job completed successfully for hash ${JOB_HASH}. Skipping..."
      circleci step halt  # CircleCI: halt remaining steps
    else
      echo "No previous successful run found. Proceeding..."
      mkdir -p /tmp/cigen_skip_cache
    fi

Completion Marking

- run:
  name: Record successful completion
  command: |
    echo "Recording successful completion for hash ${JOB_HASH}"
    echo "$(date): Job completed successfully" > "/tmp/cigen_skip_cache/job_${JOB_HASH}_amd64"
  when: on_success  # Only run if job succeeds

Platform Notes

CircleCI: uses circleci step halt to skip remaining steps in a job
Other providers: not yet implemented

Status and Roadmap

This feature is incomplete without a persistent backend. A job-status cache (to restore skip markers across runs) is planned.

Real-World Examples

Ruby Application

source_file_groups:
ruby: ['**/*.rb', 'Gemfile*', '.ruby-version']
javascript: ['client/**/*', 'package.json', 'pnpm-lock.yaml']

jobs:
ruby_lint:
source_files: '@ruby'
steps: [run: bundle exec rubocop]

ruby_test:
source_files: ['@ruby', 'spec/**/*']
steps: [run: bundle exec rspec]

javascript_test:
source_files: '@javascript'
steps: [run: npm test]

security_scan:
source_files: ['@ruby', '@javascript', 'Gemfile.lock']
steps: [run: bundle audit]

Microservices

source_file_groups:
user_service: ['services/user/**/*', 'shared/**/*']
order_service: ['services/order/**/*', 'shared/**/*']
payment_service: ['services/payment/**/*', 'shared/**/*']

jobs:
test_user_service:
source_files: '@user_service'
steps: [run: go test ./services/user/...]

test_order_service:
source_files: '@order_service'
steps: [run: go test ./services/order/...]

build_user_image:
source_files: ['@user_service', 'Dockerfile.user']
steps: [run: docker build -f Dockerfile.user .]

Best Practices

Source File Precision

# Good - precise patterns
source_files:
- 'src/**/*.py'      # Only Python source
- 'requirements.txt' # Dependencies
- 'pytest.ini'      # Test config

# Bad - too broad

source_files:

- '*_/_' # Includes everything - rarely skips

# Bad - too narrow

source_files:

- 'src/main.py' # Missing dependencies and config

Configuration Dependencies

Include all files that affect job behavior:

jobs:
docker_build:
  source_files:
    - 'src/**/*'        # Source code
    - 'Dockerfile'      # Build instructions
    - '.dockerignore'   # Build context rules
    - 'requirements.txt' # Dependencies
    - 'config/*.json'   # App configuration

lint*javascript:
source_files: - 'src/**/_.js' # Source files - '.eslintrc.json' # Lint configuration - 'package.json' # Dependencies - 'tsconfig.json' # TypeScript config

Troubleshooting

Jobs Never Skip

If jobs always run despite unchanged files:

Check patterns match files: Verify glob patterns find expected files 2. Remove broad patterns: Avoid **/* or overly inclusive patterns 3. Check for changing files: Look for files that change on every run (logs, timestamps) 4. Validate template stability: Ensure CI templates aren’t changing

Jobs Skip When They Shouldn’t

If jobs skip despite relevant changes:

Expand source files: Add missing dependencies and config files 2. Check ignored files: Ensure important files aren’t in .gitignore 3. Clear skip cache: Remove /tmp/cigen_skip_cache to reset state 4. Add debug logging: Use verbose mode to see hash calculations

Debug Hash Calculation

# Add to job for debugging
steps:
- run:
    name: Debug source file tracking
    command: |
      echo "=== Source File Debug ==="
      find . -name "*.rb" -type f | sort
      find . -name "Gemfile*" -type f | sort
      echo "=== Hash Calculation ==="
      # Show exactly which files contribute to hash

Performance Impact

Job skipping provides significant performance improvements:

Project Size	Typical Skip Rate	Time Savings
Small (< 10 jobs)	60-70%	40-50%
Medium (10-50 jobs)	70-80%	50-70%
Large (50+ jobs)	80-90%	60-80%

Cache Storage

Skip cache markers are stored in:

CircleCI: /tmp/cigen_skip_cache/ (persists across workflow steps)
GitHub Actions: Workspace cache (persists across jobs)
Self-hosted: Configurable cache backend (Redis, S3, etc.)

The cache can be cleared by removing the cache directory or using provider-specific cache clearing mechanisms.