CI/CD Pipeline Optimization: Cutting Build Times by 70%

VM/Container/Pod Resource Tuning: A Comprehensive Production Guide

Learn to tune sysctls, ulimits, cgroups, and Kubernetes resource controls for optimal performance, stability, and security in production workloads.

Slow pipelines kill developer productivity. A 30-minute build means 30 minutes of waiting per deployment. Here's how to optimize.

Analyze First

Measure Everything

# GitHub Actions timing
jobs:
  build:
    steps:
      - name: Measure step
        run: echo "::notice::Step completed in $SECONDS seconds"
```text

### Identify Bottlenecks

1. **Dependency installation** - Often 30-50% of build time
2. **Test execution** - Sequential tests waste parallel capacity
3. **Docker builds** - Unoptimized layers repeat work
4. **Deployment** - Sequential environments compound delays

## Caching Strategies

### Dependency Caching

```yaml
- uses: actions/cache@v3
  with:
    path: |
      ~/.npm
      node_modules
    key: ${{ runner.os }}-npm-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-npm-
```text

### Docker Layer Caching

```dockerfile
# Order matters - frequently changing layers last
FROM node:18 AS deps
WORKDIR /app
COPY package*.json ./
RUN npm ci

FROM node:18 AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build
```text

### GitHub Actions Cache for Docker

```yaml
- uses: docker/setup-buildx-action@v2
- uses: docker/build-push-action@v4
  with:
    cache-from: type=gha
    cache-to: type=gha,mode=max
```text

## Parallel Execution

### Matrix Strategy

```yaml
strategy:
  matrix:
    shard: [1, 2, 3, 4]
steps:
  - run: npm test -- --shard=${{ matrix.shard }}/4
```text

### Parallel Jobs

```yaml
jobs:
  lint:
    runs-on: ubuntu-latest
    steps: [lint steps]

  test-unit:
    runs-on: ubuntu-latest
    steps: [unit test steps]

  test-e2e:
    runs-on: ubuntu-latest
    steps: [e2e steps]

  build:
    needs: [lint, test-unit, test-e2e]
    steps: [build steps]
```text

## Test Optimization

### Selective Testing

```bash
# Only run affected tests
git diff --name-only origin/main | grep '\.ts$' | \
  xargs -I {} npm test -- --findRelatedTests {}
```text

### Test Parallelization

```yaml
# Jest configuration
{
  "maxWorkers": "50%",
  "shard": "1/4"
}
```text

## Resource Management

### Use Appropriate Runners

```yaml
# CPU-intensive builds
runs-on: ubuntu-latest-8-cores

# Memory-intensive operations
runs-on: ubuntu-latest-32gb
```text

### Timeout Management

```yaml
jobs:
  build:
    timeout-minutes: 15
    steps:
      - timeout-minutes: 5
        run: npm run build
```text

## Real Results

| Metric | Before | After | Improvement |
| -------- | -------- | ------- | ------------- |
| Total build time | 45 min | 12 min | 73% faster |
| npm install | 8 min | 45 sec | 90% faster |
| Test suite | 20 min | 6 min | 70% faster |
| Docker build | 12 min | 3 min | 75% faster |

## Conclusion

Optimize incrementally. Start with caching, add parallelization, then fine-tune. Measure before and after each change.