Keep/_bmad/bmm/testarch/knowledge/burn-in.md

Burn-in Test Runner

Principle

Use smart test selection with git diff analysis to run only affected tests. Filter out irrelevant changes (configs, types, docs) and control test volume with percentage-based execution. Reduce unnecessary CI runs while maintaining reliability.

Rationale

Playwright's --only-changed triggers all affected tests:

• Config file changes trigger hundreds of tests
• Type definition changes cause full suite runs
• No volume control (all or nothing)
• Slow CI pipelines

The burn-in utility provides:

• Smart filtering: Skip patterns for irrelevant files (configs, types, docs)
• Volume control: Run percentage of affected tests after filtering
• Custom dependency analysis: More accurate than Playwright's built-in
• CI optimization: Faster pipelines without sacrificing confidence
• Process of elimination: Start with all → filter irrelevant → control volume

Pattern Examples

Example 1: Basic Burn-in Setup

Context: Run burn-in on changed files compared to main branch.

Implementation:

// Step 1: Create burn-in script
// playwright/scripts/burn-in-changed.ts
import { runBurnIn } from '@seontechnologies/playwright-utils/burn-in'

async function main() {
  await runBurnIn({
    configPath: 'playwright/config/.burn-in.config.ts',
    baseBranch: 'main'
  })
}

main().catch(console.error)

// Step 2: Create config
// playwright/config/.burn-in.config.ts
import type { BurnInConfig } from '@seontechnologies/playwright-utils/burn-in'

const config: BurnInConfig = {
  // Files that never trigger tests (first filter)
  skipBurnInPatterns: [
    '**/config/**',
    '**/*constants*',
    '**/*types*',
    '**/*.md',
    '**/README*'
  ],

  // Run 30% of remaining tests after skip filter
  burnInTestPercentage: 0.3,

  // Burn-in repetition
  burnIn: {
    repeatEach: 3,  // Run each test 3 times
    retries: 1      // Allow 1 retry
  }
}

export default config

// Step 3: Add package.json script
{
  "scripts": {
    "test:pw:burn-in-changed": "tsx playwright/scripts/burn-in-changed.ts"
  }
}

Key Points:

• Two-stage filtering: skip patterns, then volume control
• skipBurnInPatterns eliminates irrelevant files
• burnInTestPercentage controls test volume (0.3 = 30%)
• Custom dependency analysis finds actually affected tests

Example 2: CI Integration

Context: Use burn-in in GitHub Actions for efficient CI runs.

Implementation:

# .github/workflows/burn-in.yml
name: Burn-in Changed Tests

on:
  pull_request:
    branches: [main]

jobs:
  burn-in:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0 # Need git history

      - name: Setup Node
        uses: actions/setup-node@v4

      - name: Install dependencies
        run: npm ci

      - name: Run burn-in on changed tests
        run: npm run test:pw:burn-in-changed -- --base-branch=origin/main

      - name: Upload artifacts
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: burn-in-failures
          path: test-results/

Key Points:

• fetch-depth: 0 for full git history
• Pass --base-branch=origin/main for PR comparison
• Upload artifacts only on failure
• Significantly faster than full suite

Example 3: How It Works (Process of Elimination)

Context: Understanding the filtering pipeline.

Scenario:

Git diff finds: 21 changed files
├─ Step 1: Skip patterns filter
│  Removed: 6 files (*.md, config/*, *types*)
│  Remaining: 15 files
│
├─ Step 2: Dependency analysis
│  Tests that import these 15 files: 45 tests
│
└─ Step 3: Volume control (30%)
   Final tests to run: 14 tests (30% of 45)

Result: Run 14 targeted tests instead of 147 with --only-changed!

Key Points:

• Three-stage pipeline: skip → analyze → control
• Custom dependency analysis (not just imports)
• Percentage applies AFTER filtering
• Dramatically reduces CI time

Example 4: Environment-Specific Configuration

Context: Different settings for local vs CI environments.

Implementation:

import type { BurnInConfig } from '@seontechnologies/playwright-utils/burn-in';

const config: BurnInConfig = {
  skipBurnInPatterns: ['**/config/**', '**/*types*', '**/*.md'],

  // CI runs fewer iterations, local runs more
  burnInTestPercentage: process.env.CI ? 0.2 : 0.3,

  burnIn: {
    repeatEach: process.env.CI ? 2 : 3,
    retries: process.env.CI ? 0 : 1, // No retries in CI
  },
};

export default config;

Key Points:

• process.env.CI for environment detection
• Lower percentage in CI (20% vs 30%)
• Fewer iterations in CI (2 vs 3)
• No retries in CI (fail fast)

Example 5: Sharding Support

Context: Distribute burn-in tests across multiple CI workers.

Implementation:

// burn-in-changed.ts with sharding
import { runBurnIn } from '@seontechnologies/playwright-utils/burn-in';

async function main() {
  const shardArg = process.argv.find((arg) => arg.startsWith('--shard='));

  if (shardArg) {
    process.env.PW_SHARD = shardArg.split('=')[1];
  }

  await runBurnIn({
    configPath: 'playwright/config/.burn-in.config.ts',
  });
}

# GitHub Actions with sharding
jobs:
  burn-in:
    strategy:
      matrix:
        shard: [1/3, 2/3, 3/3]
    steps:
      - run: npm run test:pw:burn-in-changed -- --shard=${{ matrix.shard }}

Key Points:

• Pass --shard=1/3 for parallel execution
• Burn-in respects Playwright sharding
• Distribute across multiple workers
• Reduces total CI time further

Integration with CI Workflow

When setting up CI with *ci workflow, recommend burn-in for:

• Pull request validation
• Pre-merge checks
• Nightly builds (subset runs)

Related Fragments

• ci-burn-in.md - Traditional burn-in patterns (10-iteration loops)
• selective-testing.md - Test selection strategies
• overview.md - Installation

Anti-Patterns

❌ Over-aggressive skip patterns:

skipBurnInPatterns: [
  '**/*', // Skips everything!
];

✅ Targeted skip patterns:

skipBurnInPatterns: ['**/config/**', '**/*types*', '**/*.md', '**/*constants*'];

❌ Too low percentage (false confidence):

burnInTestPercentage: 0.05; // Only 5% - might miss issues

✅ Balanced percentage:

burnInTestPercentage: 0.2; // 20% in CI, provides good coverage