Mutation Testing

Optimization Strategies

Apply these strategies before running a campaign when Phase 3 of the configuration workflow requires optimization (estimated >16 hours or user requests).

Priority 1: Verify Target Selection

Most common issue: Mutating non-source code.

Diagnostic:

mewt print config     # Check [targets] include/ignore
mewt print targets    # Check what was actually mutated

Look for unintended files:

• Mocks: src/mocks/, __mocks__/

• Tests: *_test.rs, *.test.js, tests/

• Dependencies: vendor/, node_modules/

• Generated: proto/, generated/

Fix: Update [targets] in mewt.toml to be more specific:

# Before (too broad)
[targets]
include = ["**/*.rs"]

# After (specific)
[targets]
include = ["src/**/*.rs", "lib/**/*.rs"]
ignore = ["test", "mock", "generated"]

Re-run mewt mutate and check new count.

Priority 2: Analyze Project Structure

Goal: Understand mutant distribution and test organization to choose the right optimization.

1. Get mutant counts per component:

# Quote globs to prevent shell expansion. IDs output contains one mutant per line.
mewt print mutants --target 'src/auth/**/*.rs' --format ids | wc -l
mewt print mutants --target 'src/core/**/*.rs' --format ids | wc -l
mewt print mutants --target 'src/utils/**/*.rs' --format ids | wc -l

Present breakdown to user:

Component breakdown:
- src/auth/: 200 mutants × 5s = ~17 min
- src/core/: 800 mutants × 8s = ~1.8 hrs
- src/utils/: 150 mutants × 3s = ~8 min
Total: 1150 mutants, ~2.3 hrs worst-case

2. Count mutations by severity:

# Check enabled mutation types
mewt print config | grep mutations

# Count by severity level
mewt print mutants --severity high --format ids | wc -l
mewt print mutants --severity medium --format ids | wc -l
mewt print mutants --severity low --format ids | wc -l

# Or count specific mutation types
mewt print mutants --mutation-types ER --format ids | wc -l
mewt print mutants --mutation-types CR --format ids | wc -l

# Compare to total
mewt print mutants | wc -l

Example output:

High/Medium severity: 450 mutants
Total mutants: 1200
Percentage: 37.5%

Note: The percentage varies drastically between codebases (15% to 50+ % is common).

Priority 3: Choose Optimization Approach

Based on project structure analysis, present options to user with concrete time estimates:

Option A: Run Full Campaign

• "Estimated ~X hours worst-case (likely faster in practice)"

• "Recommend starting Friday evening for weekend completion"

• When to suggest: Duration acceptable, comprehensive coverage desired

Option B: Target Critical Components

• "Focus on specific components: src/auth/ (~17 min), src/crypto/ (~45 min)"

• "Start with one component and expand scope after review?"

• When to suggest: Clear component boundaries, user wants rapid iteration

Implementation:

[targets]
# Start with critical component
include = ["src/auth/**/*.rs"]

# After review, expand scope
# include = ["src/auth/**/*.rs", "src/core/**/*.rs"]

After editing mewt.toml, purge removed targets then mutate any newly included files: Preserve any results that must be retained and confirm that discarding the affected campaign data is authorized before purging.

mewt purge        # removes targets no longer matching [targets].include/ignore
mewt mutate src/  # adds mutants for any newly included files
mewt status       # confirm reduced mutant count

Option C: High/Medium Severity Only

• "Limit to high/medium severity mutations (X mutants, ~Y hours)"

• "Low severity (operator shuffles) tests edge cases, less critical"

• When to suggest: Time-constrained, need actionable findings quickly

Implementation (by severity level):

[run]
mutations = ["ER", "CR", "IF", "IT"]  # Specific types (high/medium)

After editing mewt.toml, full regeneration is required since existing mutants may no longer be valid under the new filter:

mewt purge --all  # clear all existing mutants
mewt mutate src/  # regenerate with restricted mutation types
mewt status       # confirm reduced mutant count

Or use severity filtering during analysis instead (no database changes needed):

# Run all mutants but filter results by severity
mewt results --severity high,medium
mewt print mutants --severity high

Trade-offs to explain:

• High/med severity: ~30-40% of mutants (varies by codebase)

• Low severity: ~60-70% of mutants (operator shuffles, edge cases)

• Low severity still provides value, just lower priority

• Using severity filters during analysis allows flexibility without re-running campaign

Option D: Two-Phase Campaign (Integration-Heavy Only)

• "Phase 1: Targeted tests (estimable upfront), Phase 2: Re-test uncaught with full suite (duration depends on Phase 1 survivor count)"

• "Total: Phase 1 estimate + (survivors × full-suite time) vs naive total"

• When to suggest: Integration tests dominate, unit tests don't map cleanly to files

See Two-Phase Campaigns section below for detailed setup.

Two-Phase Campaigns

Use ONLY for integration-heavy test suites. Not recommended for well-organized unit tests.

When to Use

Good fit:

• Integration tests dominate runtime

• Unit tests provide broad coverage but don't map cleanly to specific files

• Targeted test commands significantly faster than full suite

Not recommended:

• Well-organized unit tests with clear file mappings

• Tests already fast and targeted

Setup

Phase 1 config (targeted tests):

# TWO-PHASE CAMPAIGN
# Phase 1: Targeted tests (duration estimable upfront)
# Phase 2: Re-test uncaught mutants (duration depends on Phase 1 survivor count)

[test]
# PHASE 2: Uncomment after phase 1 completes
# cmd = "cargo test"
# timeout = 60

# PHASE 1: Targeted tests
[[per_target]]
glob = "src/auth/*.rs"
test.cmd = "cargo test auth::unit"
test.timeout = 10

[[per_target]]
glob = "src/core/*.rs"
test.cmd = "cargo test core::unit"
test.timeout = 15

# Catch-all: full suite for any file not matched above.
# Required unless [targets] is scoped to exactly the globs listed above.
[[per_target]]
glob = "**/*.rs"
test.cmd = "cargo test"
test.timeout = 60

Rationale: Phase 1 uses fast targeted tests. Phase 2 re-tests only the survivors with the comprehensive suite.

Execution

Phase 1:

mewt run

Wait for completion.

Phase 2 (after phase 1 completes):

1. Extract uncaught mutants:

   Copy

   mewt results --status Uncaught --format ids > uncaught_ids.txt

2. Update mewt.toml:

   • Comment out all [[per_target]] sections (including the catch-all)

   • Uncomment Phase 2 [test] section

3. Re-test with full suite:

   Copy

   mewt test --ids-file uncaught_ids.txt

4. Review final results:

   Copy

   mewt results  # Remaining uncaught are true coverage gaps

Example speedup:

Naive approach:
  2,000 mutants × 45s = 25 hours

Two-phase approach:
  Phase 1: 2,000 mutants × 8s = 4.4 hours → 450 uncaught (example outcome)
  Phase 2: 450 uncaught × 45s = 5.6 hours → 180 truly uncaught
  Total: ~10 hours (2.5× speedup)

Note: Phase 2 duration is unknowable before Phase 1 completes — it depends entirely
on how many mutants survive. The figures above illustrate one possible outcome.
Present Phase 1 as a firm estimate; present Phase 2 as (survivors × full-suite time)
once Phase 1 results are available.

Per-Target Test Configuration

Use when: Tests are well-organized by module/file, and running targeted tests is significantly faster than the full suite.

Setup Pattern

# Test full suite for every mutant (slow but comprehensive)
[test]
cmd = "go test ./..."
timeout = 45

# ALTERNATIVE: Targeted tests per file (fast, may miss cross-module failures)
[[per_target]]
glob = "auth/*.go"
test.cmd = "go test ./auth"
test.timeout = 10

[[per_target]]
glob = "core/*.go"
test.cmd = "go test ./core"
test.timeout = 15

[[per_target]]
glob = "utils/*.go"
test.cmd = "go test ./utils"
test.timeout = 8

# Catch-all for unmatched files
[[per_target]]
glob = "*.go"
test.cmd = "go test ./..."
test.timeout = 45

Ordering matters: First match wins. Place most specific patterns first, catch-all last.

Verify Speedup

time go test ./...      # Full suite: 45s
time go test ./auth     # Targeted: 8s

If targeted tests aren't significantly faster, this optimization won't help.

Trade-offs

Benefits:

• Faster campaign execution

• Scales linearly with codebase size

Risks:

• May miss cross-module integration bugs

• Requires correct glob-to-test mapping

Mitigation:

• Use this for initial passes

• Consider two-phase approach for comprehensive validation