Mutation Testing
Trail of BitsOfficialPublished by Trail of Bits
- Skill
- Tool integration
- Software development
- CC-BY-SA-4.0
- No issues found
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
This skill provides a structured approach to mutation testing using the mewt or muton tools. It guides users through the entire lifecycle of a mutation testing campaign, from initial configuration and execution to the analysis of surviving mutants. By systematically injecting faults into source code and observing test behavior, it helps developers distinguish between equivalent mutants and genuine testing gaps, ultimately revealing bugs that standard coverage metrics might miss.
What it does
- Configures and executes mutation testing campaigns via mewt or muton.
- Provides commands to filter, inspect, and re-test specific mutants.
- Categorizes mutation results into caught, uncaught, timeout, or skipped outcomes.
- Offers a framework for identifying equivalent mutants to reduce false positives.
- Prioritizes investigation based on mutation severity and language-specific patterns.
- Supports advanced analysis for blockchain-specific constructs like Solidity and Move.
When to use it
- When you need to validate the effectiveness of your existing test suite.
- When you want to identify untested code paths or weak assertions.
- When you need to analyze why specific mutants survived your test execution.
- When you are working on a project that requires high-assurance testing, such as blockchain smart contracts.
Advantages
- Uncovers subtle bugs that traditional line coverage metrics fail to detect.
- Provides a clear, actionable workflow for triaging and resolving surviving mutants.
- Reduces noise by helping developers identify and ignore semantically equivalent mutants.
- Enables targeted re-testing of specific code segments to verify fixes.
Usage examples
- “Run a mutation testing campaign on the src/auth directory.”
- The agent will execute the mutation campaign specifically targeting the auth module and provide a summary of the results.
- “Show me all uncaught mutants in the current project.”
- The agent will list the surviving mutants, allowing you to investigate potential testing gaps.
- “How do I investigate a high-severity surviving mutant?”
- The agent will guide you through inspecting the mutated code and suggest how to write a test to catch it.
- “Is this mutant equivalent to the original code?”
- The agent will apply the verification procedure to help you determine if the mutation is semantically identical or a genuine testing gap.
Install
Download the skill as a ZIP:
https://autogineer.com/agent-skills/trailofbits/mutation-testing/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip mutation-testing.zip -d .agents/skills
Documentation by Trail of Bits
Mutation Testing (mewt/muton)
Routes to the right mutation testing workflow and loads the references that workflow needs.
Note: muton and mewt share identical interfaces. Examples use
mewt; substitutemutonand its file names (muton.toml,muton.sqlite) for muton projects.
mewt --help and mewt <subcommand> --help are the source of truth for command-line behavior. Examples below reflect the mewt 4.x API; run --help when a flag looks unfamiliar or a command fails.
When to Use
Use this skill when the user:
Mentions "mewt", "muton", or "mutation testing"
Wants to configure, scope, or speed up a mutation testing campaign
Wants to analyze mutation results — surviving/uncaught mutants, equivalent mutants, kill rate
Wants to use mutation results to find bugs in the source code
When NOT to Use
Do not use this skill when the user asks about tests or line coverage without any mutation testing context.
Routing
Pick the workflow, then load it together with the references listed for it. Workflows and references do not load each other — that decision belongs here.
Setting up, scoping, or speeding up a campaign → workflows/configuration.md → Also load references/optimization-strategies.md when the campaign estimate is long enough to need trimming, or the user asks to make it faster.
Campaign finished, hunting for bugs in untested code → workflows/bug-hunter.md
Turning results into a formal analysis report → workflows/analyzing-results.md, plus:
references/equivalent-mutants.md — equivalence catalog and verification procedure
references/severity-classification.md — severity tier criteria
references/report-template.md — report structure
references/blockchain-patterns.md — only for Solidity, Move, FunC/Tolk, Cairo, or Solana Rust targets
references/input-formats.md — unless the results came from mewt or muton. Foreign tool output may not be self-describing; this covers the parsing anchors for slither-mutate, mull, and dextool-mutate
Anything else → run mewt --help or mewt <subcommand> --help, then assist directly.
Essential Commands
# Set up and run
mewt init # Create config and database
mewt mutate [paths] # Generate mutants without testing them
mewt run [paths] # Generate mutants and run the campaign
# Read results
mewt status # Overview with per-file breakdown
mewt results # Uncaught mutants (default view)
mewt results --all # Every outcome, not just uncaught
mewt results --format json # json | sarif | ids | table
# Narrow down (these filters work on both `results` and `print mutants`)
mewt results --target 'src/auth/**' # Quote globs so the shell does not expand them
mewt results --severity high,medium
mewt results --mutation-types ER,CR
mewt results --status Uncaught # Uncaught | TestFail | Skipped | Timeout
mewt results --line 42
# Investigate and re-test
mewt print mutant --id [id] # View the mutated code
mewt test --ids [ids] # Re-test specific mutants
mewt test --ids-file uncaught_ids.txt # Re-test IDs from a file, or '-' for stdin
# Inspect configuration
mewt print config # Effective config
mewt print targets # Files actually mutated
mewt print mutations --language [lang] # Mutations and severities for a languageLanguage labels are canonical family or family/dialect values in mewt 4.x — for example rust, javascript/ts, move/sui, move/iota.
What Results Mean
Caught/TestFail: tests detected the mutation (good)
Uncaught: tests did not detect the change. Inspect the code to distinguish a testing gap from an equivalent mutation.
Timeout: tests took too long — inconclusive, not evidence of coverage
Skipped: a less severe mutant was skipped because a more severe mutant on the same line was uncaught
Interpreting Mutation Types
mewt print mutations --language [lang] lists every mutation slug, description, and severity for a language, and is authoritative — the operator set grows with each release. What that output does not tell you is what a survivor means, which is where prioritization comes from:
| Severity | Representative slugs | What an uncaught mutant tells you |
|---|---|---|
| High | ER (Error Replacement) | Tests tolerate the injected error. Investigate whether the path executes, whether error handling masks the change, and whether assertions check the outcome. |
| Medium | CR (Comment Replacement) | Removing the statement does not fail the tests. Check whether its effects matter and whether assertions observe them. |
| Medium | IF/IT (If False/True), NR (Negation Removal) | Tests do not distinguish the changed condition. Both constant replacements surviving can indicate an unexecuted condition or weak assertions on the branch outcomes. |
| Low | Operator shuffles (AOS, COS, LOS, BOS, shift/assignment variants), BL, AS, LC, WF | Check boundary inputs, arithmetic assertions, and semantic equivalence. The mutation result alone does not establish whether the code executed. |
Severity ranks the mutation, not the risk. A low-severity survivor in a fee calculation matters more than a high-severity survivor in a log line — weigh what the mutated code does. Filter with --severity to work through the results in priority order.
Files
- ATTRIBUTION.mdAdded by AutoGineer942 B
- LICENSELicence file added by AutoGineer19.7 KiB
- SKILL.md6.1 KiB
- agents/openai.yaml243 B
- assets/trail-of-bits-mark.svg3.0 KiB
- references/blockchain-patterns.md4.8 KiB
- references/equivalent-mutants.md7.0 KiB
- references/input-formats.md2.4 KiB
- references/optimization-strategies.md8.7 KiB
- references/report-template.md4.3 KiB
- references/severity-classification.md5.9 KiB
- workflows/analyzing-results.md6.7 KiB
- workflows/bug-hunter.md7.4 KiB
- workflows/configuration.md11.3 KiB
references/blockchain-patterns.md
Blockchain-Specific Patterns
Blockchain-specific mutation testing patterns for Solidity, FunC/Tolk, Move, and Solana Rust codebases. These patterns extend the general equivalence catalog and severity criteria; where they disagree with the general guidance, the blockchain-specific rule wins for blockchain code.
Blockchain-Specific Equivalent Mutants
Additional equivalence patterns beyond the general catalog.
Revert String Mutations
Pattern: require(condition, "error A") mutated to require(condition, "error B")
Observable difference: The transaction reverts on the same inputs, but its returned error data changes. Treat this as a testing gap when callers or the documented interface depend on that data.
Applies to: Solidity require, revert with custom error messages
Scope: A property concerned only with whether the call reverts can ignore the message. State that restricted property explicitly. The absence of an assertion on error data does not make two different error messages semantically identical.
Gas-Only Differences
Pattern: Mutation in a view or pure function that changes gas consumption but not the return value or state
Scope: Identical return values can establish equivalence for a property that explicitly excludes gas consumption, provided neither version runs out of gas. They do not establish identical behavior in every calling context.
Applies to: Solidity view/pure functions
Not equivalent when: The gas difference causes an out-of-gas revert that changes observable behavior, or the function is called in a state-changing context where gas matters.
Modifier Ordering
Pattern: Swapping the order of two independent modifiers on a function When equivalent: Both guards must be independent, and their order must preserve all relevant observable behavior. Check the error returned when both guards fail, as well as state effects and callbacks. Applies to: Solidity function modifiers
Not equivalent when: The modifiers interact (e.g., one sets a flag that the other reads, or one can revert based on state the other modifies).
Storage Packing
Pattern: Mutation to a type within the same storage slot that does not change the packed representation What to check: An unchanged storage layout is insufficient. A type change can alter arithmetic, comparisons, or accepted values even when the stored bits match. Trace reads and writes before deciding whether behavior is equivalent. Caution: This is very rare. Only classify as equivalent with concrete evidence of packing behavior.
Blockchain Severity Examples
Concrete tier assignments for common on-chain constructs, using the same four tiers as the general severity criteria.
Tier 1: Critical — Security-Sensitive
Solidity:
require(msg.sender == owner)— ownership checkonlyRole(ADMIN_ROLE)— role-based access control modifiernonReentrantmodifier — reentrancy guardecrecover(hash, v, r, s)— signature verificationrequire(deadline >= block.timestamp)— time-based access control
FunC/Tolk:
throw_unless(ERR_UNAUTHORIZED, equal_slices(sender, owner))— ownership checkBounce handler validation — message authentication
Solana Rust:
if !ctx.accounts.authority.is_signer— signer check
Tier 2: High — Financial/State Integrity
Solidity:
balances[to] += amount— balance updatefee = amount * feeRate / DENOMINATOR— fee calculationrequire(ratio > MIN_COLLATERAL_RATIO)— liquidation thresholdIERC20(token).safeTransfer(to, amount)— token transferrequire(amountOut >= minAmountOut)— slippage check
FunC/Tolk:
send_raw_message(msg, mode)— value transfer via messageJetton transfer handler — token movement logic
Solana Rust:
account.lamports -= amount— lamport transferpool.total_shares += new_shares— share accounting
Tier 3: Medium — Business Logic
Solidity:
require(proposalId < proposals.length)— bounds check in governancerewards[user] += calculateReward(...)— non-critical reward distributionif (queue.length > MAX_QUEUE_SIZE) revert()— queue management
Tier 4: Low — Observability
Solidity:
emit Transfer(from, to, amount)— event emissionfunction balanceOf(...) view returns (uint256)— view getterError message string in
require(cond, "message")
Blockchain-Specific Ambiguous Cases
Protocol Standard Compliance
If event emission is required by an ERC or similar blockchain standard (e.g., Transfer in ERC-20) and downstream contracts or indexers depend on it, consider Tier 3 instead of Tier 4. This is stricter than the general case because on-chain protocol compliance is enforced by tooling and integrations.
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Declares allowed tools
Security
No issues found by our static scan· No issues found
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- Trail of Bits
- Publisher
- Trail of Bits (Official)
- Source
- trailofbits/skills · plugins/mutation-testing/skills/mutation-testing · commit e4dd0bb
- Licence
- Creative Commons Attribution-ShareAlike 4.0 International (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by Trail of Bits or Trail of Bits.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download mutation-testing.zip