Generated banner for Microbenchmarking
Generated logo for Microbenchmarking

Microbenchmarking

.NET (Microsoft)OfficialPublished by .NET (Microsoft)

  • Skill
  • Tool integration
  • Software development
  • MIT
  • No issues found

From the author

Activate this skill when BenchmarkDotNet (BDN) is involved in the task — creating, running, configuring, or reviewing BDN benchmarks. Also activate when microbenchmarking .NET code would be useful and BenchmarkDotNet is the likely tool. Consider activating when answering a .NET performance question requires measurement and BenchmarkDotNet may be needed. Covers microbenchmark design, BDN configuration and project setup, how to run BDN microbenchmarks efficiently and effectively, and using BDN for side-by-side performance comparisons. Do NOT use for profiling/tracing .NET code (dotnet-trace, PerfView), production telemetry, or load/stress testing (Crank, k6).
— the description by .NET (Microsoft)

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides comprehensive guidance on using BenchmarkDotNet (BDN) to measure, configure, and analyze .NET code performance. It covers the full lifecycle of microbenchmarking, from designing test cases and configuring jobs to executing benchmarks and interpreting results. The skill emphasizes comparative analysis, helping developers identify performance regressions or improvements across different implementations, runtime versions, or dependency versions. It includes best practices for avoiding common measurement pitfalls, such as constant folding and improper setup, ensuring that benchmark data is statistically sound and actionable.

What it does

  • Guides the design of microbenchmarks for coverage, investigation, and change validation.
  • Explains how to configure BenchmarkDotNet jobs, parameters, and run strategies.
  • Provides patterns for side-by-side performance comparisons of code, runtimes, and dependencies.
  • Details how to set up benchmark projects and entry points using BenchmarkSwitcher or BenchmarkRunner.
  • Offers techniques for validating benchmark correctness and managing execution costs.
  • Advises on interpreting BDN reports and handling verbose output.

When to use it

  • When you need to measure the performance of specific .NET code paths.
  • When comparing the performance of two or more implementation approaches.
  • When investigating performance regressions in a .NET application.
  • When validating the performance impact of a code change before merging.
  • When evaluating how different .NET runtime versions or library versions affect performance.

Advantages

  • Reduces measurement errors by providing verified, current patterns for BDN.
  • Enables statistically significant performance comparisons through controlled job configurations.
  • Helps prevent common pitfalls like JIT-related measurement noise or dead code elimination.
  • Provides structured approaches for both temporary investigations and permanent coverage suites.
  • Optimizes developer time by suggesting appropriate job presets based on the task.

Usage examples

“How do I compare the performance of two different sorting algorithms using BenchmarkDotNet?”
The agent will provide a code example using a single class with two methods marked with [Benchmark], one of which uses [Benchmark(Baseline = true)], and explain how to run them to see the relative ratio.
“I need to check if my code is faster on .NET 9 than .NET 8. How do I set that up?”
The agent will guide you to configure your .csproj with multiple target frameworks and use the --runtimes CLI flag or a custom config object to run the benchmark against both runtimes.
“My benchmark is taking too long to run. How can I speed it up during development?”
The agent will suggest using the --job Short preset for quicker iterations and explain how to use --filter to run only the specific methods currently under development.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/dotnet/microbenchmarking/download
  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip microbenchmarking.zip -d .agents/skills

Documentation by .NET (Microsoft)

Benchmark Authoring Guidelines

BenchmarkDotNet (BDN) is a .NET library for writing and running microbenchmarks. Throughout this skill, "BDN" refers to BenchmarkDotNet.

Note: Evaluations of LLMs writing BenchmarkDotNet benchmarks have revealed common failure patterns caused by outdated assumptions about BDN's behavior — particularly around runtime comparison, job configuration, and execution defaults that have changed in recent versions. The reference files in this skill contain verified, current information. You MUST read the reference files relevant to the task before writing any code — your training data likely contains outdated or incorrect BDN patterns.

Key concepts

  • Job — describes how to run a benchmark: runtime, iteration counts, launch count, run strategy, and environment settings. Multiple jobs can be configured to run the same benchmarks under different conditions.

  • Benchmark case — one method × one parameter combination × one job. The atomic unit BDN measures.

  • Operation — the logical unit of work being measured. All BDN output columns (Mean, Error, etc.) report time per operation.

  • Invocation — a single call to the benchmark method. By default, 1 invocation = 1 operation. With OperationsPerInvoke=N, each invocation counts as N operations.

  • Iteration — a timed batch of invocations. BDN measures the total time for all invocations in an iteration, then divides by the total operation count to get per-operation time.

Benchmarks are comparative instruments

A single benchmark number has limited value — it can confirm the order of magnitude of a measurement, but the exact value changes across machines, operating systems, and runtime configurations. Benchmarks produce the most useful information when compared against something. Before writing benchmarks, identify the comparison axis for the current task:

  • Approaches (A vs B): comparing alternative implementations side-by-side in the same run.

  • Runtimes: comparing the same code across .NET versions (e.g., net8.0 vs net9.0).

  • Package versions: comparing different versions of a NuGet dependency.

  • Builds (before/after): comparing a saved DLL of the old code against the current source.

  • Runtime configuration (GC mode, JIT settings): understanding how runtime settings affect performance — compared via multiple jobs in a single run.

  • Scale (N=100 vs N=1000): understanding how performance changes as input size grows.

  • Hardware/OS: comparing across different machines or operating systems — requires separate runs on each environment.

  • Historical measurements: comparing against measurements recorded at a previous point in time.

BDN can compare the first six axes side-by-side in a single run, but each requires specific CLI flags or configuration that differ from what you might expect — read references/comparison-strategies.md for the correct approach for each strategy before configuring a comparison.

Use cases and benchmark lifecycle

There are four distinct reasons a developer writes a benchmark, and each one changes how the benchmark should be designed and where it should live:

  1. Coverage suite: Write benchmarks to maximize coverage of real-world usage patterns so that regressions affecting most users are caught. These benchmarks are permanent — they belong in the project's benchmark suite, follow its conventions (directory structure, base classes, naming), and are checked in.

  2. Issue investigation: Someone has reported a specific performance problem. Write benchmarks to reproduce and diagnose that specific issue. These benchmarks are task-scoped — they persist across the investigation (reproduce → isolate → verify fix) but are not part of the permanent suite.

  3. Change validation: A developer has a PR or change and wants to understand its performance characteristics before merging. These benchmarks are task-scoped — they persist across the review cycle but are not checked in.

  4. Development feedback: A developer is actively working on a task and wants to use benchmarks to evaluate approaches and get information early. These benchmarks are task-scoped and throwaway — they persist across the development session but are deleted when the decision is made.

For use case 1, add to the existing benchmark project following its conventions. For use cases 2–4, create a standalone project in a working directory that persists for the task but is clearly not part of the permanent codebase.

For coverage suite benchmarks, design from the perspective of real callers — what code patterns use this API, what inputs they pass, and what performance characteristics matter to them. Each permanent benchmark should justify its maintenance cost through real-world relevance. For temporary benchmarks, keep the case count intentional — each additional test case costs wall-clock time (read Cost awareness).

Cost awareness

Each benchmark case (one method × one parameter combination × one job) takes 15–25 seconds with default settings. [Params] creates a Cartesian product: two [Params] with 3 and 4 values across 5 methods = 60 cases ≈ 20 minutes. Multiple jobs multiply this further. Before running, estimate the total case count and match the job preset to the situation:

PresetPer-case timeWhen to use
--job Dry<1sValidate correctness — confirms compilation and execution without measurement
--job Short5–8sQuick measurements during development or investigation
(default)15–25sFinal measurements for a coverage suite
--job Medium33–52sHigher confidence when results matter
--job Long3–12 minHigh statistical confidence

If benchmark runs take longer than expected, results seem unstable, or you need to tune iteration counts or execution settings, read references/bdn-internals-and-tuning.md for detailed information about BDN's execution pipeline and configuration options.

Entry points and configuration

BDN programs use either BenchmarkSwitcher (provides interactive benchmark selection for humans, parses CLI arguments) or BenchmarkRunner (runs specified benchmarks directly). Both support CLI flags like --filter and --runtimes, but only when args is passed through — without it, CLI flags are silently ignored. When using BenchmarkSwitcher, always pass --filter to avoid hanging on an interactive prompt.

BDN behavior is customized through attributes, config objects, and CLI flags.

Read references/project-setup-and-running.md for entry point setup, config object patterns, and CLI flags. If you need to collect data beyond wall-clock time — such as memory allocations, hardware counters, or profiling traces — read references/diagnosers-and-exporters.md.

Running benchmarks

BenchmarkDotNet console output is extremely verbose — hundreds of lines per case showing internal calibration, warmup, and measurement details. Redirect all output to a file to avoid consuming context on verbose iteration output:

dotnet run -c Release -- --filter "*MethodName" --noOverwrite > benchmark.log 2>&1

Each benchmark method can take several minutes. Rather than running all benchmarks at once, use --filter to run a subset at a time (e.g. one or two methods per invocation), read the results, then run the next subset. This keeps each invocation short — avoiding session or terminal timeouts — and lets you verify results incrementally. Read references/project-setup-and-running.md for filter syntax, CLI flags, and project setup.

After each run, read the Markdown report (*-report-github.md) from the results directory for the summary table. Only read benchmark.log if you need to investigate errors or unexpected results.

Writing new benchmarks

Step 1: Plan the test cases

Before writing any code, determine:

  • Which use case this benchmark serves (coverage, investigation, change validation, or development feedback).

  • Which comparison axis applies (what will the number be compared against?).

  • What real-world scenarios to benchmark, based on how callers actually use the API.

Each benchmark case should justify its cost. An uncovered scenario is usually more valuable than another parameter combination for one already covered, but when a specific parameter dimension genuinely affects performance characteristics, the depth is warranted.

Decide on the list of test cases. For each test case, think through:

  • How to express variation: BenchmarkDotNet provides several mechanisms for parameterizing benchmarks — [Params] and [ParamsSource] for property-level parameters, [Arguments] and [ArgumentsSource] for method-level arguments, [ParamsAllValues] to enumerate all values of a bool or enum, and [GenericTypeArguments] for varying type parameters on generic benchmark classes. Choose the mechanism that best fits the dimension being varied. Read references/writing-benchmarks.md for the full set of options and correctness patterns.

  • Where input data comes from — consider which sources are appropriate (these can be combined):

    • Hard-coded values — small, fixed values where the exact input matters (e.g., specific strings, known edge-case sizes). Store in fields or [Params] to avoid constant folding.

    • Asset files — static data that is too large or impractical to embed in source code such as binary blobs.

    • Programmatically generated via [ParamsSource]/[ArgumentsSource]/[GlobalSetup] — when data shape matters more than specific content, or when input must be parameterized by size.

  • Whether randomness is appropriate: If using generated data, use seeded randomness for reproducibility. When generating random data, use a large enough sample that the generated distribution is representative (e.g., 4 random values may cluster in a narrow range, while 1000 will better exercise the full distribution).

Step 2: Implement the benchmarks

For coverage suite benchmarks, add to the existing benchmark project and follow its conventions. For temporary benchmarks (investigation, change validation, development feedback), create a standalone project — read references/project-setup-and-running.md for project setup and entry point configuration.

Adding the BenchmarkDotNet package: Always use dotnet add package BenchmarkDotNet (no version) — this lets NuGet resolve the latest compatible version. Do NOT manually write a <PackageReference> with a version number into the .csproj; BDN versions in training data are outdated and may lack support for current .NET runtimes.

Write the benchmark code. Follow the patterns in references/writing-benchmarks.md to avoid common measurement errors — in particular:

  • Return results from benchmark methods to prevent dead code elimination

  • Move initialization to [GlobalSetup] — setup inside the benchmark method is measured; use [IterationSetup] only when the benchmark mutates state that must be reset between iterations

  • Do not add manual loops — BDN controls invocation count automatically

  • Mark a baseline when comparing alternatives — use [Benchmark(Baseline = true)] for method-level comparisons or .AsBaseline() on a job for multi-job comparisons so results show relative ratios

  • Store inputs in fields or [Params], not as literals or const values — the JIT can fold constant expressions at compile time, making the benchmark measure a precomputed result instead of the actual computation

Step 3: Validate and run

Validate before committing to a long run:

  1. Run with --job Dry first to catch compilation errors and runtime exceptions without spending time on measurement.

  2. Run a single representative case with default settings to verify the output looks correct and the numbers are in the expected range.

  3. Only run the full suite after validation passes.

When iterating on benchmark design, use --job Short until confident, then switch to default for final numbers.

Files

references/writing-benchmarks.md

Benchmark authoring techniques

This reference covers BenchmarkDotNet features and practices for writing correct benchmark methods. Incorrect benchmarks silently produce misleading results — the techniques here prevent common measurement errors.

Dead code elimination

The JIT compiler can eliminate code whose result is never used. If a benchmark method returns void and doesn't store its result anywhere observable, the JIT may partially or fully optimize away the computation.

Returning a value from the benchmark method prevents this. When the method has a non-void return type, the JIT cannot prove the result is unused (the call site in BDN's generated harness is opaque to the JIT), so it must preserve the computation:

// Correct — non-void return prevents DCE
[Benchmark]
public int Parse() => int.Parse("12345");

// Wrong — void return, JIT may eliminate the call
[Benchmark]
public void Parse() => int.Parse("12345");

For benchmarks that must return void (e.g., methods with side effects like Array.Sort), DCE is not a concern because the operation itself has observable effects.

For iterating deferred sequences where returning the sequence object doesn't force enumeration, use the Consumer class explicitly — see Deferred execution.

Setup and cleanup

GlobalSetup / GlobalCleanup

A method marked [GlobalSetup] runs once per benchmark case, before all iterations of that case. Use it to allocate buffers, load data, and initialize state that the benchmark reads but does not mutate.

private int[] _array;

[GlobalSetup]
public void Setup() => _array = Enumerable.Range(0, 1000).ToArray();

[Benchmark]
public int Sum() => _array.Sum();

Initialization logic inside the benchmark method itself is measured — this is almost never what you want.

When a class has multiple benchmarks needing different setup, use the Target or Targets property:

[GlobalSetup(Target = nameof(BenchmarkA))]
public void SetupA() { ... }

[GlobalSetup(Targets = new[] { nameof(BenchmarkB), nameof(BenchmarkC) })]
public void SetupBC() { ... }

[GlobalCleanup] runs once after all iterations. Use it to dispose resources (files, connections) created during setup.

Note: when [MemoryRandomization] is enabled, BDN re-runs [GlobalCleanup] and [GlobalSetup] after every iteration (overriding the normal once-only behavior). See references/bdn-internals-and-tuning.md for details.

IterationSetup / IterationCleanup

A method marked [IterationSetup] runs before every iteration. Use it only when the benchmark mutates state that must be reset — for example, sorting an array in-place.

Critical constraint: when [IterationSetup] or [IterationCleanup] is used, BDN defaults to InvocationCount=1 and UnrollFactor=1 (unless explicitly overridden). This means the entire iteration is a single invocation, so the operation should be long enough (generally hundreds of milliseconds or more) for a single-invocation measurement to be reliable — with only one invocation, OS scheduling jitter and timer resolution become a significant fraction of the measured time.

[Params(10_000_000)] // large enough for single-invocation measurement
public int Size { get; set; }

private int[] _data, _scratch;

[GlobalSetup]
public void Setup() => _data = Enumerable.Range(0, Size).Reverse().ToArray();

[IterationSetup]
public void ResetArray() => Array.Copy(_data, _scratch = new int[Size], Size);

[Benchmark]
public void Sort() => Array.Sort(_scratch);

If [GlobalSetup] is sufficient, do not use [IterationSetup]. It adds overhead and constrains the measurement.

OperationsPerInvoke

When part of the benchmark setup cannot be moved to [GlobalSetup] — typically because it involves a stack-only type like Span<T> — use OperationsPerInvoke to amortize the setup cost by performing multiple logical operations per invocation.

private byte[] _array = new byte[18];

[Benchmark(OperationsPerInvoke = 16)]
public Span<byte> Slice()
{
    Span<byte> span = new Span<byte>(_array); // setup: can't go in GlobalSetup (stack-only)

    // 16 operations, unrolled without a loop
    span = span.Slice(1); span = span.Slice(1); span = span.Slice(1); span = span.Slice(1);
    span = span.Slice(1); span = span.Slice(1); span = span.Slice(1); span = span.Slice(1);
    span = span.Slice(1); span = span.Slice(1); span = span.Slice(1); span = span.Slice(1);
    span = span.Slice(1); span = span.Slice(1); span = span.Slice(1); span = span.Slice(1);

    return span;
}

BDN divides the measured time by OperationsPerInvoke, so the reported result reflects a single Slice operation with the Span creation cost amortized. The operations are manually unrolled (not in a loop) to avoid loop overhead affecting the measurement.

Avoiding loops

BDN determines how many times to invoke the benchmark method per iteration automatically (the pilot stage). Adding a manual loop inside the benchmark is unnecessary and harmful:

// Wrong — manual loop adds loop overhead and hides per-operation variance
[Benchmark]
public int ParseLoop()
{
    int result = 0;
    for (int i = 0; i < 1000; i++)
        result += int.Parse("12345");
    return result;
}

// Correct — BDN handles invocation count
[Benchmark]
public int Parse() => int.Parse("12345");

The exception is OperationsPerInvoke — when per-invocation setup cannot be avoided, repeating the operation inside the method and declaring the count lets BDN divide the result correctly.

No side effects

Benchmarks should be idempotent — running the method N times should produce the same performance characteristics as running it once. A benchmark that grows a list on every invocation violates this:

// Wrong — list grows with every invocation, later calls are slower
List<int> _numbers = new();

[Benchmark]
public void Add() => _numbers.Add(12345);

If the operation inherently mutates state (e.g., Array.Sort), use [IterationSetup] to reset state between iterations, with input large enough for reliable single-invocation measurement.

Async benchmarks

BDN natively supports benchmark methods that return Task, Task<T>, ValueTask, or ValueTask<T>. BDN awaits the result automatically — there is no special attribute or configuration needed:

[Benchmark]
public async Task<int> ReadAsync()
{
    using var stream = new FileStream(_path, FileMode.Open, FileAccess.Read, FileShare.Read, 4096, useAsync: true);
    var buffer = new byte[4096];
    return await stream.ReadAsync(buffer);
}

Async setup/cleanup: [GlobalSetup] and [GlobalCleanup] methods can also return Task or ValueTask — BDN awaits them automatically. [IterationSetup] and [IterationCleanup] do not support async return types; they must be synchronous.

When NOT to use async: if the operation being benchmarked is synchronous, do not wrap it in async/await. The async state machine adds overhead that would be included in the measurement:

// Wrong — adds async state machine overhead to a synchronous operation
[Benchmark]
public async Task<int> ParseAsync() => await Task.FromResult(int.Parse("12345"));

// Correct — benchmark the synchronous method directly
[Benchmark]
public int Parse() => int.Parse("12345");

Deferred execution

Returning IEnumerable<T>, IQueryable<T>, or Lazy<T> from a benchmark measures only the creation of the deferred sequence — not its execution. This is almost always a bug:

// Wrong — measures only the creation of the LINQ query object, not its execution
[Benchmark]
public IEnumerable<int> WhereQuery() => _numbers.Where(n => n > 50);

// Correct — materialize the result so the query actually executes
[Benchmark]
public List<int> WhereQuery() => _numbers.Where(n => n > 50).ToList();

BDN includes a DeferredExecutionValidator that detects this mistake and raises a validation error, preventing the benchmark from running. To fix it, either materialize the result (.ToList(), .ToArray()) or consume it manually using the .Consume() extension method:

private Consumer _consumer = new();

[Benchmark]
public void WhereQuery() => _numbers.Where(n => n > 50).Consume(_consumer);

Constant folding

The JIT can evaluate expressions at compile time when all inputs are constants or literals. If a benchmark operates on constant inputs, the JIT may fold the computation into a precomputed result, and the benchmark measures nothing:

// Wrong — JIT can compute Math.Sqrt(144.0) at compile time
[Benchmark]
public double Sqrt() => Math.Sqrt(144.0);

// Correct — field value is not a compile-time constant
private double _value = 144.0;

[Benchmark]
public double Sqrt() => Math.Sqrt(_value);

Store inputs in fields, [Params], or [Arguments] — never as literals or const values in the benchmark method.

Randomness and reproducibility

When benchmarks use generated input data, use a fixed seed so results are comparable across runs:

[GlobalSetup]
public void Setup()
{
    var rng = new Random(42); // fixed seed
    _data = new byte[Size];
    rng.NextBytes(_data);
}

Benchmark class constraints

BDN generates source code for a derived type from the benchmark class, which is compiled as part of its generated project. This imposes constraints:

  • The class must be public

  • The class must not be sealed, static, or abstract

  • The class must be a class, not a struct or ref struct

  • Benchmark methods must be public and instance (not static)

Parameterization summary

FeatureTargetUse when
[Params]Field/PropertyValues needed in setup; applies to all benchmarks in the class
[ParamsSource]Field/PropertyDynamic or complex values needed in setup
[ParamsAllValues]Field/PropertyEnum or bool — test every possible value
[Arguments]MethodInline values for a specific benchmark method
[ArgumentsSource]MethodDynamic or complex values for a specific benchmark method
[GenericTypeArguments]ClassVary type parameters (e.g., value type vs reference type)

[Params] applies to all benchmarks in the class. If different benchmarks need different parameter ranges, split them into separate classes.

[Arguments] and [ArgumentsSource] are method-specific and their initialization time is excluded from the measurement. [ArgumentsSource] methods return IEnumerable<T> for single-parameter benchmarks, or IEnumerable<object[]> for multi-parameter benchmarks (each array is one set of arguments). [ParamsSource] methods return IEnumerable<T> (each element is a single value assigned to the field/property).

// Single-parameter: IEnumerable<T>
[Benchmark]
[ArgumentsSource(nameof(Inputs))]
public int Parse(string text) => int.Parse(text);

public IEnumerable<string> Inputs()
{
    yield return "12345";
    yield return "999999999";
    yield return "-42";
}

// Multi-parameter: IEnumerable<object[]>
[Benchmark]
[ArgumentsSource(nameof(Inputs))]
public bool TryParse(string text, NumberStyles style) => int.TryParse(text, style, null, out _);

public IEnumerable<object[]> Inputs()
{
    yield return new object[] { "12345", NumberStyles.Integer };
    yield return new object[] { "0xFF", NumberStyles.HexNumber };
}

[ParamsSource] works the same way but targets a field/property and applies to all benchmarks in the class:

[ParamsSource(nameof(Sizes))]
public int N;

public IEnumerable<int> Sizes() => new[] { 100, 1_000, 10_000 };

[GenericTypeArguments] varies type parameters on a generic benchmark class. Each attribute creates a separate closed generic type that BDN runs independently:

[GenericTypeArguments(typeof(int))]
[GenericTypeArguments(typeof(string))]
public class CollectionBenchmark<T>
{
    private List<T> _list;

    [GlobalSetup]
    public void Setup() => _list = new List<T>(Enumerable.Repeat(default(T)!, 1000));

    [Benchmark]
    public bool Contains() => _list.Contains(default!);
}

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
.NET (Microsoft)
Publisher
.NET (Microsoft) (Official)
Source
dotnet/skills · plugins/dotnet-diag/skills/microbenchmarking · commit 96c3d68
Licence
MIT License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by .NET (Microsoft) or .NET (Microsoft).

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download microbenchmarking.zip