Msbuild

Diagnosing MSBuild Build Performance

Source: original build-perf-diagnostics skill.

Performance Analysis Methodology

1. Establish the relevant cold, warm changed-input, or no-change baseline. Keep Rebuild and ordinary Build distinct.

2. Use an existing matching binlog first. If one is missing, follow binlog generation without changing the reported scenario.

3. Replay the binary log locally to a diagnostic text log with a performance summary. The searches below assume that log is named full-01.log, as in the replay command; use its actual path.

4. Find expensive projects, targets, and tasks, then inspect the corresponding instance, execution reason, inputs, and dependency waits. Select one cause for a controlled change, not every plausible optimization.

Performance summaries report cumulative/inclusive durations, not additive shares of elapsed wall time or CPU usage. Parallel work overlaps; orchestration targets and tasks can include child work and waits.

In particular, ResolveProjectReferences can mostly represent waiting for dependent builds, while a yielded node performs other work. Do not optimize it solely because it tops the target summary. Follow the dependencies to actual Csc, ResolveAssemblyReference, Copy, or custom-task work. Even MSBuild task duration can include child builds rather than exclusive self-time.

Key Metrics and Thresholds

Use these as investigation hints, not universal budgets or proof of a cause:

• Build duration: historical rough examples are under 10 seconds for a small workload, 60 seconds for a medium one, and 5 minutes for a large one. Compare the repository's actual scenario and machine, not labels alone.

• Node utilization: activity below roughly 80% warrants checking dependency waits, limited parallel work, and resource contention. Text-summary totals alone cannot establish utilization.

• Single-target domination: investigate a target near or above 50% of elapsed build time, but separate inclusive waits and nested work.

• Analyzer versus compiler time: around 30% or more of Csc time deserves investigation. Analyzer timings can overlap; do not add them as wall time or remove diagnostics solely to reach a percentage.

• RAR time: over 5 seconds per project, especially over 15 seconds, merits examination of reference count, paths, and resolution work.

Common Bottlenecks

1. ResolveAssemblyReference (RAR) Slowness

• Symptoms: substantial ResolveAssemblyReference task time, for example over 5 seconds per affected project.

• Root causes: many assembly references, network-based paths, or expensive assembly search paths.

• Fixes: remove demonstrated unused references/search paths and avoid unnecessary network resolution, preserving all consumers.

• Diagnostic guardrail: DesignTimeBuild, ResolveAssemblyReferencesSilent, and ResolveAssemblyWarnOrErrorOnTargetArchitectureMismatch affect build or diagnostic behavior; changing them is not an established RAR performance fix.

• Key insight: RAR can legitimately execute on no-change builds to account for external resolution state, including installed targeting packs or GAC assemblies. Do not force it to skip using an incomplete freshness check.

• Reduce transitive references carefully: DisableTransitiveProjectReferences is a last resort requiring explicit references to consumed dependencies. ReferenceOutputAssembly="false" is appropriate when only build ordering is needed, but still retains that ordering. Trim only unused package references. Use the dependency-graph audit.

2. Roslyn Analyzers and Source Generators

• Symptoms: Csc takes much longer than a comparable compilation, sometimes more than twice a measured compiler-only scenario.

• Diagnosis: inspect Csc and analyzer/generator timing in the replayed log. If timing detail is absent, obtain compiler-supported diagnostic timing rather than guessing from package count. A controlled -p:RunAnalyzers=false comparison must compile the same inputs on both sides, not compare a no-op against forced compilation.

• Conditional development policy: after measuring and agreeing the policy, examples include:

  Copy

  <PropertyGroup>
    <RunAnalyzers Condition="'$(ContinuousIntegrationBuild)' != 'true'">false</RunAnalyzers>
    <EnforceCodeStyleInBuild Condition="'$(ContinuousIntegrationBuild)' == 'true'">true</EnforceCodeStyleInBuild>
  </PropertyGroup>

• Per-configuration: if development Debug builds are the intended scope, also guard the CI case, for example Condition="'$(Configuration)' == 'Debug' and '$(ContinuousIntegrationBuild)' != 'true'". CI can build Debug too.

• Rule/package scope: configure genuinely redundant or expensive diagnostics appropriately, including .editorconfig severity policy, instead of blindly removing analyzer packages.

• Key principle: preserve required analyzer enforcement in CI. Verify effective properties and that the pipeline actually sets the CI signal. A conditional assignment does not undo an earlier or later unconditional override.

• Source generators: do not assume RunAnalyzers=false disables or measures every generator. Preserve required generated code and inspect generator-specific evidence.

• GlobalPackageReference: examine the project, Directory.Build.props, Directory.Packages.props, and package imports. Active global analyzer references can affect production and test projects; a declaration alone does not prove restoration or inclusion in the compiler's analyzer list.

• EnforceCodeStyleInBuild: when enabled it adds code-style checks to compiler execution; it does not mean analyzers execute when compilation itself skips. Conditionalize only under an agreed development/CI policy.

3. Serialization Bottlenecks (Single-threaded Targets)

• Symptoms: one project dominates while other nodes are idle or waiting.

• Common culprits: a long dependency chain, a critical-path project, or custom orchestration issuing independent project requests sequentially.

• Fixes: optimize the measured critical path, remove false dependencies, or split a bottleneck only when it exposes independent work. In custom MSBuild tasks, BuildInParallel="true" also needs multiple available nodes.

• Follow parallel build tuning. More nodes cannot parallelize required serial dependencies.

4. Excessive File I/O (Copy Tasks)

• Symptoms: high Copy aggregate time or excessive copy invocations.

• Root causes: thousands of unnecessary files, network destinations, or accidental per-item task batching instead of an appropriate item batch.

• First fixes: inspect item metadata, source/destination paths, and invocation counts. Reduce only unnecessary copies and choose the correct copy mode; replacing Always blindly with PreserveNewest can leave mutated destination files stale.

• Skip unchanged files: inspect whether the actual target consumes SkipCopyUnchangedFiles; it may already be true. Custom Copy tasks expose SkipUnchangedFiles. Neither setting is a content-hash comparison.

• Hardlinks: identify the hot copy target in the binlog, then set only the property that target passes to Copy.UseHardlinksIfPossible: CreateHardLinksForCopyFilesToOutputDirectoryIfPossible for primary build outputs, CreateHardLinksForAdditionalFilesIfPossible for content/additional files, or CreateHardLinksForCopyLocalIfPossible for copy-local references. Custom Copy tasks need their own UseHardlinksIfPossible value. Hardlinks can help compatible same-volume, immutable-file workloads. A destination mutation through a hardlink can modify its source, so do not use this for writable/resettable content.

• Shared layout: UseCommonOutputDirectory=true assumes a genuinely shared, collision-free output layout. Verify ownership and consumers first. Artifacts output, including --artifacts-path, requires SDK 8+ and is a separate measured layout change.

• Dev Drive: on supported Windows development or self-hosted CI machines, evaluate Dev Drive for a measured I/O bottleneck. Do not promise automatic copy-on-write gains for every task or disable security tooling for a benchmark.

5. Evaluation Overhead

• Symptoms: significant startup/evaluation time before target execution.

• Root causes: expensive Directory.Build.props logic, globs traversing large trees, imports, or SDK-resolution overhead. NuGet-based SDK resolver latency depends on the toolset and environment; historical timings are not a fixed per-project cost.

• Fixes: simplify only measured costly logic and narrow offending globs. EnableDefaultItems=false is a last resort requiring complete intentional item lists, not the first optimization. Changing SDK resolution must preserve the build's actual SDK contract.

• Use evaluation performance for focused evidence and fixes.

6. NuGet Restore in Build

• Symptoms: restore contributes substantial work repeatedly. Distinguish an up-to-date check from actual repeated network/dependency-resolution work.

• Separate restore from build: successfully restore matching inputs, then use dotnet build --no-restore. Keep build-only and restore-inclusive baselines distinct; dependency changes require fresh assets.

• Static graph restore: RestoreUseStaticGraphEvaluation=true in Directory.Build.props is a candidate for a measured large restore graph. Validate equivalent resolved assets and toolset support. It is distinct from graph build and its benefit is workload-dependent.

• Investigate changing inputs and feed/cache behavior rather than purging shared package caches.

7. Large Project Count and Graph Shape

• Symptoms: many tiny projects add evaluation/invocation overhead, or deep dependency chains serialize otherwise parallelizable work.

• Consider: graph mode for scheduling, consolidating small leaves for measured overhead, or splitting a large bottleneck into independent pieces. These are different remedies for different evidence.

• Graph shape matters: wide independent branches expose parallelism; a long required chain remains serial regardless of node count.

• Actions: identify the exact unnecessary edge or costly project. Preserve direct API/metadata contracts and compare the resulting critical path. Static graph builds do not automatically provide isolation or a cache.

Using Binlog Replay for Performance Analysis

1. Replay the existing binlog with diagnostic verbosity and a performance summary.

2. Read the target/task summaries, usually near the end of the replayed log:

   Copy

   Select-String -Path .\full-01.log -Pattern 'Target Performance Summary|Task Performance Summary' -Context 0,50

   Use the cumulative timings to select candidates, not to compute additive percentages of elapsed build time.

3. Find per-project timings and instances:

   Copy

   Select-String -Path .\full-01.log -Pattern 'Done Building Project|Project Performance Summary' -Context 0,20

4. Inspect recorded scheduling evidence:

   Copy

   Select-String -Path .\full-01.log -Pattern 'node.*assigned|building with|scheduler'

   These messages alone are not a utilization measurement. Report missing scheduling/timestamp detail rather than inferring an exact timeline.

5. Check analyzer timing, when the original capture contains it:

   Copy

   Select-String -Path .\full-01.log -Pattern 'Total analyzer execution time|analyzer.*elapsed|CompilerAnalyzerDriver'

6. Drill into the relevant project's slow target and its tasks:

   Copy

   Select-String -Path .\full-01.log -Pattern 'Target "CoreCompile"|Target "ResolveAssemblyReferences"' -Context 0,20

Quick Wins Checklist

Choose only items supported by the diagnosis, one experiment at a time:

• [ ] Confirm the intended -m node budget rather than assuming host defaults.

• [ ] Separate restore and build only with current matching assets.

• [ ] Measure static graph restore and verify equivalent assets.

• [ ] Consider hardlinks only for compatible immutable-file workloads.

• [ ] Scope analyzer changes to an agreed development policy; retain CI enforcement.

• [ ] Inspect ProduceReferenceAssembly before enabling it; many SDK configurations already enable it. Verify implementation-only and public-API changes downstream.

• [ ] Check incremental behavior on an ordinary second build.

• [ ] Inspect evaluated output/intermediate paths for collisions between projects, configurations, TFMs, and RIDs; do not just give everything one common directory.

• [ ] Measure graph mode for a statically discoverable multi-project build.

• [ ] Evaluate SDK 8+ artifacts layout only with its path-consumer checks.

• [ ] Consider a separate Dev Drive environment experiment for Windows I/O limits.

Impact Categorization

• HIGH IMPACT: a verified critical-path cost above roughly 10% of elapsed build time, or substantial real work behind a target reported above 50%.

• MEDIUM IMPACT: verified costs around 2-10%.

• QUICK WINS: low-effort changes with demonstrated modest benefit, not unmeasured property toggles.

Report the project/TFM, target/task, evidence, one targeted change, and a same-scenario before/after comparison with spread. Verify required artifacts, diagnostics, and relevant tests/consumers. Stop if the result is unmeasured, within noise, or breaks correctness; do not claim success from disabled validation or inclusive timing arithmetic.