AutoGineer
Generated banner for Nemo Mbridge Perf Tp Dp Comm Overlap
Generated logo for Nemo Mbridge Perf Tp Dp Comm Overlap

Nemo Mbridge Perf Tp Dp Comm Overlap

NVIDIAOfficialPublished by NVIDIA

  • Skill
  • Tool integration
  • Software development
  • Apache-2.0
  • Medium risk
Download ZIP

From the author

Operational guide for enabling TP, DP, and PP communication overlap in Megatron-Bridge, including config knobs, code anchors, pitfalls, and verification.
— the description by NVIDIA

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a comprehensive guide for enabling and tuning communication overlap (TP, DP, and PP) within the Megatron-Bridge framework. It covers essential configuration knobs, code-level gating mechanisms, common pitfalls to avoid, and verification procedures to ensure optimal performance during large-scale model training.

What it does

  • Provides configuration templates for enabling TP, DP, and PP communication overlap.
  • Identifies key code anchors that control overlap gating and P2P communication logic.
  • Highlights common pitfalls regarding sequence parallelism, Transformer Engine dependencies, and bucket sizing.
  • Details environment variable tuning for launch-time performance optimization.
  • Outlines verification steps using unit tests to confirm overlap configuration success.

When to use it

  • When configuring communication overlap for Megatron-Bridge training jobs.
  • When investigating throughput regressions related to communication overlap settings.
  • When troubleshooting why TP or PP overlap features are not activating as expected.
  • When tuning environment variables like CUDA_DEVICE_MAX_CONNECTIONS for performance.

Advantages

  • Centralizes configuration knowledge for complex parallel training setups.
  • Provides clear guidance on the interaction between model parallelism and overlap features.
  • Includes specific verification commands to validate configuration integrity.
  • Warns against common misconfigurations in mixed precision and environment settings.

Usage examples

“How do I enable TP communication overlap in my Megatron-Bridge config?”
The skill provides the CommOverlapConfig snippet to set tp_comm_overlap=True and notes the requirement for sequence_parallel=True and Transformer Engine availability.
“Why is my PP overlap not working?”
The guide explains that PP overlap is only auto-selected when pipeline_model_parallel_size > 1 and virtual_pipeline_model_parallel_size > 1.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/nvidia/nemo-mbridge-perf-tp-dp-comm-overlap/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip nemo-mbridge-perf-tp-dp-comm-overlap.zip -d .agents/skills

Documentation by NVIDIA

TP / DP / PP Communication Overlap Skill

For stable background and recommendation level, see:

  • @docs/training/communication-overlap.md

Enablement

Minimal Bridge override:

from megatron.bridge.training.comm_overlap import CommOverlapConfig

cfg.model.tensor_model_parallel_size = 4
cfg.model.sequence_parallel = True
cfg.model.pipeline_model_parallel_size = 4
cfg.model.virtual_pipeline_model_parallel_size = 2

cfg.comm_overlap = CommOverlapConfig(
    tp_comm_overlap=True,
)

cfg.ddp.use_distributed_optimizer = True
cfg.ddp.overlap_grad_reduce = True
cfg.ddp.overlap_param_gather = True

Optional TP preset:

from megatron.bridge.training.comm_overlap import userbuffers_bf16_h100_h12288_tp4_mbs1_seqlen2048

cfg.comm_overlap.tp_comm_overlap_cfg = userbuffers_bf16_h100_h12288_tp4_mbs1_seqlen2048

Precision knobs belong to mixed precision:

cfg.mixed_precision.grad_reduce_in_fp32 = False
cfg.mixed_precision.fp8_param_gather = False

Code Anchors

Bridge overlap gating:

if self.user_comm_overlap_cfg.tp_comm_overlap is True:
    if model_cfg.tensor_model_parallel_size < 2:
        ...
    elif not model_cfg.sequence_parallel:
        ...
    elif not HAVE_TE:
        ...

PP overlap selection:

if model_cfg.pipeline_model_parallel_size > 1:
    if vp_size > 1:
        comm_overlap_cfg.overlap_p2p_comm = True
        comm_overlap_cfg.batch_p2p_comm = False
    else:
        comm_overlap_cfg.overlap_p2p_comm = False
        comm_overlap_cfg.batch_p2p_comm = True

DP overlap defaults:

if self.data_parallel_size > 1:
    comm_overlap_cfg.bucket_size = 128 * 1024 * 1024
    comm_overlap_cfg.overlap_grad_reduce = True
    comm_overlap_cfg.overlap_param_gather = True

Launch-time env tuning:

executor.env_vars["CUDA_DEVICE_MAX_CONNECTIONS"] = str(cuda_device_max_connections)
...
executor.env_vars["NVTE_FWD_LAYERNORM_SM_MARGIN"] = str(self.layernorm_sm_margin)
executor.env_vars["NVTE_BWD_LAYERNORM_SM_MARGIN"] = str(self.layernorm_sm_margin)

Pitfalls

  1. TP overlap silently disables itself if sequence_parallel=False or Transformer Engine is unavailable.

  2. PP overlap is not enabled for all PP cases. Bridge only auto-selects overlap_p2p_comm=True when PP > 1 and VPP > 1.

  3. bucket_size is a parameter-count knob, not a byte-size knob.

  4. grad_reduce_in_fp32 and fp8_param_gather should be set through mixed precision, not as standalone DDP tuning first.

  5. CUDA_DEVICE_MAX_CONNECTIONS and LayerNorm SM margin are launch-time plugin settings, not CommOverlapConfig fields.

Verification

Use the checked-in overlap unit coverage first:

uv run python -m pytest tests/unit_tests/training/test_comm_overlap.py -q

Optional second check if nemo_run is available:

uv run python -m pytest tests/unit_tests/recipes/test_run_plugins.py -q

Success criteria:

  • first command reports 26 passed

  • second command validates plugin-owned env wiring when not skipped

Files

skill-card.md

Description: <br>

Operational guide for enabling TP, DP, and PP communication overlap in Megatron-Bridge, including config knobs, code anchors, pitfalls, and verification. <br>

This skill is ready for commercial/non-commercial use. <br>

Owner

NVIDIA <br>

License/Terms of Use: <br>

Apache 2.0 <br>

Use Case: <br>

Developers and engineers enabling TP, DP, and PP communication overlap in Megatron-Bridge training configurations to maximize training throughput on NVIDIA GPUs. <br>

Deployment Geography for Use: <br>

Global <br>

Known Risks and Mitigations: <br>

Risk: Review before execution as proposals could introduce incorrect or misleading guidance into skills. <br> Mitigation: Review and scan skill before deployment. <br>

Reference(s): <br>

Skill Output: <br>

Output Type(s): [Configuration instructions, Shell commands] <br> Output Format: [Markdown with inline Python and bash code blocks] <br> Output Parameters: [1D] <br> Other Properties Related to Output: [None] <br>

Evaluation Agents Used: <br>

  • Claude Code (claude-code) <br>

  • Codex (codex) <br>

Evaluation Tasks: <br>

Evaluated against 1 internal skill evaluation task with 2 attempts per task. <br>

Evaluation Metrics Used: <br>

Reported benchmark dimensions: <br>

  • Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. <br>

  • Correctness: Checks whether the agent follows the expected workflow and produces the correct final output. <br>

  • Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant. <br>

  • Effectiveness: Checks whether the agent performs measurably better with the skill than without it. <br>

  • Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work. <br>

Underlying evaluation signals used in this run: <br>

  • security: Checks for unsafe operations, secret leakage, and unauthorized access. <br>

  • skill_execution: Verifies that the agent loaded the expected skill and workflow. <br>

  • skill_efficiency: Checks routing quality, decoy avoidance, and redundant tool usage. <br>

  • accuracy: Grades final-answer correctness against the reference answer. <br>

  • goal_accuracy: Checks whether the overall user task completed successfully. <br>

  • behavior_check: Verifies expected behavior steps, including safety expectations. <br>

  • token_efficiency: Compares token usage with and without the skill. <br>

Evaluation Results: <br>

DimensionNumclaude-codecodex
Security2100% (+0%)100% (+0%)
Correctness2100% (+0%)91% (+0%)
Discoverability2100% (+0%)66% (+0%)
Effectiveness297% (-1%)96% (+4%)
Efficiency293% (-0%)55% (+2%)

Skill Version(s): <br>

v0.2.0rc6 (source: git tag) <br>

Ethical Considerations: <br>

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal team to ensure this skill meets requirements for the relevant industry and use case and addresses unforeseen product misuse. <br>

(For Release on NVIDIA Platforms Only) <br> Please report quality, risk, security vulnerabilities or NVIDIA AI Concerns here. <br>

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Medium risk

  • Large base64-encoded blobs

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
NVIDIA
Publisher
NVIDIA (Official)
Source
NVIDIA/skills · skills/nemo-mbridge-perf-tp-dp-comm-overlap · commit fce3cca
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE-APACHE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by NVIDIA or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download nemo-mbridge-perf-tp-dp-comm-overlap.zip