Nemo Mbridge Perf Moe Dispatcher Selection

NVIDIAOfficialPublished by NVIDIA

Skill
Guidelines
Software development
Apache-2.0
Medium risk

From the author

Choose the right MoE token dispatcher (`alltoall`, DeepEP, or HybridEP) for the hardware, EP degree, and optimization stage. Summarizes patterns from DSV3, Qwen3, Qwen3-Next, and VLM bring-up work.

— the description by NVIDIA

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured approach to selecting the optimal Mixture-of-Experts (MoE) token dispatcher for large-scale training workloads. It outlines decision criteria based on hardware architecture (H100, B200, GB200/300), Expert Parallelism (EP) degree, and specific model-family patterns. The guide emphasizes the importance of environment validation, proper SM tuning, and maintaining consistent routing modes when benchmarking different dispatcher backends.

What it does

Provides a decision matrix for choosing between alltoall, DeepEP, and HybridEP dispatchers.
Outlines hardware-specific recommendations for Hopper and Blackwell-class systems.
Details tuning parameters for DeepEP and HybridEP, including SM allocation.
Explains the necessity of verifying backend package availability before benchmarking.
Highlights key interactions with CUDA graphs, routing modes, and pipeline layouts.
Identifies common pitfalls when comparing dispatcher performance across different software stacks.

When to use it

When initially configuring MoE training for a new model or hardware target.
When troubleshooting performance regressions in MoE communication.
When optimizing memory headroom or throughput for large-scale MoE models.
When preparing to benchmark different dispatcher backends to ensure valid comparisons.

Advantages

Reduces trial-and-error by providing evidence-based starting points for dispatcher selection.
Ensures performance comparisons are scientifically valid by highlighting critical control variables.
Provides actionable tuning parameters for specific communication kernels.
Helps identify environment-related failures versus actual performance bottlenecks.

Usage examples

“Which MoE dispatcher should I use for a Qwen3 235B model on GB200 hardware?”: The guide recommends HybridEP as the target for the strongest steady-state results on GB200, noting it often provides better memory headroom than alltoall.
“How do I tune DeepEP performance?”: The guide suggests setting moe_token_dispatcher_type to flex, moe_flex_dispatcher_backend to deepep, and tuning the moe-deepep-num-sms parameter, starting with a default of 20.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/nvidia/nemo-mbridge-perf-moe-dispatcher-selection/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip nemo-mbridge-perf-moe-dispatcher-selection.zip -d .agents/skills

Documentation by NVIDIA

MoE Dispatcher Selection Guide

Stable docs: @docs/training/moe-optimization.md Card: @skills/nemo-mbridge-perf-moe-dispatcher-selection/card.yaml

Quick Decision

By hardware

Hardware	First choice	Why
H100	DeepEP, if the runtime package is installed	Strong default for cross-node EP on Hopper
B200	DeepEP, if the runtime package is installed	Good first choice unless a platform-specific HybridEP path is available
GB200 / GB300 NVL72	HybridEP, if the runtime package is installed	Best fit for NVLink-domain-aware dispatch and lower memory pressure
Unknown or first bring-up	`alltoall`	Easiest path for correctness and debugging

By EP degree

EP size	Guidance
Small EP	Dispatcher choice is usually second-order; start with `alltoall` or DeepEP
Medium EP	DeepEP often becomes worthwhile
Large EP	HybridEP is usually the best target on NVL72 systems

Model-Family Patterns

Workload	Common best path	Notes
DSV3 at large scale	HybridEP on GB200 or GB300, DeepEP on H100	Dispatcher choice matters more as EP and PP both grow
Qwen3 235B	DeepEP on H100, HybridEP on GB200	HybridEP usually wins on GB200 and often uses less memory
Qwen3 30B	DeepEP	Smaller models still benefit, but the absolute gap is smaller
Qwen3-Next	Close race in BF16, HybridEP stronger in FP8 or memory-tight runs	Good reminder to test, not assume
MoE VLMs	Start simple, then test HybridEP on GB200-class systems	Vision workloads are sensitive to both memory and host overhead

Rounded Evidence Summary

Backend availability gate

Do not interpret a dispatcher timing until the container has proven that the selected backend package is available. --moe_flex_dispatcher_backend None selects the standard alltoall dispatcher, while deepep and hybridep select moe_token_dispatcher_type="flex" and then require their corresponding runtime packages at model construction time. If DeepEP or HybridEP is missing, record the import failure as an environment limitation and treat alltoall as the only measured correctness fallback for that run.

Qwen3 30B A3B on H100

A short 2026-05-17 H100 smoke run used Qwen3 30B A3B BF16, 16 GPUs, EP=16, the recipe's Transformer Engine CUDA graph scopes (moe_router, moe_preprocess), and model.moe_permute_fusion=false due to a Triton JIT compatibility issue in the run container. The alltoall fallback completed five steps with 45.65 s mean step time after warmup, 132.9 mean TFLOP/s/GPU after warmup, final loss 11.44050, and 61.351 GB peak max allocated memory. DeepEP and HybridEP selected the requested flex backend in the dumped configs but failed before the first iteration because the packages were not installed. This confirms the availability gate; it is not a throughput ranking for flex dispatchers on H100.

DSV3 on GB200 or GB300

The broad trend is more important than any single row in the tracker:

plain alltoall is usually the conservative baseline
DeepEP improves that baseline once EP communication becomes visible
HybridEP adds another step up on NVL72 systems, especially after CUDA graphs, routing improvements, and CPU-side cleanup are already in place

In practice, the stack often moves from roughly "low-teens MFU" territory with an untuned baseline into "high-teens to low-20s MFU" territory after the full dispatcher and kernel stack is tuned.

Qwen3 235B on GB200

For Qwen3 235B, the practical ordering is usually:

alltoall for initial bring-up
DeepEP if you want a familiar tuned path
HybridEP for the strongest steady-state result on GB200

HybridEP is usually modestly faster than alltoall on this workload and often has noticeably better memory headroom.

Qwen3-Next on GB200

This family is a good reminder that dispatcher wins are workload-dependent:

in BF16, alltoall and HybridEP can be close
in FP8 or memory-constrained settings, HybridEP tends to look better
pipeline layout and grouped-GEMM changes can matter almost as much as the dispatcher itself

Tuning Parameters

DeepEP

DeepEP is selected by setting moe_token_dispatcher_type="flex" and moe_flex_dispatcher_backend="deepep".

--moe-deepep-num-sms 20

Tune the SM count allocated to DeepEP communication kernels (default 20). The optimal value depends on the workload and EP degree. First confirm the DeepEP package imports in the target container; a missing package fails during model construction, before any dispatcher timing is available.

HybridEP

HybridEP is selected by setting moe_token_dispatcher_type="flex" and moe_flex_dispatcher_backend="hybridep".

--moe-hybridep-num-sms 16

Tune the SM count allocated to HybridEP communication (default 16). The performance harness uses 32 for HybridEP workloads. Sweep between 16 and 32 for the target hardware. Set NUM_OF_HYBRID_EP_RANKS_PER_NVLINK_DOMAIN to match the NVLink domain size of the deployment. If it does not match the actual topology, performance and sometimes correctness will suffer. First confirm the HybridEP package imports in the target container; a missing package fails during model construction, before any dispatcher timing is available.

Routing mode

--moe-router-force-load-balancing

For performance benchmarking, force-balance routing is the safer default. It usually outperforms dropless routing in large-scale benchmarks and makes results more comparable across dispatcher backends.

Key Interactions

Feature	Interaction
CUDA graphs	Best paired with `attn moe_router moe_preprocess` on dropless MoE
EP overlap	Helps when dispatcher time is still visible after backend tuning
FP8	Often increases the relative importance of communication and host overhead
CPU affinity	Can matter as much as dispatcher choice on GB200 or GB300
Pipeline layout	Poor PP or VPP layout can erase dispatcher gains

When To Use Each

`alltoall`

first correctness bring-up
small EP configurations
debugging communication regressions

DeepEP

Hopper or B200 deployments
cross-node EP is clearly visible in profiles
you want a mature intermediate step before testing HybridEP

HybridEP

GB200 or GB300 NVL72 systems
large EP degrees
memory headroom matters in addition to throughput

Pitfalls

Do not compare dispatchers on different stacks: container, routing mode, PP layout, and CUDA-graph scope can move the result as much as the dispatcher.
HybridEP is topology-sensitive: it is not a universal win outside the hardware it was designed for.
Both dispatchers need SM tuning: default moe_deepep_num_sms (20) and moe_hybridep_num_sms (16) are reasonable starting points but rarely optimal.
Force-balance and dropless are not interchangeable baselines: keep the routing mode fixed when comparing dispatcher backends.
Memory and throughput can trade off differently by model: Qwen3-style runs may show a smaller speed delta than DSV3, but still justify HybridEP for memory headroom.
Backend import failures are not performance data: if DeepEP or HybridEP is missing from the container, do not compare its failed job against a completed alltoall job. Fix the environment first, then rerun the same stack.

Files

ATTRIBUTION.mdAdded by AutoGineer883 B
BENCHMARK.md4.3 KiB
LICENSELicence file added by AutoGineer11.2 KiB
SKILL.md7.8 KiB
card.yaml4.0 KiB
evals/evals.json1.5 KiB
skill-card.md3.9 KiB
skill.oms.sig4.7 KiB

LICENSE

Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.


                                 Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

   1. Definitions.

      "License" shall mean the terms and conditions for use, reproduction,
      and distribution as defined by Sections 1 through 9 of this document.

      "Licensor" shall mean the copyright owner or entity authorized by
      the copyright owner that is granting the License.

      "Legal Entity" shall mean the union of the acting entity and all
      other entities that control, are controlled by, or are under common
      control with that entity. For the purposes of this definition,
      "control" means (i) the power, direct or indirect, to cause the
      direction or management of such entity, whether by contract or
      otherwise, or (ii) ownership of fifty percent (50%) or more of the
      outstanding shares, or (iii) beneficial ownership of such entity.

      "You" (or "Your") shall mean an individual or Legal Entity
      exercising permissions granted by this License.

      "Source" form shall mean the preferred form for making modifications,
      including but not limited to software source code, documentation
      source, and configuration files.

      "Object" form shall mean any form resulting from mechanical
      transformation or translation of a Source form, including but
      not limited to compiled object code, generated documentation,
      and conversions to other media types.

      "Work" shall mean the work of authorship, whether in Source or
      Object form, made available under the License, as indicated by a
      copyright notice that is included in or attached to the work
      (an example is provided in the Appendix below).

      "Derivative Works" shall mean any work, whether in Source or Object
      form, that is based on (or derived from) the Work and for which the
      editorial revisions, annotations, elaborations, or other modifications
      represent, as a whole, an original work of authorship. For the purposes
      of this License, Derivative Works shall not include works that remain
      separable from, or merely link (or bind by name) to the interfaces of,
      the Work and Derivative Works thereof.

      "Contribution" shall mean any work of authorship, including
      the original version of the Work and any modifications or additions
      to that Work or Derivative Works thereof, that is intentionally
      submitted to Licensor for inclusion in the Work by the copyright owner
      or by an individual or Legal Entity authorized to submit on behalf of
      the copyright owner. For the purposes of this definition, "submitted"
      means any form of electronic, verbal, or written communication sent
      to the Licensor or its representatives, including but not limited to
      communication on electronic mailing lists, source code control systems,
      and issue tracking systems that are managed by, or on behalf of, the
      Licensor for the purpose of discussing and improving the Work, but
      excluding communication that is conspicuously marked or otherwise
      designated in writing by the copyright owner as "Not a Contribution."

      "Contributor" shall mean Licensor and any individual or Legal Entity
      on behalf of whom a Contribution has been received by Licensor and
      subsequently incorporated within the Work.

   2. Grant of Copyright License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      copyright license to reproduce, prepare Derivative Works of,
      publicly display, publicly perform, sublicense, and distribute the
      Work and such Derivative Works in Source or Object form.

   3. Grant of Patent License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)
      with the Work to which such Contribution(s) was submitted. If You
      institute patent litigation against any entity (including a
      cross-claim or counterclaim in a lawsuit) alleging that the Work
      or a Contribution incorporated within the Work constitutes direct
      or contributory patent infringement, then any patent licenses
      granted to You under this License for that Work shall terminate
      as of the date such litigation is filed.

   4. Redistribution. You may reproduce and distribute copies of the
      Work or Derivative Works thereof in any medium, with or without
      modifications, and in Source or Object form, provided that You
      meet the following conditions:

      (a) You must give any other recipients of the Work or
          Derivative Works a copy of this License; and

      (b) You must cause any modified files to carry prominent notices
          stating that You changed the files; and

      (c) You must retain, in the Source form of any Derivative Works
          that You distribute, all copyright, patent, trademark, and
          attribution notices from the Source form of the Work,
          excluding those notices that do not pertain to any part of
          the Derivative Works; and

      (d) If the Work includes a "NOTICE" text file as part of its
          distribution, then any Derivative Works that You distribute must
          include a readable copy of the attribution notices contained
          within such NOTICE file, excluding those notices that do not
          pertain to any part of the Derivative Works, in at least one
          of the following places: within a NOTICE text file distributed
          as part of the Derivative Works; within the Source form or
          documentation, if provided along with the Derivative Works; or,
          within a display generated by the Derivative Works, if and
          wherever such third-party notices normally appear. The contents
          of the NOTICE file are for informational purposes only and
          do not modify the License. You may add Your own attribution
          notices within Derivative Works that You distribute, alongside
          or as an addendum to the NOTICE text from the Work, provided
          that such additional attribution notices cannot be construed
          as modifying the License.

      You may add Your own copyright statement to Your modifications and
      may provide additional or different license terms and conditions
      for use, reproduction, or distribution of Your modifications, or
      for any such Derivative Works as a whole, provided Your use,
      reproduction, and distribution of the Work otherwise complies with
      the conditions stated in this License.

   5. Submission of Contributions. Unless You explicitly state otherwise,
      any Contribution intentionally submitted for inclusion in the Work
      by You to the Licensor shall be under the terms and conditions of
      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

   6. Trademarks. This License does not grant permission to use the trade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.

   7. Disclaimer of Warranty. Unless required by applicable law or
      agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
      implied, including, without limitation, any warranties or conditions
      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
      PARTICULAR PURPOSE. You are solely responsible for determining the
      appropriateness of using or redistributing the Work and assume any
      risks associated with Your exercise of permissions under this License.

   8. Limitation of Liability. In no event and under no legal theory,
      whether in tort (including negligence), contract, or otherwise,
      unless required by applicable law (such as deliberate and grossly
      negligent acts) or agreed to in writing, shall any Contributor be
      liable to You for damages, including any direct, indirect, special,
      incidental, or consequential damages of any character arising as a
      result of this License or out of the use or inability to use the
      Work (including but not limited to damages for loss of goodwill,
      work stoppage, computer failure or malfunction, or any and all
      other commercial damages or losses), even if such Contributor
      has been advised of the possibility of such damages.

   9. Accepting Warranty or Additional Liability. While redistributing
      the Work or Derivative Works thereof, You may choose to offer,
      and charge a fee for, acceptance of support, warranty, indemnity,
      or other liability obligations and/or rights consistent with this
      License. However, in accepting such obligations, You may act only
      on Your own behalf and on Your sole responsibility, not on behalf
      of any other Contributor, and only if You agree to indemnify,
      defend, and hold each Contributor harmless for any liability
      incurred by, or claims asserted against, such Contributor by reason
      of your accepting any such warranty or additional liability.

   END OF TERMS AND CONDITIONS

   APPENDIX: How to apply the Apache License to your work.

      To apply the Apache License to your work, attach the following
      boilerplate notice, with the fields enclosed by brackets "[]"
      replaced with your own identifying information. (Don't include
      the brackets!)  The text should be enclosed in the appropriate
      comment syntax for the file format. We also recommend that a
      file or class name and description of purpose be included on the
      same "printed page" as the copyright notice for easier
      identification within third-party archives.

   Copyright [yyyy] [name of copyright owner]

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Node.js

Security

What our scanner noticed· Medium risk

Large base64-encoded blobs

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

NVIDIA

Publisher

NVIDIA (Official)

Source

NVIDIA/skills · skills/nemo-mbridge-perf-moe-dispatcher-selection · commit fce3cca

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE-APACHE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by NVIDIA or NVIDIA.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download nemo-mbridge-perf-moe-dispatcher-selection.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by NVIDIA

MoE Dispatcher Selection Guide

Quick Decision

By hardware

By EP degree

Model-Family Patterns

Rounded Evidence Summary

Backend availability gate

Qwen3 30B A3B on H100

DSV3 on GB200 or GB300

Qwen3 235B on GB200

Qwen3-Next on GB200

Tuning Parameters

DeepEP

HybridEP

Routing mode

Key Interactions

When To Use Each

alltoall

DeepEP

HybridEP

Pitfalls

Files

LICENSE

Requirements and compatibility

Security

Attribution and licence

Related skills

Setup TS Deep Modules

To Spec

Improve Codebase Architecture

Domain Modeling

Codebase Design

Prototype

Download ZIP

`alltoall`