Hyperpod Performance Debugger
AWS LabsOfficialPublished by AWS Labs
- Skill
- Research and analysis
- Cloud and DevOps
- Apache-2.0
- Low risk
- Has scripts
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
The HyperPod Performance Debugger is a read-only diagnostic tool designed to identify performance degradation in Amazon SageMaker HyperPod clusters. It focuses on two primary issues: uneven NCCL bandwidth across nodes and poor filesystem performance. The skill collects host-side telemetry, including GPU health, EFA connectivity, and filesystem utilization, to provide actionable insights. It acts as a triage layer, surfacing critical signals and routing complex hardware or configuration issues to specialized sibling skills for remediation.
What it does
- Collects host-side performance snapshots including GPU, EFA, and filesystem metrics.
- Identifies straggler nodes and uneven NCCL bandwidth across cluster nodes.
- Detects filesystem bottlenecks and high I/O wait times on FSx mounts.
- Surfaces hardware-level signals like Xid errors, ECC counts, and NVLink status.
- Provides clear routing paths to specialized debugging skills for hardware, versioning, or configuration issues.
- Recommends built-in HyperPod health features for ongoing cluster monitoring.
When to use it
- When training jobs exhibit uneven performance across different nodes.
- When data loading or checkpointing operations are causing significant training stalls.
- When you suspect network topology mismatches or cross-AZ latency issues.
- When you need to verify host-side health signals before escalating to support.
Advantages
- Provides a standardized, read-only diagnostic workflow for complex cluster environments.
- Reduces mean time to resolution by automatically routing issues to the correct specialized skill.
- Helps distinguish between application-level I/O patterns and infrastructure-level bottlenecks.
- Integrates with native HyperPod health features to prevent future regressions.
Usage examples
- “My training job is running slower on some nodes than others. How do I check for NCCL issues?”
- The agent will guide you to run the performance snapshot script and analyze the output for pairwise bandwidth variance or straggler nodes.
- “Training is stalling during checkpointing. Is my FSx filesystem saturated?”
- The agent will help you collect CloudWatch metrics and host-side iostat data to determine if the filesystem throughput is the bottleneck.
Install
Download the skill as a ZIP:
https://autogineer.com/agent-skills/awslabs/hyperpod-performance-debugger/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip hyperpod-performance-debugger.zip -d .agents/skills
Documentation by AWS Labs
HyperPod Performance Debugger
Uneven NCCL performance across nodes — workload faster on some node sets than others, pairwise bandwidth variance, suspected straggler.
Poor filesystem performance — training stalled on data loading, checkpoint save/load dominating step time, FSx throughput saturated.
Scope and delegation
Route findings outside the two in-scope scenarios to the owner skill below.
| Concern observed | Route to |
|---|---|
| GPU hardware fault, ECC, NVLink, Xid, DCGM diagnostics, drain/replace | hyperpod-node-debugger (§ F Hardware/Auto-Repair, § G GPU) |
Cannot allocate memory at os.fork(), root volume exhausted | hyperpod-node-debugger (§ I Resource Exhaustion) |
| NCCL timeouts, hangs, AllReduce stalls, EFA TCP fallback, RDMA memlock | hyperpod-nccl |
| EFA / NCCL / CUDA / NVIDIA driver version drift across nodes | hyperpod-version-checker |
| EFA self-referencing security-group rule missing — single node | hyperpod-node-debugger § A (EFA / Security Group) |
| EFA self-referencing security-group rule missing — cluster-wide | hyperpod-cluster-debugger § A (EFA Health Checks) |
| Slurm node state changes (drain / resume / reboot) | hyperpod-slurm-debugger |
| Diagnostic bundle for AWS Support | hyperpod-issue-report |
| Shell access on a node | hyperpod-ssm |
Operating policy
Read-only. Print commands the customer runs; do not execute commands that modify state.
Container vs host version comparisons go through
hyperpod-version-checker.Xid lines, ECC counts, NVLink lane state, and thermal readings get surfaced; the catalog and verdict live in
hyperpod-node-debugger§ G.
Workflow
Confirm the symptom is uneven NCCL or poor filesystem performance. If neither, route to the matching sibling skill above.
Run
scripts/perf-snapshot.sh(read-only) to gather host-side signals for the suspect node and FSx filesystems mounted on it.For each
[CONCERN]line in the script output, open the matching section below and read the supporting reference.After the per-incident diagnosis, recommend the HyperPod platform health features in § Continuous health coverage so the customer gets ongoing protection.
Step 1: Run the snapshot
bash scripts/perf-snapshot.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION>
# Scope to one suspect node:
bash scripts/perf-snapshot.sh --cluster <C> --region <R> --node <INSTANCE_ID>The script samples one node by default. It collects host-side data via hyperpod-ssm: nvidia-smi output (temperature, SM clocks, PCIe link width, ECC, NVLink, topo -m), recent dmesg Xid lines, EFA port state and fi_info provider visibility, EFA installer + kernel module versions, CPU governor, NVL72 Fabric Manager state, FSx CloudWatch utilization, df -h / lfs df -h per mount, host iowait, /dev/shm size, and root-volume usage. All read-only.
Tags: [OK] healthy · [CONCERN] signal worth investigating (carries a → pointer to the owner skill) · [INFO] informational.
Host vs container scope. The script runs on the host via SSM and reports host-scope values. Many setups ship the EFA / libfabric / OFI-NCCL / CUDA stack inside the training container by design — a host value of unknown is not by itself a defect. What matters for performance is the stack the workload actually uses. Verify versions inside the container (and across nodes) via hyperpod-version-checker before drawing conclusions.
Step 2: Match signal → section
| Observation | Section |
|---|---|
| Pairwise NCCL bandwidth varies across node pairs / suspected straggler | A: Uneven NCCL Performance |
| Nodes spread across AZs / network-node-layer labels / UltraServer boundaries | A |
| EFA port not ACTIVE on a node, missing OFI plugin, or FI provider not visible | A + route to hyperpod-node-debugger § A; hyperpod-version-checker for cross-node version compare |
iostat shows high iowait, FSx CloudWatch utilization sustained near 100% | B: Poor Filesystem Performance |
| DataLoader stalls, checkpoint dominates step time | B |
Xid line in dmesg, uncorrectable ECC, inactive NVLink lane, GPU ≥ 88°C | Route to hyperpod-node-debugger § G |
| Container vs host version drift suspected | Route to hyperpod-version-checker |
Cannot allocate memory at os.fork(), root volume full, OOM events | Route to hyperpod-node-debugger § I |
NCCL timeout, hang, TCP fallback (NET/OFI Using TCP), RDMA memlock | Route to hyperpod-nccl |
A: Uneven NCCL Performance
The customer reports identical training jobs running with different step times on different node sets, pairwise bandwidth variance, or some allocations consistently slower than others despite identical code.
Per the official troubleshooting guide, the common contributing factors are network topology differences between nodes (cross-AZ, cross-rack, cross-UltraServer), degraded EFA performance on some nodes, mixed instance types or generations within an instance group, and CPU frequency scaling differences.
Diagnostic pass (read-only)
The host-side data points — GPU thermal/ECC/PCIe/clocks, Xid, NVLink lanes, EFA port state and provider visibility, CPU governor, EFA/OFI/driver versions, nvidia-smi topo -m — are all collected by scripts/perf-snapshot.sh (Step 1 above). The script tags [CONCERN] with thresholds and emits routing pointers; rerun it per suspect node via --node <INSTANCE_ID>.
For driver / CUDA / NCCL / EFA / OFI version drift across nodes, run hyperpod-version-checker skill.
Pairwise NCCL bandwidth test
Run the standard nccl-tests recipes from awslabs/awsome-distributed-training. For an N-node cluster, run all-reduce across every pair and record busbw for each pair. Pairs more than ~5% below the run mean (the threshold the AWS validation script flags) are problematic candidates.
Expected busbw per SKU is published in the AI-on-HyperPod NCCL test guide. Benchmark the specific instance type before relying on a number.
Pairwise scripts, HyperPod topology surfaces (HyperPod API, EKS labels, Slurm topology.conf), and GB200 NVL72 specifics are in references/perf-details.md § Uneven NCCL.
Topology verification
HyperPod exposes topology through three operator-visible surfaces:
HyperPod API:
aws sagemaker describe-cluster-nodereturnsNodeDetails.Placement.AvailabilityZone/AvailabilityZoneIdandNodeDetails.UltraServerInfo.Id(UltraServer SKUs only).EKS labels:
topology.kubernetes.io/zone,topology.k8s.aws/network-node-layer-{1,2,3}(highest-numbered = closest to instance),topology.k8s.aws/ultraserver-id.Slurm: HyperPod auto-generates
topology.conf. Inspect viascontrol show topology.
Tightly coupled work shares the same AZ, the same highest-numbered network-node-layer label (EKS) or the same Slurm topology block, and — for NVL72 jobs — the same UltraServerInfo.Id / topology.k8s.aws/ultraserver-id. If the cluster is spread across AZs or layers, topology must be re-established at provisioning time. Route provisioning changes to hyperpod-cluster-debugger § B (Capacity & AZ).
B: Poor Filesystem Performance
The customer reports training bottlenecked on data loading, checkpoint save/load dominating step time, executables/scripts loading slowly, or iowait high.
Per the official troubleshooting guide, the resolution path follows this order:
Check CloudWatch metrics on the filesystem.
Check the provisioned performance configuration against workload requirements.
Investigate which operations are causing the I/O — workload demand vs inefficient pattern.
Consider upgrading provisioned performance.
Choose the filesystem type that matches the I/O pattern.
This skill covers steps 1–3. Steps 4–5 are customer decisions; surface the data and let the customer pick.
Diagnostic pass (read-only)
scripts/perf-snapshot.sh (Step 1 above) covers the on-node side of this pass: it discovers FSx mounts, calls aws cloudwatch get-metric-statistics on DataReadBytes and (for OpenZFS) FileServerDiskIopsUtilization, prints df -h for /fsx /opt/dlami/nvme /opt/sagemaker, runs lfs df -h per Lustre mount, and reports iostat iowait. It tags [CONCERN] when OpenZFS IOPS utilization sustains ≥ 80% or iowait > 20%.
For longer windows or additional metrics (DataWriteBytes, Lustre DiskIopsUtilization, OpenZFS FileServerDiskThroughputUtilization), drive the query directly:
aws cloudwatch get-metric-statistics --region <REGION> \
--namespace AWS/FSx --metric-name DataReadBytes \
--dimensions Name=FileSystemId,Value=<FSID> \
--start-time "$(date -u -d '3 hours ago' +%Y-%m-%dT%H:%M:%S)" \
--end-time "$(date -u +%Y-%m-%dT%H:%M:%S)" \
--period 60 --statistics Sum MaximumThe full per-filesystem-type metric catalog is in references/perf-details.md § Filesystem.
Branches
Provisioned capacity is saturated. CloudWatch utilization sustained near 100% across the workload window. Customer decision: scale up the filesystem.
FSx for Lustre throughput scales with
StorageCapacity × PerUnitStorageThroughput; capacity changes are non-disruptive.FSx for OpenZFS — increase provisioned IOPS or throughput.
I/O pattern is inefficient. CloudWatch shows headroom but the workload is still I/O-bound. Customer decision: change the application.
DataLoader: raise
num_workers, setpin_memory=True,persistent_workers=True.Checkpointing: use async + sharded (
torch.distributed.checkpoint.async_saveplus FSDPSHARDED_STATE_DICT).FULL_STATE_DICTserializes through rank 0 and is a frequent root cause.Small-file workloads: Lustre is optimized for large sequential I/O. For millions of small files, use WebDataset / tar shards, FSx for OpenZFS, or NVMe scratch.
Filesystem-selection guidance and the async-checkpoint pattern are in references/perf-details.md § Filesystem.
Continuous health coverage
Once the immediate incident is diagnosed, recommend HyperPod's built-in health features so problems are caught before the next training run rather than after another customer-reported regression.
Enable
NodeRecovery=Automaticon the cluster. The Health Monitoring Agent (HMA) continuously monitors GPU- and Trainium-based instances and marks instances unhealthy on detected failure. With auto-recovery enabled, HyperPod reboots or replaces the node — no operator intervention.Enable
OnStartDeepHealthCheckson every GPU instance group with both check categories:InstanceStress—stress-ngon CPU/memory/disk, GPU and PCI device count verification, DCGM level-4 diagnostics (memory test included), and EFA loopback bandwidth/latency.InstanceConnectivity— multi-node NCCL all-reduce.
Every newly provisioned or auto-replaced node passes the same hardware bar before accepting jobs.
Run on-demand deep health checks when this skill or any sibling surfaces a hardware concern but the cluster is mid-workload.
aws sagemaker start-cluster-health-checkruns the same checks against a specific instance group; nodes are placed in a Slurm maintenance reservation and the check is queued until any running job completes (not preempted). Console: HyperPod → Clusters → Instances → Run deep health checks.Not supported when
NodeProvisioningMode=Continuous; one on-demand request per cluster at a time. Requires the latest AMI — runUpdateClusterSoftwarefirst.
Logs land in CloudWatch at /aws/sagemaker/Clusters/<cluster_name>/<cluster_id> under DeepHealthCheckResults/<log_stream_id>, and on each node at /var/log/aws/clusters/sagemaker-deep-health-check.log.
References
references/perf-details.md — pairwise NCCL test recipes, HyperPod topology check, GB200 NVL72 placement; CloudWatch metric catalog per filesystem type, async-checkpoint pattern, filesystem selection guide.
External:
Amazon SageMaker HyperPod troubleshooting guide (official): https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md
AI-on-HyperPod NCCL performance tests (expected
busbwper SKU): https://awslabs.github.io/ai-on-sagemaker-hyperpod/docs/slurm-orchestration/validation-and-testing/performance-testing/nccl-testsAmazon SageMaker HyperPod resiliency (NodeRecovery, HMA, auto-resume): https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-resiliency.html
Amazon SageMaker HyperPod deep health checks: https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-resiliency-slurm-deep-health-checks.html
StartClusterHealthCheck API: https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_StartClusterHealthCheck.html
Amazon EC2 instance topology / network-node-layer labels: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/how-ec2-instance-topology-works.html
Amazon FSx for Lustre performance: https://docs.aws.amazon.com/fsx/latest/LustreGuide/performance.html
Amazon FSx for OpenZFS metrics: https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/fsx-openzfs-metrics.html
AWS Elastic Fabric Adapter and NCCL: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start-nccl.html
awslabs/awsome-distributed-training NCCL tests: https://github.com/awslabs/awsome-distributed-training/tree/main/micro-benchmarks/nccl-tests
Files
- ATTRIBUTION.mdAdded by AutoGineer942 B
- LICENSELicence file added by AutoGineer9.9 KiB
- NOTICELicence file added by AutoGineer89 B
- SKILL.md16.1 KiB
- references/perf-details.md11.8 KiB
- scripts/perf-snapshot.shScript27.8 KiB
LICENSE
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs a command-line tool, Needs Node.js
Security
What our scanner noticed· Low risk
- Scripts that your agent may run
This skill includes scripts that your agent may run. Read them before you use it.
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- AWS Labs
- Publisher
- AWS Labs (Official)
- Source
- awslabs/agent-plugins · plugins/sagemaker-ai/skills/hyperpod-performance-debugger · commit fbefb37
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- NOTICE (added by AutoGineer from NOTICE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by AWS Labs or AWS Labs.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download hyperpod-performance-debugger.zip