Hyperpod Issue Report

From the author

Generate comprehensive issue reports from HyperPod clusters (EKS and Slurm) by collecting diagnostic logs and configurations for troubleshooting and AWS Support cases. Use when users need to collect diagnostics from HyperPod cluster nodes, generate issue reports for AWS Support, investigate node failures or performance problems, document cluster state, or create diagnostic snapshots. Triggers on requests involving issue reports, diagnostic collection, support case preparation, or cluster troubleshooting that requires gathering logs and system information from multiple nodes.

— the description by AWS Labs

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

The HyperPod Issue Report skill provides a structured workflow to gather comprehensive diagnostic data from EKS and Slurm-based HyperPod clusters. It utilizes SSM to collect system logs, GPU status, and configuration files from cluster nodes in parallel, storing the results in an S3 bucket. The tool automatically detects the cluster type and supports targeted collection by instance group or specific node IDs, facilitating efficient troubleshooting and preparation for AWS Support cases.

What it does

Collects diagnostic logs and system configurations from HyperPod nodes via SSM.
Supports both EKS and Slurm cluster architectures with automatic detection.
Aggregates cluster-wide resource information and node-specific logs into S3.
Generates a summary report of collection success and failure statistics.
Provides EKS-specific cluster resource snapshots using kubectl.
Offers options to tune performance for large clusters by adjusting worker concurrency.

When to use it

When investigating node failures or performance degradation in a HyperPod cluster.
When preparing diagnostic data for an AWS Support case.
When you need to document the current state of a cluster for auditing or troubleshooting.
When you need to collect logs from multiple nodes simultaneously to identify patterns.

Advantages

Reduces manual effort by automating log collection across large clusters.
Ensures consistent diagnostic data capture for both EKS and Slurm environments.
Provides configurable parallel execution to handle clusters of varying sizes.
Centralizes diagnostic artifacts in S3 for easy sharing and analysis.

Usage examples

“Generate an issue report for my HyperPod cluster named my-cluster in us-west-2 and save it to s3://my-diagnostic-bucket/reports.”: The agent will verify the cluster, ensure the S3 bucket exists, and run the collection script to gather logs from all nodes, outputting a summary and the S3 path.
“Collect diagnostics only for the worker-group instance group in my cluster.”: The agent will execute the collection script with the --instance-groups flag, limiting the diagnostic gathering to nodes within that specific group.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/awslabs/hyperpod-issue-report/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip hyperpod-issue-report.zip -d .agents/skills

Documentation by AWS Labs

Collect diagnostic logs from HyperPod cluster nodes via SSM, store results in S3. Supports both EKS and Slurm clusters with auto-detection. Uses the bundled scripts/hyperpod_issue_report.py for reliable parallel collection.

Prerequisites

AWS CLI configured with permissions: sagemaker:DescribeCluster, sagemaker:ListClusterNodes, ssm:StartSession, s3:PutObject, s3:GetObject, eks:DescribeCluster
Python 3.8+ and uv (see uv installation docs for install options)
SSM Agent running on target nodes; node IAM roles need s3:GetObject/s3:PutObject on the report bucket
For EKS clusters: kubectl installed and configured (see Workflow step 2)

Workflow

1. Gather Information

Collect from the user:

Cluster identifier (required): accepts cluster name or full cluster ARN (e.g., arn:aws:sagemaker:us-west-2:123456789012:cluster/abc123)
AWS region (required unless extractable from ARN)
S3 path for report storage (required, e.g. s3://bucket/prefix). If the user doesn't have a bucket, create one (e.g., s3://hyperpod-diagnostics-<account-id>-<region>)
Issue description (optional)
Target scope: all nodes, specific instance groups, or specific node IDs (optional)

2. Verify Environment

aws sts get-caller-identity
aws sagemaker describe-cluster --cluster-name <name-or-arn> --region <region>

If the S3 bucket doesn't exist, create it:

aws s3 mb s3://<bucket-name> --region <region>

For EKS clusters (check Orchestrator.Eks in describe-cluster output):

Ensure kubectl is installed (which kubectl). If missing, install it for the current platform.
Configure kubeconfig using the EKS cluster name from the describe-cluster response:
```
aws eks update-kubeconfig --name <eks-cluster-name> --region <region>
```

3. Run the Collection Script

uv run scripts/hyperpod_issue_report.py \
  --cluster <cluster-name-or-arn> \
  --region <region> \
  --s3-path s3://<bucket>[/prefix]

Use --help for all options including --instance-groups, --nodes, --max-workers, and --debug. Note: --instance-groups and --nodes are mutually exclusive. Node identifiers accept instance IDs (i-*), EKS names (hyperpod-i-*), or Slurm names (ip-*).

4. Present Results

After collection, the script shows statistics and offers interactive download. Report the S3 location and offer to:

Download the report locally
Help analyze collected diagnostics (see references/collection-details.md for what's in each file)
Prepare a summary for AWS Support

Troubleshooting

See references/troubleshooting.md for error handling, large cluster tuning, and known limitations.

Files

ATTRIBUTION.mdAdded by AutoGineer926 B
LICENSELicence file added by AutoGineer9.9 KiB
NOTICELicence file added by AutoGineer89 B
SKILL.md3.5 KiB
references/collection-details.md3.6 KiB
references/troubleshooting.md3.5 KiB
scripts/hyperpod_issue_report.pyScript65.3 KiB

references/troubleshooting.md

Troubleshooting

Error Handling

Issue	Cause	Fix
`kubectl not found in PATH`	kubectl not installed	Install kubectl for the current platform, then re-run
`kubectl must be configured for EKS clusters`	kubectl missing or wrong context	Run `aws eks update-kubeconfig --name <eks-cluster-name> --region <region>`. Get the EKS cluster name from `aws sagemaker describe-cluster` output (`Orchestrator.Eks.ClusterArn`)
Cluster name from ARN not found	ARN contains cluster ID, not name	Pass the full ARN to `--cluster` instead of extracting the ID portion. Alternatively, use `aws sagemaker list-clusters` to find the cluster name
No instance reports in S3	Node IAM role missing S3 permissions	Add `s3:GetObject`/`s3:PutObject` to node role for the report bucket
SSM connectivity failed	SSM agent down, missing IAM, or network	Check `systemctl status amazon-ssm-agent`
"Failed to detect shell prompt"	Custom SSM session config (custom `.bashrc`, SSM preferences)	Not compatible without modifying prompt detection; use manual SSM sessions as workaround
SSM throttling	Too many concurrent sessions	Reduce `--max-workers`; automatic retry handles transient throttling
Nodes unresponsive	Node completely down	Noted in report; other nodes' diagnostics may reveal pattern
EKS log collector fails	Script download or execution error	Check `eks-log-collector-output.txt`; verify disk space in `/var/log/` and `/tmp/`

Large Cluster Handling

Default --max-workers 16 tested up to 130 nodes (99.2% success rate, ~15 min)
If throttled (ThrottlingException): reduce to --max-workers 8
For 200+ nodes: batch by instance group or increase to --max-workers 32 if no throttling
kubectl collection may take 20-30 minutes for 1000+ node clusters

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs a command-line tool, Needs Python, Needs Node.js

Security

What our scanner noticed· Low risk

Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Observability And Instrumentation
Addy Osmani· Community
A systematic approach to instrumenting production code with structured logs, metrics, and traces for effective system observability.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository113.9k
Spark Memory Thermal Ops
Seth Hobson· Community
Manage unified memory and thermal constraints for long-running machine learning jobs on NVIDIA DGX Spark.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 more
MITLow riskHas scriptsGitHub stars of the source repository42.3k
Gke Alert Configuration
GoogleOfficial
Configure GKE alerting policies in Terraform using PromQL and Google Cloud Managed Service for Prometheus.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 more
Apache-2.0Low riskHas scriptsGitHub stars of the source repository23.2k
Application Performance Observability Engineer
Seth Hobson· Community
Expert observability engineer for designing production-grade monitoring, logging, tracing, and reliability systems.
Cloud and DevOpsClaude Code
MITGitHub stars of the source repository42.3k
Distributed Debugging Devops Troubleshooter
Seth Hobson· Community
An expert DevOps troubleshooter for incident response, system reliability, and complex distributed system debugging.
Cloud and DevOpsClaude Code
MITGitHub stars of the source repository42.3k
Distributed Tracing
Seth Hobson· Community
Implement distributed tracing using Jaeger and Tempo to monitor request flows and performance across microservices.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository42.3k

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download hyperpod-issue-report.zip