Aks Gpu Inference — Agent Skill by Microsoft

From the author

Diagnose Day-2 AKS GPU and KAITO incidents using profile-aware, read-only evidence. WHEN: 'Insufficient nvidia.com/gpu', GPU pod Pending, model-load OOM, DCGM/VRAM, KAITO Workspace not ready, or GPU autoscaling. DO NOT USE FOR: setup (airunway-aks-setup), non-GPU incidents (aks-troubleshooting), standalone VM quota (azure-quotas), or generic cost (cost-analysis or cost-optimization from the optional azure-cost plugin).

— the description by Microsoft

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured workflow for diagnosing operational issues in Azure Kubernetes Service (AKS) GPU-enabled clusters and KAITO workspaces. It focuses on identifying the root causes of common failures such as pod scheduling delays, VRAM exhaustion, OOM errors, and workspace readiness issues. The skill emphasizes a read-only approach, requiring explicit user authorization before performing any state-changing operations like scaling or node maintenance.

What it does

Analyzes GPU pod scheduling failures and pending states.
Correlates container OOM events with device-level VRAM telemetry.
Validates KAITO workspace readiness and configuration.
Inspects GPU profiles and autoscaling settings.
Provides a framework for gathering evidence from DCGM exporters and Kubernetes status.
Ensures all diagnostic actions comply with host-level security and authorization policies.

When to use it

When encountering 'Insufficient nvidia.com/gpu' errors.
When GPU-enabled pods remain in a Pending state.
When diagnosing model-load OOM or CUDA device-allocation failures.
When investigating KAITO workspace not-ready incidents.
When troubleshooting GPU autoscaling or node-pool capacity issues.

Advantages

Provides a systematic, profile-aware diagnostic path for complex GPU workloads.
Distinguishes between container-level memory issues and GPU-specific device exhaustion.
Maintains strict read-only safety by requiring authorization for any mutation.
Integrates telemetry from DCGM and Kubernetes events for precise incident correlation.

Usage examples

“My GPU pod is stuck in Pending, help me diagnose.”: The agent will bind to the cluster, capture pod events and the current gpuProfile, and investigate scheduling constraints or node capacity issues.
“Why is my model failing with a CUDA OOM error?”: The agent will correlate the model-load timestamp with DCGM metrics like FB_USED and FB_FREE to determine if the failure was caused by VRAM exhaustion.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/microsoft/aks-gpu-inference/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip aks-gpu-inference.zip -d .agents/skills

Documentation by Microsoft

AKS GPU Inference Day-2

Quick Reference

Property	Value
Best for	AKS GPU/KAITO failures
Evidence	Bound target, events/status, `gpuProfile`, DCGM

When to Use This Skill

Use for scheduling, VRAM/OOM, KAITO readiness, and workload scaling. Route exclusions as described above.

MCP Tools

Use a fitting host-advertised Azure read or the references' read-only queries.

Host Capability Gate

Before executing any referenced pipeline, verify that the host authorizes the required shell and kubectl/az commands against the bound target. File-backed collection also requires approved artifact storage and access to any bundled files it uses. A governed Azure CLI tool alone does not establish these capabilities. Equivalent host reads may replace commands only where their advertised schemas provide the required evidence.

If execution is unavailable or prohibited, say so and analyze supplied or redacted events, status, logs, and metrics, or give the operator a scoped collection plan. State which reads did not run and leave conclusions requiring missing evidence unconfirmed. Never route kubectl through Azure MCP or bypass host policy. The authorization requirements below still apply.

Workflow

Bind subscription, cluster, kube context, pool, and affected resource.
Capture exact events/status and observed gpuProfile; state which reads ran.
Route to scheduling, observability, KAITO, or scaling.
Separate container/host OOM from device-allocation failures. For OOMKilled, correlate container memory limits/usage and node conditions. For Managed+Install device-memory evidence, use the exporter on port 19400 and incident-window FB_USED/FB_FREE; sizing tables do not establish a cause.
For KAITO not-ready, warn that Workspace deletion leaves its GPU pools; cleanup is separate and needs explicit authorization.
Report evidence, confidence, missing evidence, and owner handoff.

Require explicit authorization before scaling, cordon/drain, deletion, add-on enablement, or monitoring mutation.

Error Handling

Condition	Response
Missing/contradictory evidence	Mark unconfirmed; request the owner/read
Unknown profile	Preserve evidence; do not prescribe stack repair
Mutation required	Propose separately and wait for authorization

Files

ATTRIBUTION.mdAdded by AutoGineer977 B
LICENSELicence file added by AutoGineer1.1 KiB
SKILL.md3.0 KiB
references/gpu-cost-and-scaling.md1.6 KiB
references/gpu-observability.md2.7 KiB
references/gpu-scheduling.md3.2 KiB
references/kaito-workspaces.md1.8 KiB

references/gpu-observability.md

GPU observability and model-load OOM

Bind telemetry to the affected cluster, namespace, pod, container, GPU, and incident window. Keep host/container-memory OOM and CUDA device-allocator OOM as separate live hypotheses. A Kubernetes OOM symptom alone does not prove GPU memory exhaustion, and a CUDA allocator error does not prove which workload condition exhausted the device. Exclude model-size calculations and GPU SKU-sizing tables, not container or host memory evidence.

Correlate the evidence

kubectl get pod -n <namespace> <pod> -o json
kubectl logs -n <namespace> <pod> -c <container> --previous --timestamps
kubectl get node <gpu-node> -o json
az aks nodepool show --subscription <subscription> -g <resource-group> \
  --cluster-name <cluster> -n <pool> --query gpuProfile -o json

For an observed managed profile (managementMode: Managed, driver: Install), AKS exposes its managed DCGM exporter on port 19400 and labels enabled nodes kubernetes.azure.com/dcgm-exporter=enabled. For an unmanaged or unknown profile, inspect the installed exporter's declared port; do not copy the managed port assumption.

Correlate the model-load timestamp and container termination with:

Signal	Use
`DCGM_FI_DEV_FB_USED`	Frame-buffer memory consumed before failure
`DCGM_FI_DEV_FB_FREE`	Remaining frame-buffer memory before failure
`DCGM_FI_DEV_GPU_UTIL`	Kernel-active time; not an efficiency score
Termination reason `OOMKilled` (often exit 137)	Supports a container-memory path; exit 137 alone is insufficient
`torch.cuda.OutOfMemoryError` or CUDA allocation failure in model logs	Supports a device-allocation path; correlate with per-GPU incident-window telemetry

If FB_USED rises while FB_FREE approaches exhaustion in the same GPU and incident window, the evidence supports VRAM pressure. Healthy aggregate frame-buffer values outside that GPU and incident window do not establish a cause. If historical telemetry is absent, keep device exhaustion unconfirmed. If the container was OOMKilled, inspect its memory limit/usage and node memory conditions rather than discarding the host-memory path.

When the user supplied evidence and prohibited commands, state that no reads ran. Require explicit authorization before monitoring mutation, restart, scaling, cordon/drain, or node-pool changes. Propose a target-bound telemetry collection plan instead.

Sources: managed exporter port, GPU observability, PyTorch CUDA OOM.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Node.js

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Observability And Instrumentation
Addy Osmani· Community
A systematic approach to instrumenting production code with structured logs, metrics, and traces for effective system observability.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository113.9k
Spark Memory Thermal Ops
Seth Hobson· Community
Manage unified memory and thermal constraints for long-running machine learning jobs on NVIDIA DGX Spark.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 more
MITLow riskHas scriptsGitHub stars of the source repository42.3k
Gke Alert Configuration
GoogleOfficial
Configure GKE alerting policies in Terraform using PromQL and Google Cloud Managed Service for Prometheus.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 more
Apache-2.0Low riskHas scriptsGitHub stars of the source repository23.2k
Application Performance Observability Engineer
Seth Hobson· Community
Expert observability engineer for designing production-grade monitoring, logging, tracing, and reliability systems.
Cloud and DevOpsClaude Code
MITGitHub stars of the source repository42.3k
Distributed Debugging Devops Troubleshooter
Seth Hobson· Community
An expert DevOps troubleshooter for incident response, system reliability, and complex distributed system debugging.
Cloud and DevOpsClaude Code
MITGitHub stars of the source repository42.3k
Distributed Tracing
Seth Hobson· Community
Implement distributed tracing using Jaeger and Tempo to monitor request flows and performance across microservices.
Cloud and DevOpsClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository42.3k

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download aks-gpu-inference.zip