Gke AI Troubleshooting Tpu Dynamic Slices Monitoring

GKE TPU Dynamic Slices Monitoring & Management

Monitors the status of TPU Slice custom resources, troubleshoots provisioning failures, validates workload manifests on dynamic slices, and performs cleanups.

Attribution

MUST READ: Before executing any gcloud, REST API, or MCP command, you MUST read Usage Attribution. For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix gcloud with CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-dynamic-slices-monitoring)" and append agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-dynamic-slices-monitoring) to User-Agent headers.

Prerequisites

• Cloud Logging enabled for the project.

• kubectl and gcloud CLIs configured to access the GKE cluster.

Diagnostic Workflow

Step 0: Context Acquisition & Time Window Definition

Gather project, cluster, and slice context using cluster tools or the following parameters:

• Project ID: {project_id} (e.g., my-gcp-project)

• Cluster Name: {cluster_name} (e.g., tpu-cluster)

• Region/Zone: {location} (e.g., us-central1-a)

• Slice Name: {slice_name} (e.g., test-slice)

• Issue Time: {timestamp} (Optional; default to the last 30 minutes window [T - 30m] to [T + 30m])

Step 1: Describe the Slice Custom Resource [Low Risk]

When asked to inspect, troubleshoot, or check a slice status, immediately execute kubectl describe slice {slice_name} using available cluster tools to perform the inspection. Parse the resulting Status.Conditions output against the condition table below to diagnose the exact state and provide concrete recommendations.

• Command:

  Copy

  kubectl describe slice {slice_name}

State & Reason Analysis

Analyze the Status.Conditions (especially Type: Ready and its Reason and Status):

Provisioning Failure Troubleshooting Checklist

When investigating slice creation or provisioning failures (SliceCreationFailed or FAILED), perform the following verification steps:

1. Node Existence & Allocation Check: Verify that the selected TPU nodes exist in the cluster and are not already allocated to another slice (kubectl get nodes -l cloud.google.com/gke-tpu-slice, kubectl get slice -A).

2. Topology Alignment: Confirm that the partition count matches the requested topology dimensions (e.g. topology 2x2 requires 4 nodes).

3. Reservation Block Alignment Check: Confirm that all selected TPU nodes belong to the same reservation and reservation block.

Step 2: Verify Workload Specification [Low Risk]

Ensure workload manifests are configured correctly to target the dynamic slice.

1. Single-Slice Workload Requirements

Check that the Pod template contains the following annotations and selectors:

• Annotations:

  • cloud.google.com/gke-tpu-slice-topology: "{topology}" (e.g., "4x4x4")

• NodeSelector:

  • cloud.google.com/gke-tpu-topology: "{topology}" (e.g., "4x4x4")

  • cloud.google.com/gke-tpu-accelerator: "{accelerator_type}" (e.g., "tpu7x")

  • cloud.google.com/gke-tpu-slice: "{slice_name}" (e.g., "test-slice")

2. Multi-Slice (JobSet) Workload Requirements

If deploying a multi-slice JobSet, verify:

• JobSet Annotation:

  • alpha.jobset.sigs.k8s.io/exclusive-topology: cloud.google.com/gke-tpu-slice

• Pod Template Annotations:

  • cloud.google.com/gke-tpu-slice-topology: "{topology}"

• Pod Template NodeSelector:

  • cloud.google.com/gke-tpu-topology: "{topology}"

  • cloud.google.com/gke-tpu-accelerator: "{accelerator_type}"

  • Note: Do NOT manually specify cloud.google.com/gke-tpu-slice in the nodeSelector; JobSet handles slice assignment automatically.

Resolution & Management Workflow

Resolution 1: Force Delete a Stuck Slice [High Risk]

If a slice is stuck in DEACTIVATING or deletion hangs indefinitely due to stuck finalizers:

1. Identify Cause: Explain that finalizers on the slice resource (metadata.finalizers) are preventing Kubernetes from completing resource deletion.

2. Propose Resolution: Propose removing finalizers from the metadata path (/metadata/finalizers) using a JSON patch operation:

   Copy

   kubectl patch slice {slice_name} --type json -p='[{"op": "remove", "path": "/metadata/finalizers"}]'

3. Provide Warning: Explicitly warn the user that removing finalizers bypasses standard controller dismantling and may leave underlying VM, network, or accelerator resources uncleaned or orphaned.

4. CRITICAL SAFETY MANDATE: The response MUST explicitly ask the user for confirmation (e.g. "Removing finalizers on /metadata/finalizers via JSON patch is a high-risk operation that may leave orphaned resources. Do you confirm you want to apply this patch to slice {slice_name}?") and pause for user confirmation before applying or executing the patch.

Resolution 2: Disable and Clean Up Slice Controller [High Risk]

If dynamic slicing needs to be disabled:

1. Check for existing Slices:

   Copy

   kubectl get slice -A

   Ensure all slices are deleted before disabling the controller.

2. Disable Slice Controller via gcloud:

   Copy

   gcloud container clusters update {cluster_name} \
       --location={location} \
       --no-enable-slice-controller

3. Delete the Slice CRD:

   Copy

   kubectl delete crd slices.accelerator.gke.io

4. Clean up Node Labels: Remove GKE TPU Slice labels from all nodes in the cluster:

   Copy

   kubectl label nodes --all cloud.google.com/gke-tpu-slice- cloud.google.com/gke-tpu-slice-topology-

• Safety Rule: Propose the exact commands and confirm before executing disabling or destructive cleanup steps.