Gke Node Notready
GoogleOfficialPublished by Google
- Skill
- Review and audit
- Cloud and DevOps
- Apache-2.0
- No issues found
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
This skill provides a structured diagnostic workflow for investigating GKE nodes that report a NotReady or Unknown status. It guides the user through gathering evidence from node conditions, events, kubelet logs, and system metrics to identify root causes such as resource pressure, container runtime failures, or networking issues. The skill operates with a read-only diagnostic boundary, ensuring that it only proposes safe remediations for human review rather than performing automated mutations on the cluster.
What it does
- Extracts cluster and node context from user input or environment settings.
- Analyzes node conditions and events to identify the primary cause of NotReady status.
- Scans kubelet and serial console logs for specific error signatures like PLEG issues or OOM events.
- Maps diagnostic findings to common root causes including resource exhaustion and CNI failures.
- Provides deep links to Cloud Logging and Monitoring for efficient evidence review.
- Proposes actionable remediation steps for human application.
When to use it
- When GKE nodes report a NotReady or Unknown status.
- When the kubelet stops posting node status to the control plane.
- When workloads are evicted or stuck in Pending state due to node health issues.
- When investigating persistent node health faults rather than transient provisioning states.
Advantages
- Enforces a safe, read-only diagnostic boundary.
- Provides a systematic, evidence-based approach to troubleshooting.
- Reduces manual effort by generating direct links to relevant logs and metrics.
- Clearly defines when to escalate issues to support.
Usage examples
- “My GKE node node-1 is showing NotReady status, can you help?”
- The agent will identify the node, inspect its conditions and events, check kubelet logs for error signatures, and provide a summary of the root cause with recommended remediation steps.
- “Why is my GKE node stuck in Unknown status?”
- The agent will analyze the node's recent events and logs to determine if the kubelet is failing to report status, potentially identifying issues like network connectivity or runtime crashes.
Install
Download the skill as a ZIP:
https://autogineer.com/agent-skills/google/gke-node-notready/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip gke-node-notready.zip -d .agents/skills
Documentation by Google
GKE Node NotReady Troubleshooting Skill
Use this skill to systematically diagnose why one or more GKE nodes report a
NotReady (or Ready: Unknown) status and to propose safe remediations. A
NotReady status means the node's kubelet is not reporting to the control plane
correctly, so Kubernetes stops scheduling new Pods on the node, which can reduce
application capacity and cause downtime.
This skill operates non-interactively and enforces a read-only diagnostics
boundary: gather evidence first, then propose a fix (a kubectl/gcloud
command or a GitOps manifest change) for a human to apply. Never mutate the
cluster, drain, delete, or recreate nodes automatically.
[!IMPORTANT] First rule out an expected
NotReady: a node that is newly provisioning, upgrading, being repaired, cordoned, or scaling down will transiently reportNotReady. Only treat it as a fault if it persists beyond the expected window.
Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read Usage Attribution. For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix gcloud with
CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-node-notready)"
and append agent-skills/1.0 (skill:gke-node-notready) to User-Agent headers.
🔍 Diagnostic Workflow
Step 0: Context discovery & time window
Parameter extraction — obtain
project_id,cluster_name,cluster_location, andnode_namenon-interactively from the user prompt, activeSETTINGS.md, or environment defaults (kubectl config current-context,gcloud config get-value project).Credentials & fallback — attempt
gcloud container clusters get-credentials {cluster_name} --location {cluster_location} --project {project_id}. If the cluster is unreachable or commands fail (sandbox/dry-run/offline), present the exact diagnostic commands for a human to run and continue the analysis from the reported symptoms.Time window — determine
{issue_time}(explicit, relative, or now) and center a 1-hour window around it (start = issue_time - 30m,end = issue_time + 30m) for all log/metric queries.
Step 1: Identify NotReady nodes and gather initial status
# List nodes and spot NotReady status, node IPs, and container-runtime version.
kubectl get nodes -o wide
# Inspect the affected node's Conditions and Events (the primary clues).
kubectl describe node "{node_name}"Equivalent via Cloud Logging (preferred when kubectl access is limited or for
historical events). Open it as a Logs Explorer deep link — URL-encode the
query and append the project and Step 0 time window:
https://console.cloud.google.com/logs/query;query={URL_ENCODED_QUERY};timeRange={start}%2F{end}?project={project_id}
(encode / as %2F, or use ;duration=PT1H for a rolling hour):
resource.type="k8s_node"
log_id("events")
resource.labels.node_name="{node_name}"
resource.labels.cluster_name="{cluster_name}"
resource.labels.location="{cluster_location}"Interpret the Conditions table:
Ready: False/Ready: Unknownwith reasonKubeletNotReady/NodeStatusUnknown("Kubelet stopped posting node status") → kubelet or runtime problem; continue to Step 2.MemoryPressure: True,DiskPressure: True,PIDPressure: True→ resource exhaustion; go to Step 4b.NetworkUnavailable: True→ networking/CNI problem; go to Step 4d.
Step 2: Scan kubelet logs for error signatures
Open these kubelet logs as a Logs Explorer deep link using the same
logs/query;query={URL_ENCODED_QUERY};timeRange=...?project=... pattern as Step 1.
resource.type="k8s_node"
resource.labels.node_name="{node_name}"
resource.labels.cluster_name="{cluster_name}"
resource.labels.location="{cluster_location}"
log_id("kubelet")
severity>=WARNINGAlso review the node's serial-console logs (log_id("serialconsole.googleapis.com/serial_port_1_output")
or the resource.type="gce_instance" serial logs) for kernel TaskHung,
OOM-killer, or disk I/O errors that correlate with the kubelet failures.
Step 3: Map the signature to a root cause (decision table)
| Kubelet / event signature | Likely root cause | Go to |
|---|---|---|
runtime is down, Container runtime not ready, errors on /run/containerd/containerd.sock (connection refused / DeadlineExceeded) | Container runtime (containerd) down or unresponsive | Step 4a |
Got sys oom event from cadvisor / kernel OOM-killer in serial logs | System (node-level) OOM killed critical processes | Step 4b |
PLEG is not healthy | PLEG stalled, usually node overload (CPU/disk) | Step 4c |
TaskHung for containerd/kubelet, high disk latency | Disk throttling / I/O starvation | Step 4b |
failed to ensure lease, leases.coordination.k8s.io ... namespace kube-node-lease ... terminating | kube-node-lease termination → NotReady flapping | Step 4f |
| Kubelet cannot reach API server, TLS/dial timeouts | Kubelet ↔ control-plane connectivity | Step 4d |
NetworkPluginNotReady, cni plugin not initialized, NetworkUnavailable | CNI plugin failure | Step 4d |
| Node-critical DaemonSet Pods (CNI, kube-proxy, metadata) blocked from admission | Admission webhook interference | Step 4e |
Only generic NodeNotReady, no other signature | Cause unclear — widen to Step 4d, then escalate | Escalation |
Step 4: Branch investigations
4a. Container runtime (containerd) down
Confirm the kubelet cannot talk to containerd (socket errors above). Check for
containerd restarts/crashes in serial logs. Remediation (propose, don't run):
recreate/repair the node (kubectl drain then let the node pool recreate it, or
gcloud container clusters upgrade/node auto-repair); if it recurs across nodes,
suspect a node image or custom DaemonSet interfering with containerd.
4b. Resource pressure & OOM
# Node allocatable vs. usage.
kubectl describe node "{node_name}" | sed -n '/Allocated resources/,/Events/p'Cloud Monitoring metrics to inspect (read-only): kubernetes.io/node/memory/used_bytes,
kubernetes.io/node/cpu/core_usage_time, kubernetes.io/node/ephemeral_storage/used_bytes.
DiskPressure / disk throttling: full boot disk or slow PD → increase disk size / use a faster PD type; reduce image/log churn.
System OOM: node memory exhausted → set/raise Pod memory
requests/limits, reduce over-commit, or use larger machine types. Distinguish system OOM (node-wide, kills kubelet/runtime) from cgroup OOM (single container).PIDPressure: too many processes → cap Pod PIDs / reduce workload density.
4c. PLEG is not healthy
PLEG is not healthy almost always means the node is overloaded (CPU saturation,
disk latency, or too many Pods/containers per node) so the runtime can't relist
in time. Correlate with 4b metrics. Remediation: reduce node density, add
CPU/disk headroom, or spread workloads.
4d. Networking
# Are node-critical networking Pods healthy on this node?
kubectl get pods -n kube-system -o wide --field-selector spec.nodeName={node_name}Kubelet ↔ control-plane: dial/TLS timeouts to the API server → check firewall rules, Private Google Access, authorized networks, and route/NAT changes.
CNI failure (
NetworkPluginNotReady): the CNI DaemonSet (netd/calico/dataplane) is not running on the node → inspect those Pods' logs/events.
4e. Admission webhook interference
A misconfigured/failing validating or mutating webhook with a broad scope can block node-critical system Pods from being admitted, keeping the node NotReady.
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurationsLook for webhooks that intercept kube-system / node-critical objects with
failurePolicy: Fail. Remediation (propose): scope the webhook out of
kube-system/node-critical namespaces or set an appropriate namespaceSelector.
4f. kube-node-lease termination flapping
If the node flaps NotReady with leases.coordination.k8s.io ... namespace kube-node-lease ... is being terminated, the kube-node-lease namespace was
deleted/terminating. Remediation (propose): do not delete the
kube-node-lease namespace; if terminating, identify the finalizer/actor holding
it and restore the namespace.
Step 5: Remediation boundary & escalation
Present the root cause + evidence (the exact conditions, events, log lines, or metrics observed). Provide Cloud Logging deep links (and Cloud Monitoring links for the Step 4b metrics) to the supporting entries — using the deep-link pattern from Steps 1-2 — so a human can open the evidence directly.
Propose the fix as a command or GitOps manifest change for a human to apply — never apply, drain, or recreate nodes automatically. When to escalate (do this instead of proposing more self-service diagnostics):
Escalate when either:
the relevant logs are unavailable — excluded by a logging filter, or older than the log bucket's retention (the
_Defaultbucket defaults to 30 days, so incidents older than that are permanently deleted); orthe kubelet/event signature is not in the Step 3 table and the root cause remains undetermined after the branch investigations.
In those cases, do all three:
State the limitation plainly (for example, "kubelet logs for that date are past the 30-day
_Defaultretention window and are permanently deleted").Summarize the findings you did gather (node conditions, events, metrics, and any Admin Activity audit logs still in the
_Requiredbucket, default 400-day retention).Route to GKE support / engineering escalation with those findings. Do not keep proposing further self-service investigation, and do not fabricate a diagnosis when the evidence is missing.
References
This skill is derived from public Google Cloud documentation:
Troubleshoot nodes with the NotReady status — node conditions and the kubelet / PLEG / system-OOM / containerd /
kube-node-lease/ CNI / admission-webhook signatures and their remediations.Troubleshoot node registration — Node Registration Checker for nodes that never finish registering.
View GKE logs and Cloud Logging routing overview — log queries (
resource.type="k8s_node",log_id("kubelet")) and log-bucket retention (_Default30 days,_Required400 days).Logs Explorer interface — building and sharing a query by URL (the
logs/query;query=...deep-link format used above).
Files
- ATTRIBUTION.mdAdded by AutoGineer852 B
- LICENSELicence file added by AutoGineer11.1 KiB
- SKILL.md12.0 KiB
- references/attribution.md1.8 KiB
LICENSE
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs a command-line tool, Needs Node.js
Security
No issues found by our static scan· No issues found
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- Publisher
- Google (Official)
- Source
- google/skills · skills/cloud/gke-node-notready · commit 2194d3b
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by Google or Google.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download gke-node-notready.zip