Cloud Monitoring Alert Configuration
GoogleOfficialPublished by Google
- Skill
- Workflow
- Cloud and DevOps
- Apache-2.0
- Low risk
- Has scripts
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
This skill provides a structured workflow for defining, reviewing, and managing Cloud Monitoring alerting policies. It leverages Terraform for infrastructure-as-code and PromQL for metric conditions, ensuring consistent observability across Google Cloud resources including Cloud Run, GKE, Compute Engine, and AI Agent platforms. The skill enforces best practices like the Four Golden Signals, dynamic grouping, and proper labeling, while providing clear paths for delegating SLO-specific configurations and notification channel management.
What it does
- Generates parameterized Terraform configurations for Cloud Monitoring alert policies.
- Implements PromQL-based conditions for high-signal monitoring.
- Provides resource-specific guidance for GKE, Cloud Run, Compute Engine, and AI agents.
- Enforces standardized labeling for all generated alerting resources.
- Integrates with specialized scripts to analyze traffic patterns for AI agent telemetry.
- Delegates SLO and notification channel configuration to dedicated skills.
When to use it
- When creating or migrating Terraform-based alerting policies for Google Cloud infrastructure.
- When setting up monitoring for AI agent reliability, quality, and cost telemetry.
- When deduplicating existing alert policies to improve signal-to-noise ratios.
- When configuring alerts for latency, traffic, error rates, or saturation across cloud services.
Advantages
- Ensures consistent observability standards across diverse cloud environments.
- Reduces manual configuration errors through structured Terraform templates.
- Promotes best practices like dynamic grouping and the Four Golden Signals.
- Maintains clear separation of concerns by delegating SLO and notification tasks.
- Provides data-driven traffic analysis for accurate AI agent alert thresholds.
Usage examples
- “Create a Terraform alerting policy for high latency on my GKE service.”
- The agent will guide you through selecting the appropriate GKE PromQL queries, generate the google_monitoring_alert_policy Terraform resource, and prompt you to attach notification channels.
- “Configure a cost alert for my Vertex AI Agent based on token usage.”
- The agent will run the traffic analysis script to determine your workload pattern and generate a PromQL-based alert policy tailored to your agent's historical token consumption.
Install
Download the skill as a ZIP:
https://autogineer.com/agent-skills/google/cloud-monitoring-alert-configuration/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip cloud-monitoring-alert-configuration.zip -d .agents/skills
Documentation by Google
Configure Cloud Monitoring Alerts
Configure high-signal alerting policies in Cloud Monitoring using
Terraform (google_monitoring_alert_policy) and PromQL
(condition_prometheus_query_language).
Usage Attribution
For all gcloud CLI, REST API, and MCP attribution requirements when executing
commands with this skill, follow the instructions in
Usage Attribution.
Enforce Core Alerting Rules
Apply the Four Golden Signals: Cover latency, traffic, errors, and saturation for the target workload.
Use Dynamic Grouping: Aggregate metrics by resource labels and parameterize thresholds with Terraform variables. Never hardcode ephemeral pod or instance identifiers.
Default to PromQL: Use Prometheus Query Language (PromQL) for metric conditions, and use log-based alerts (
condition_matched_logorgoogle_logging_metric) for lifecycle and audit events.Attach Skill Attribution Labels: Always include
user_labelswithcreated-with-google-skill = "cloud-monitoring-alert-configuration"on every generatedgoogle_monitoring_alert_policyresource.Require User Confirmation Before Mutating Resources: Never execute commands that modify infrastructure, enable telemetry, create online monitors, or apply Terraform without waiting for explicit user confirmation.
Prompt for Notification Channels: Alerting policies without notification channels incur charges without notifying anyone during an incident. Whenever notification channels are not specified in the user prompt across any workflow, explicitly inform the user that no notification channels are attached to the alerting policies, advise them to configure and attach notification channels before deploying, and ask at the end of your response if they want to configure notification channels.
Route by Target Resource Type
Identify the target resource type and read only the matching reference file
directly. Never run directory listings or workspace file searches to look for
queries or .tf files unless the prompt
explicitly provides a file path. If a user asks to fix or validate a PromQL
query without providing the query text or a file path, construct a
representative broken-and-corrected query example directly.
Configure Cloud Run Alerts
HTTP Services: cloud_run_services.md
Batch Jobs: cloud_run_jobs.md
Worker Pools: cloud_run_worker_pools.md
Configure Google Kubernetes Engine Alerts
PromQL Queries and Golden Signals: GKE PromQL Queries
Metrics Catalog and Open-Source Alerts: GKE Metrics
Prerequisites and Setup: GKE Setup
Configure AI Agent Alerts
Reliability: Reliability
Quality: Quality
Cost: Cost
Safety: Safety
Security: Security
Telemetry Enablement: Telemetry
Traffic Profiles: Historical and No History
Configure Compute Engine and AI Hypercomputer Alerts
Compute Engine VMs, MIGs, GPU XID Insights, Host or Reservation Maintenance (
type=scheduled,unscheduled, ormultiple), and Machine or Domain Health: compute_engine_alerts.md
Configure Generic Resource Alerts
Generic Workloads, Custom Metrics, MQL Migration, and HCL Deduplication: generic_alert_policies.md
Delegate Service Level Objective Alerts
Delegate to the google-cloud-slo-alert-configuration skill. That skill
owns the SLO setup wizard, the burn-rate PromQL templates
(references/promql_templates.md), the SLI metric mapping
(references/service_metrics.md), and the App Hub scoping labels
(references/app_hub_labels.md). Do NOT duplicate those templates here or
claim that this skill owns its own SLO PromQL template library.
Delegate whenever the user asks for an availability or latency SLO, an error budget or burn-rate alert, or a Service Level Objective on any resource. When SLOs are only one part of a broader request (for example, a full production alerting suite for a Cloud Run service), delegate the SLO policies and continue generating the remaining golden-signal policies from the resource-specific reference in the same response.
Every resource reference carries a Context to hand off list naming the SLI
metric, its good or bad filter, and the grouping labels — under a ## Delegate Service Level Objective Alerts heading for Cloud Run and generic resources, and
inside the SLO section itself for GKE and Agent Platform. Read it and pass that
context on, so the SLO skill does not repeat discovery you have already done.
When delegating, you MUST also:
State the SLO target and window explicitly if the user gave them, or let the SLO skill apply its SRE defaults (99.9% over a rolling 28 days) if they did not.
Require the returned policies to carry
user_labelswithcreated-with-google-skill = "cloud-monitoring-alert-configuration"so attribution stays consistent with the rest of the generated suite. This literal value does NOT change when you delegate: the label records the entry-point skill, not whichever skill supplied the template. Do NOT setcreated-with-google-skill = "google-cloud-slo-alert-configuration"on policies generated through this delegation.Preserve the burn-rate duration contract, which
google-cloud-slo-alert-configurationdoes not specify: Fast Burn (1hand5mwindows, factor14.4) carriesduration = "300s", and Slow Burn ([3d]and[6h]lookback, factor1.0) omits thedurationfield entirely — do not emitduration = "0s"in its place. Cloud Monitoring rejects a non-zero retest buffer when the lookback window exceeds 25 hours.When no notification channels are provided, inform the user that no notification channels are attached to the alerting policies, advise them to configure and attach notification channels, and ask if they want to configure notification channels.
[!IMPORTANT] Always tell the user about the referral, even on the fallback path. Some reference files in this skill include standalone burn-rate templates to use when the
google-cloud-slo-alert-configurationskill is not installed. If you emit policies from those fallback templates, your response MUST still tell the user thatgoogle-cloud-slo-alert-configurationowns SLO configuration, state that the emitted policies use the local fallback templates, and list the handoff context. Never silently present SLO policies as if this skill owned them.
[!NOTE] Exception: Agent Platform (OpenTelemetry-instrumented AI agents on any runtime, including Vertex AI Agent Engine and Cloud Run) is the one resource whose Fast and Slow burn policies live in this skill and remain authoritative. Delegate only for SLO customization — see Reliability.
Delegate Related Observability Tasks
Notification Channels: Delegate to
cloud-monitoring-notification-channels-configurationto configure email, PagerDuty, Pub/Sub, or Slack channels. When an alerting policy has no notification channels attached, inform the user that no notification channels are attached and that they should configure notification channels and attach them to the alerting policy.PromQL Authoring: Delegate to
cloud-monitoring-promql-queryfor standalone PromQL query authoring or metric exploration.
Follow the Alert Configuration Workflow
Discover: Identify target resources, metric schemas, and scoping projects using the lookup commands in the matching reference file (such as discover_metric_schema.py for Compute Engine metrics and other Google Cloud resource metrics, or gather_agent_info.py, check_telemetry.py, analyze_traffic.py, create_online_monitor.py, list_trace_scope_table_names.py, and list_log_scope_table_names.py for AI agents).
Route: Load the reference file for the target resource type or delegate SLOs to
google-cloud-slo-alert-configuration.Generate: Produce parameterized Terraform HCL configurations with required
user_labels.Validate: When the loaded reference file requires a validation script (such as lint_promql.py, validate_config.py, lint_syntax.py, or scan_duplicates.py), run or print that exact script command. Do not list or search the
scripts/directory.Explain and Prompt: Explain each alert policy in plain English. When no notification channels are provided, tell the user that no notification channels are attached to the alerting policy, advise them to configure and attach notification channels, and ask if they want to configure notification channels.
Files
- ATTRIBUTION.mdAdded by AutoGineer890 B
- LICENSELicence file added by AutoGineer11.1 KiB
- SKILL.md11.1 KiB
- references/agent_platform_cost_alert_policies.md9.7 KiB
- references/agent_platform_has_historical_traffic_data.md5.8 KiB
- references/agent_platform_no_historical_traffic_data.md4.1 KiB
- references/agent_platform_quality_alert_policies.md12.8 KiB
- references/agent_platform_reliability_alert_policies.md28.5 KiB
- references/agent_platform_safety_alert_policies.md7.0 KiB
- references/agent_platform_security_alert_policies.md6.8 KiB
- references/agent_platform_telemetry_enablement.md2.6 KiB
- references/attribution.md1.8 KiB
- references/cloud_run_jobs.md5.3 KiB
- references/cloud_run_services.md37.8 KiB
- references/cloud_run_worker_pools.md13.7 KiB
- references/compute_engine_alerts.md40.5 KiB
- references/generic_alert_policies.md18.9 KiB
- references/kubernetes_gke_configuration_prerequisites.md13.3 KiB
- references/kubernetes_metrics_and_alerts_catalog.md22.0 KiB
- references/kubernetes_promql_queries.md18.2 KiB
- scripts/agent_platform/analyze_traffic.pyScript12.8 KiB
- scripts/agent_platform/analyze_traffic_test.pyScript7.8 KiB
- scripts/agent_platform/check_telemetry.pyScript5.8 KiB
- scripts/agent_platform/check_telemetry_test.pyScript5.4 KiB
- scripts/agent_platform/config_utils.pyScript9.4 KiB
- scripts/agent_platform/config_utils_test.pyScript10.2 KiB
- scripts/agent_platform/create_online_monitor.pyScript5.8 KiB
- scripts/agent_platform/create_online_monitor_test.pyScript5.7 KiB
- scripts/agent_platform/gather_agent_info.pyScript14.8 KiB
- scripts/agent_platform/gather_agent_info_test.pyScript11.0 KiB
- scripts/agent_platform/lint_syntax.pyScript1.9 KiB
- scripts/agent_platform/list_log_scope_table_names.pyScript4.7 KiB
- scripts/agent_platform/list_log_scope_table_names_test.pyScript8.6 KiB
- scripts/agent_platform/list_trace_scope_table_names.pyScript4.3 KiB
- scripts/agent_platform/list_trace_scope_table_names_test.pyScript6.4 KiB
- scripts/agent_platform/mock_traffic_data.pyScript100.4 KiB
- scripts/agent_platform/requirements.txt102 B
- scripts/agent_platform/scan_duplicates.pyScript3.7 KiB
- scripts/agent_platform/scan_duplicates_test.pyScript3.9 KiB
- scripts/discover_metric_schema.pyScript13.1 KiB
- scripts/discover_metric_schema_test.pyScript10.4 KiB
- scripts/kubernetes/validate_config.pyScript9.2 KiB
- scripts/lint_promql.pyScript2.5 KiB
references/cloud_run_services.md
Configure Cloud Run Service Alerts
Best-practice observability for HTTP-triggered Cloud Run Services.
Review the Table of Contents
Apply Fast-Path Defaults: Lines 26-52
Generate Core SLO and Instance Saturation Alerts: Lines 33-42
Generate Extended Dashboard and Resource Protection Alerts: Lines 43-52
Apply Default Constants and Override Parameters: Lines 53-76
Delegate SLOs and Apply Fallback Templates: Lines 77-114
Apply PromQL and Terraform Query Templates: Lines 115-888
Monitor Container Instance Saturation: Lines 117-223
Monitor Container Memory and CPU Utilization (OOM Prevention and Throttling): Lines 224-341
Configure Availability SLO Burn Rate Queries: Lines 342-488
Configure Request Latency SLO Queries (P95 and P99): Lines 489-596
Detect Client Error Surges (4xx Response Code Storm): Lines 597-666
Detect Traffic and FinOps Anomalies: Lines 667-876
Diagnose Cold Start Latency: Lines 877-888
Map Telemetry Data Sources: Lines 889-903
Apply Fast-Path Defaults
[!IMPORTANT] When the user prompt explicitly names the target Cloud Run service, job, or worker pool, do not run any shell commands, directory listings, workspace file searches, or
gclouddiscovery commands. Proceed directly to generating the complete Terraform HCL from the templates in this reference. Every generatedgoogle_monitoring_alert_policyresource MUST includeuser_labelswithcreated-with-google-skill = "cloud-monitoring-alert-configuration".
When standard, recommended, or full alerting policies are requested for a Cloud Run HTTP service (or thresholds are unspecified), always generate the complete recommended suite covering both the Core SLO and Instance Saturation Suite (Policies 1–7) and the Extended Dashboard and Resource Protection Suite (Policies 8–12). Delegate the SLO burn-rate policies (Policies 1–4) to the google-cloud-slo-alert-configuration skill when it is installed; generate Policies 1–4 from the local fallback templates in this file only when google-cloud-slo-alert-configuration is unavailable (while still surfacing the referral to google-cloud-slo-alert-configuration as described in Delegate SLOs and Apply Fallback Templates):
Generate Core SLO and Instance Saturation Alerts
Availability SLO (Fast Burn): 99.9% target, 1h & 5m windows, $14.4\times$ burn rate,
duration = "300s"(Critical/Paging).Availability SLO (Slow Burn): 99.9% target, 3d & 6h windows, $1.0\times$ burn rate, omit
duration(Warning/Ticket).Request Latency SLO (P95): Window-based 1h lookback, P95 $\le 500\text{ms}$ in 99% of 5m windows (Warning).
Request Latency SLO (P99): Window-based 1h lookback, P99 $\le 1000\text{ms}$ in 99% of 5m windows (Warning).
Instance Saturation (Warning): >80% of service
max_instancesceiling (Warning).Instance Saturation (Critical): $\ge 95%$ of service
max_instancesceiling (Critical).Traffic Drop Anomaly: 5m throughput drop compared to 1h moving average baseline with min-volume guard (
rate(...[1h]) > 1.0).
Generate Extended Dashboard and Resource Protection Alerts
Container Memory Saturation & OOM Risk: P95 container memory utilization $\ge 90%$ (Critical - Exit Code 137 prevention).
Container CPU Saturation & Throttling: P95 container CPU utilization $> 85%$ (Warning).
Client Error Surge (4xx): 4xx response code ratio $> 10%$ with 5 req/s min-volume guard (Warning).
Billable Instance Time Surge: 5m billable time rate $> 3\times$ 1h moving average baseline (FinOps Warning).
Traffic Surge Anomaly: 5m request rate $> 3\times$ 1h moving average baseline with min-volume guard (Warning).
Apply Default Constants and Override Parameters
All generated Terraform alert policies use explicit default constants. Override any parameter using HCL variables or prompt instructions:
| Constant / Parameter | Explicit Default | HCL Variable Name | Description & Override Option |
|---|---|---|---|
availability_slo_target | 0.999 (99.9%) | var.availability_slo_target | Availability target ratio. Override to 0.99 or 0.9999. |
latency_p95_threshold_ms | 500 (500ms) | var.latency_p95_threshold_ms | P95 latency limit in milliseconds. |
latency_p99_threshold_ms | 1000 (1000ms) | var.latency_p99_threshold_ms | P99 latency limit in milliseconds. |
saturation_warning_ratio | 0.80 (80%) | var.saturation_warning_ratio | Instance saturation warning threshold relative to max_instances. |
saturation_critical_ratio | 0.95 (95%) | var.saturation_critical_ratio | Instance saturation critical threshold relative to max_instances. |
memory_saturation_ratio | 0.90 (90%) | var.memory_saturation_ratio | P95 container memory utilization critical limit (OOM prevention). |
cpu_saturation_ratio | 0.85 (85%) | var.cpu_saturation_ratio | P95 container CPU utilization warning limit. |
client_error_4xx_ratio | 0.10 (10%) | var.client_error_4xx_ratio | Maximum allowable ratio of 4xx client errors. |
traffic_drop_ratio | 0.20 (80% drop) | var.traffic_drop_ratio | Throughput drop threshold compared to 1h moving average. |
traffic_min_hourly_rate | 1.0 (1 req/s) | var.traffic_min_hourly_rate | Min hourly rate guard to prevent scale-to-zero false alarms. |
traffic_surge_multiplier | 3.0 (3x surge) | var.traffic_surge_multiplier | Throughput spike multiplier compared to 1h moving average. |
traffic_surge_min_rate | 15.0 (15 req/s) | var.traffic_surge_min_rate | Minimum 5m request rate guard (req/s) to suppress traffic surge false alarms on low volume. |
billable_time_surge_multiplier | 3.0 (3x surge) | var.billable_time_surge_multiplier | Billable compute time multiplier compared to 1h moving average. |
billable_time_min_rate | 5.0 (5 s/s) | var.billable_time_min_rate | Minimum 5m billable instance-seconds rate guard to prevent FinOps false alarms on low baseline usage. |
retest_duration_buffer | "300s" (5m) | var.retest_duration_buffer | Retest window for lookbacks $\le 25$h. Omitted for lookbacks $>25$h. |
Delegate SLOs and Apply Fallback Templates
Availability and latency SLOs are owned by the
google-cloud-slo-alert-configuration skill. Delegate to it whenever the user
asks for an SLO, an error budget, or burn-rate alerting on a Cloud Run service.
That skill owns the SLO wizard, the burn-rate templates
(references/promql_templates.md), the SLI mapping
(references/service_metrics.md), and the App Hub scoping labels
(references/app_hub_labels.md).
Context to hand off (so the SLO skill does not repeat discovery):
Resource type: Cloud Run service (
cloud_run_revisionmonitored resource).Grouping labels:
service_name,location(addproject_idfor multi-project scopes).Availability SLI:
run_googleapis_com:request_count; bad =response_code_class="5xx".Latency SLI:
run_googleapis_com:request_latencies_bucket; thresholdslatency_p95_threshold_ms/latency_p99_threshold_ms.SLO target:
availability_slo_target(default0.999).
Delegate the SLO policies and continue generating the remaining golden-signal policies (saturation, CPU/memory, 4xx, FinOps, traffic) from this reference in the same response.
Fallback — only if google-cloud-slo-alert-configuration is not installed:
use the standalone burn-rate templates in this reference, which are a complete
solution on their own. Always keep the monitored_resource="cloud_run_revision"
matcher and the min-volume guard shown in
Configure Availability SLO Burn Rate Queries
so idle or scale-to-zero services cannot page. Even then, your response MUST
name google-cloud-slo-alert-configuration as the owner of SLO configuration,
state that what you emitted is a local fallback, and list the handoff context
above. Never silently present SLO policies as if this skill owned them.
Apply PromQL and Terraform Query Templates
Monitor Container Instance Saturation
Monitors when active container instances approach configured scaling ceilings (max_instances).
[!NOTE] PromQL queries for
container_instance_countMUST include the label matchermonitored_resource="cloud_run_revision"to prevent API validation errors resulting from metric ambiguity withcloud_run_worker_pool.
Configure the Warning Query (>80% Capacity)
sum(
max_over_time(run_googleapis_com:container_instance_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]", state="active"}[5m])
) by (service_name, location) / [SERVICE_MAX_INSTANCES] > 0.80Configure the Critical Query (>=95% Capacity)
sum(
max_over_time(run_googleapis_com:container_instance_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]", state="active"}[5m])
) by (service_name, location) / [SERVICE_MAX_INSTANCES] >= 0.95Define the Instance Saturation Terraform Policies
variable "cloud_run_services" {
description = "Map of Cloud Run services to monitor and their max_instances ceilings"
type = map(object({
max_instances = number
}))
default = {
"api-gateway" = { max_instances = 1000 }
}
}
variable "saturation_warning_ratio" {
description = "Ratio threshold for container instance saturation warning"
type = number
default = 0.80
}
variable "saturation_critical_ratio" {
description = "Ratio threshold for container instance saturation critical"
type = number
default = 0.95
}
resource "google_monitoring_alert_policy" "instance_saturation_warning" {
for_each = var.cloud_run_services
project = var.scoping_project_id
display_name = "[Cloud Run] Container Instance Saturation Warning (>80% Limit) - ${each.key}"
combiner = "OR"
conditions {
display_name = "Active Instances Exceed 80% of ${each.key} Ceiling (${each.value.max_instances})"
condition_prometheus_query_language {
query = <<-EOT
sum(
max_over_time(run_googleapis_com:container_instance_count{monitored_resource="cloud_run_revision", service_name="${each.key}", state="active"}[5m])
) by (service_name, location) / ${each.value.max_instances} > ${var.saturation_warning_ratio}
EOT
duration = "300s"
}
}
user_labels = {
severity = "warning"
service = each.key
tier = "saturation"
"created-with-google-skill" = "cloud-monitoring-alert-configuration"
}
alert_strategy { auto_close = "604800s" }
notification_channels = var.notification_channels
}
resource "google_monitoring_alert_policy" "instance_saturation_critical" {
for_each = var.cloud_run_services
project = var.scoping_project_id
display_name = "[Cloud Run] Container Instance Saturation Critical (>=95% Limit) - ${each.key}"
combiner = "OR"
conditions {
display_name = "Active Instances Exceed 95% of ${each.key} Ceiling (${each.value.max_instances})"
condition_prometheus_query_language {
query = <<-EOT
sum(
max_over_time(run_googleapis_com:container_instance_count{monitored_resource="cloud_run_revision", service_name="${each.key}", state="active"}[5m])
) by (service_name, location) / ${each.value.max_instances} >= ${var.saturation_critical_ratio}
EOT
duration = "300s"
}
}
user_labels = {
severity = "critical"
service = each.key
tier = "saturation"
"created-with-google-skill" = "cloud-monitoring-alert-configuration"
}
alert_strategy { auto_close = "604800s" }
notification_channels = var.notification_channels
}Monitor Container Memory and CPU Utilization (OOM Prevention and Throttling)
Monitors actual CPU and Memory resource consumption inside container instances.
Monitor Container Memory Saturation (OOM Risk)
Triggers when the 95th percentile memory utilization reaches critical levels, preventing container abrupt termination from Linux kernel OOMKilled (exit code 137).
histogram_quantile(0.95,
sum(
rate(run_googleapis_com:container_memory_utilizations_bucket{
monitored_resource="cloud_run_revision",
service_name="[SERVICE_NAME]"
}[5m])
) by (le, service_name, location)
) >= 0.90Monitor Container CPU Saturation and Throttling
Triggers when the 95th percentile container CPU utilization exceeds 85%, indicating compute starvation or inadequate vCPU sizing.
histogram_quantile(0.95,
sum(
rate(run_googleapis_com:container_cpu_utilizations_bucket{
monitored_resource="cloud_run_revision",
service_name="[SERVICE_NAME]"
}[5m])
) by (le, service_name, location)
) > 0.85Define the Resource Saturation Terraform Policies
variable "memory_saturation_ratio" {
description = "P95 container memory utilization threshold (OOM danger)"
type = number
default = 0.90
}
variable "cpu_saturation_ratio" {
description = "P95 container CPU utilization threshold (throttling danger)"
type = number
default = 0.85
}
resource "google_monitoring_alert_policy" "container_memory_saturation_critical" {
for_each = var.cloud_run_services
project = var.scoping_project_id
display_name = "[Cloud Run] Container Memory Saturation Critical (P95 >= 90% OOM Risk) - ${each.key}"
combiner = "OR"
conditions {
display_name = "P95 Memory Utilization >= 90% on ${each.key}"
condition_prometheus_query_language {
query = <<-EOT
histogram_quantile(0.95,
sum(
rate(run_googleapis_com:container_memory_utilizations_bucket{
monitored_resource = "cloud_run_revision",
service_name = "${each.key}"
}[5m])
) by (le, service_name, location)
) >= ${var.memory_saturation_ratio}
EOT
duration = "300s"
}
}
user_labels = {
severity = "critical"
service = each.key
tier = "saturation"
"created-with-google-skill" = "cloud-monitoring-alert-configuration"
}
alert_strategy { auto_close = "604800s" }
notification_channels = var.notification_channels
}
resource "google_monitoring_alert_policy" "container_cpu_saturation_warning" {
for_each = var.cloud_run_services
project = var.scoping_project_id
display_name = "[Cloud Run] Container CPU Saturation Warning (P95 > 85%) - ${each.key}"
combiner = "OR"
conditions {
display_name = "P95 CPU Utilization > 85% on ${each.key}"
condition_prometheus_query_language {
query = <<-EOT
histogram_quantile(0.95,
sum(
rate(run_googleapis_com:container_cpu_utilizations_bucket{
monitored_resource = "cloud_run_revision",
service_name = "${each.key}"
}[5m])
) by (le, service_name, location)
) > ${var.cpu_saturation_ratio}
EOT
duration = "300s"
}
}
user_labels = {
severity = "warning"
service = each.key
tier = "saturation"
"created-with-google-skill" = "cloud-monitoring-alert-configuration"
}
alert_strategy { auto_close = "604800s" }
notification_channels = var.notification_channels
}Configure Availability SLO Burn Rate Queries
Configure Availability Fast Burn (1h and 5m Windows)
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]", response_code_class="5xx"}[5m])) by (service_name, location)
/
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[5m])) by (service_name, location)
> (1 - [SLO_TARGET]) * 14.4
)
and
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]", response_code_class="5xx"}[1h])) by (service_name, location)
/
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[1h])) by (service_name, location)
> (1 - [SLO_TARGET]) * 14.4
)
and
(
# Min-volume guard: ensure at least 1 req/s to prevent false alerts on scale-to-zero or low-traffic services
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[1h])) by (service_name, location) > 1.0
)Configure Availability Slow Burn (3d and 6h Windows)
[!WARNING] Lookback windows >25h (like
[3d]) require omittingdurationin HCL.
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]", response_code_class="5xx"}[6h])) by (service_name, location)
/
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[6h])) by (service_name, location)
> (1 - [SLO_TARGET]) * 1.0
)
and
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]", response_code_class="5xx"}[3d])) by (service_name, location)
/
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[3d])) by (service_name, location)
> (1 - [SLO_TARGET]) * 1.0
)
and
(
# Min-volume guard: ensure at least 1 req/s to prevent false alerts on scale-to-zero or low-traffic services
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[6h])) by (service_name, location) > 1.0
)Define the Availability SLO Terraform Policies
variable "availability_slo_target" {
description = "Availability SLO target ratio"
type = number
default = 0.999
}
resource "google_monitoring_alert_policy" "availability_slo_fast_burn" {
for_each = var.cloud_run_services
project = var.scoping_project_id
display_name = "[Cloud Run] Availability SLO Fast Burn (1h/5m 14.4x) - ${each.key}"
combiner = "OR"
conditions {
display_name = "5xx Error Burn Rate Exceeds 14.4x on ${each.key} (1h and 5m windows)"
condition_prometheus_query_language {
query = <<-EOT
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}", response_code_class="5xx"}[5m])) by (service_name, location)
/
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}"}[5m])) by (service_name, location)
> (1 - ${var.availability_slo_target}) * 14.4
)
and
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}", response_code_class="5xx"}[1h])) by (service_name, location)
/
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}"}[1h])) by (service_name, location)
> (1 - ${var.availability_slo_target}) * 14.4
)
and
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}"}[1h])) by (service_name, location) > ${var.traffic_min_hourly_rate}
)
EOT
duration = "300s"
}
}
user_labels = {
severity = "critical"
service = each.key
tier = "slo"
"created-with-google-skill" = "cloud-monitoring-alert-configuration"
}
alert_strategy { auto_close = "604800s" }
notification_channels = var.notification_channels
}
resource "google_monitoring_alert_policy" "availability_slo_slow_burn" {
for_each = var.cloud_run_services
project = var.scoping_project_id
display_name = "[Cloud Run] Availability SLO Slow Burn (3d/6h 1.0x) - ${each.key}"
combiner = "OR"
conditions {
display_name = "5xx Error Burn Rate Exceeds 1.0x on ${each.key} (3d and 6h windows)"
condition_prometheus_query_language {
query = <<-EOT
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}", response_code_class="5xx"}[6h])) by (service_name, location)
/
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}"}[6h])) by (service_name, location)
> (1 - ${var.availability_slo_target}) * 1.0
)
and
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}", response_code_class="5xx"}[3d])) by (service_name, location)
/
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}"}[3d])) by (service_name, location)
> (1 - ${var.availability_slo_target}) * 1.0
)
and
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}"}[6h])) by (service_name, location) > ${var.traffic_min_hourly_rate}
)
EOT
}
}
user_labels = {
severity = "warning"
service = each.key
tier = "slo"
"created-with-google-skill" = "cloud-monitoring-alert-configuration"
}
alert_strategy { auto_close = "604800s" }
notification_channels = var.notification_channels
}Configure Request Latency SLO Queries (P95 and P99)
Configure the P95 Request Latency SLO
avg_over_time(
(
histogram_quantile(0.95, sum(rate(run_googleapis_com:request_latencies_bucket{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[5m])) by (le, service_name, location)) > [LATENCY_THRESHOLD_MS]
)[1h:5m]
) > (1 - [WINDOW_SLO_TARGET])Configure the P99 Request Latency SLO
avg_over_time(
(
histogram_quantile(0.99, sum(rate(run_googleapis_com:request_latencies_bucket{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[5m])) by (le, service_name, location)) > [LATENCY_THRESHOLD_MS]
)[1h:5m]
) > (1 - [WINDOW_SLO_TARGET])Define the Request Latency SLO Terraform Policies
variable "latency_p95_threshold_ms" {
description = "P95 request latency threshold in milliseconds"
type = number
default = 500
}
variable "latency_p99_threshold_ms" {
description = "P99 request latency threshold in milliseconds"
type = number
default = 1000
}
variable "window_slo_target" {
description = "Window-based latency SLO target ratio"
type = number
default = 0.99
}
resource "google_monitoring_alert_policy" "request_latency_slo_p95" {
for_each = var.cloud_run_services
project = var.scoping_project_id
display_name = "[Cloud Run] Request Latency P95 SLO (>500ms) - ${each.key}"
combiner = "OR"
conditions {
display_name = "P95 Latency Exceeds ${var.latency_p95_threshold_ms}ms in >1% of 5m Windows on ${each.key}"
condition_prometheus_query_language {
query = <<-EOT
avg_over_time(
(
histogram_quantile(0.95, sum(rate(run_googleapis_com:request_latencies_bucket{monitored_resource="cloud_run_revision", service_name="${each.key}"}[5m])) by (le, service_name, location)) > ${var.latency_p95_threshold_ms}
)[1h:5m]
) > (1 - ${var.window_slo_target})
EOT
duration = "300s"
}
}
user_labels = {
severity = "warning"
service = each.key
tier = "latency"
"created-with-google-skill" = "cloud-monitoring-alert-configuration"
}
alert_strategy { auto_close = "604800s" }
notification_channels = var.notification_channels
}
resource "google_monitoring_alert_policy" "request_latency_slo_p99" {
for_each = var.cloud_run_services
project = var.scoping_project_id
display_name = "[Cloud Run] Request Latency P99 SLO (>1000ms) - ${each.key}"
combiner = "OR"
conditions {
display_name = "P99 Latency Exceeds ${var.latency_p99_threshold_ms}ms in >1% of 5m Windows on ${each.key}"
condition_prometheus_query_language {
query = <<-EOT
avg_over_time(
(
histogram_quantile(0.99, sum(rate(run_googleapis_com:request_latencies_bucket{monitored_resource="cloud_run_revision", service_name="${each.key}"}[5m])) by (le, service_name, location)) > ${var.latency_p99_threshold_ms}
)[1h:5m]
) > (1 - ${var.window_slo_target})
EOT
duration = "300s"
}
}
user_labels = {
severity = "warning"
service = each.key
tier = "latency"
"created-with-google-skill" = "cloud-monitoring-alert-configuration"
}
alert_strategy { auto_close = "604800s" }
notification_channels = var.notification_channels
}Detect Client Error Surges (4xx Response Code Storm)
Detects client authorization failures (401/403) or concurrency rate-limiting (429) across inbound requests.
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]", response_code_class="4xx"}[5m])) by (service_name, location)
/
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[5m])) by (service_name, location)
> [CLIENT_ERROR_RATIO]
)
and
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[5m])) by (service_name, location) > [MIN_REQ_RATE]
)Define the 4xx Client Error Terraform Policy
variable "client_error_4xx_ratio" {
description = "Ratio threshold for 4xx client errors"
type = number
default = 0.10
}
variable "client_error_min_rate" {
description = "Minimum request rate (req/s) to guard against low-volume noise"
type = number
default = 5.0
}
resource "google_monitoring_alert_policy" "client_error_4xx_surge" {
for_each = var.cloud_run_services
project = var.scoping_project_id
display_name = "[Cloud Run] Client Error Surge (4xx > 10%) - ${each.key}"
combiner = "OR"
conditions {
display_name = "4xx Error Ratio Exceeds 10% on ${each.key}"
condition_prometheus_query_language {
query = <<-EOT
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}", response_code_class="4xx"}[5m])) by (service_name, location)
/
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}"}[5m])) by (service_name, location)
> ${var.client_error_4xx_ratio}
)
and
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}"}[5m])) by (service_name, location) > ${var.client_error_min_rate}
)
EOT
duration = "300s"
}
}
user_labels = {
severity = "warning"
service = each.key
tier = "client_error"
"created-with-google-skill" = "cloud-monitoring-alert-configuration"
}
alert_strategy { auto_close = "604800s" }
notification_channels = var.notification_channels
}Detect Traffic and FinOps Anomalies
Detect Traffic Drop Anomalies
Detects sudden traffic drops compared to 1h moving average, guarded against scale-to-zero.
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[5m])) by (service_name, location)
/
(sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[1h])) by (service_name, location) + 0.001)
) < [DROP_THRESHOLD_RATIO]
and
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[1h])) by (service_name, location) > [MIN_HOURLY_RATE]
)Define the Traffic Drop Terraform Policy
variable "traffic_drop_ratio" {
description = "Ratio threshold compared to 1h average for traffic drop anomaly"
type = number
default = 0.20 # 80% drop
}
variable "traffic_min_hourly_rate" {
description = "Minimum hourly request rate (req/s) guard to prevent scale-to-zero false alarms"
type = number
default = 1.0 # 1 req/s
}
resource "google_monitoring_alert_policy" "traffic_drop_anomaly" {
for_each = var.cloud_run_services
project = var.scoping_project_id
display_name = "[Cloud Run] Traffic Drop Anomaly (>80% Drop) - ${each.key}"
combiner = "OR"
conditions {
display_name = "Traffic Dropped >80% Compared to 1h Average on ${each.key}"
condition_prometheus_query_language {
query = <<-EOT
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}"}[5m])) by (service_name, location)
/
(sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}"}[1h])) by (service_name, location) + 0.001)
) < ${var.traffic_drop_ratio}
and
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}"}[1h])) by (service_name, location) > ${var.traffic_min_hourly_rate}
)
EOT
duration = "300s"
}
}
user_labels = {
severity = "warning"
service = each.key
tier = "traffic"
"created-with-google-skill" = "cloud-monitoring-alert-configuration"
}
alert_strategy { auto_close = "604800s" }
notification_channels = var.notification_channels
}Detect Traffic Surge Anomalies
Detects sudden volumetric traffic spikes (such as DDoS or unexpected viral load) compared to the 1h moving average baseline (> 3x baseline with a minimum rate guard of > 15 req/s).
Write the Traffic Surge PromQL Query
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[5m])) by (service_name, location)
/
(sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[1h])) by (service_name, location) + 0.1)
> 3.0
)
and
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[5m])) by (service_name, location) > 15.0
)Define the Traffic Surge Terraform Policy
variable "traffic_surge_multiplier" {
description = "Throughput spike multiplier compared to 1h moving average baseline"
type = number
default = 3.0 # 3x surge
}
variable "traffic_surge_min_rate" {
description = "Minimum 5m request rate guard (req/s) to suppress false alarms on low volume"
type = number
default = 15.0 # 15 req/s
}
resource "google_monitoring_alert_policy" "traffic_surge_anomaly" {
for_each = var.cloud_run_services
project = var.scoping_project_id
display_name = "[Cloud Run] Traffic Surge Anomaly (>3x Baseline) - ${each.key}"
combiner = "OR"
conditions {
display_name = "Traffic Surged >3x Compared to 1h Average on ${each.key}"
condition_prometheus_query_language {
query = <<-EOT
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}"}[5m])) by (service_name, location)
/
(sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}"}[1h])) by (service_name, location) + 0.1)
> ${var.traffic_surge_multiplier}
)
and
(
sum(rate(run_googleapis_com:request_count{monitored_resource="cloud_run_revision", service_name="${each.key}"}[5m])) by (service_name, location) > ${var.traffic_surge_min_rate}
)
EOT
duration = "300s"
}
}
user_labels = {
severity = "warning"
service = each.key
tier = "traffic"
"created-with-google-skill" = "cloud-monitoring-alert-configuration"
}
alert_strategy { auto_close = "604800s" }
notification_channels = var.notification_channels
}Detect Billable Instance Time Surges (FinOps Cost Control)
Monitors runaway autoscaling or unoptimized compute consumption by tracking surges in billable container instance time (> 3x 1-hour moving average baseline with a minimum billable instance-seconds guard > 5.0).
Write the Billable Instance Time PromQL Query
(
sum(rate(run_googleapis_com:container_billable_instance_time{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[5m])) by (service_name, location)
/
(sum(rate(run_googleapis_com:container_billable_instance_time{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[1h])) by (service_name, location) + 0.1)
> 3.0
)
and
(
sum(rate(run_googleapis_com:container_billable_instance_time{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[5m])) by (service_name, location) > 5.0
)Define the Billable Instance Time Terraform Policy
variable "billable_time_surge_multiplier" {
description = "Billable compute time surge multiplier compared to 1h moving average baseline"
type = number
default = 3.0 # 3x surge
}
variable "billable_time_min_rate" {
description = "Minimum 5m billable instance-seconds rate guard to suppress alerts on low baseline usage"
type = number
default = 5.0 # 5 billable instance-seconds/s
}
resource "google_monitoring_alert_policy" "billable_instance_time_surge" {
for_each = var.cloud_run_services
project = var.scoping_project_id
display_name = "[Cloud Run] FinOps Billable Instance Time Surge (>3x Baseline) - ${each.key}"
combiner = "OR"
conditions {
display_name = "Billable Instance Time >3x Compared to 1h Average on ${each.key}"
condition_prometheus_query_language {
query = <<-EOT
(
sum(rate(run_googleapis_com:container_billable_instance_time{monitored_resource="cloud_run_revision", service_name="${each.key}"}[5m])) by (service_name, location)
/
(sum(rate(run_googleapis_com:container_billable_instance_time{monitored_resource="cloud_run_revision", service_name="${each.key}"}[1h])) by (service_name, location) + 0.1)
> ${var.billable_time_surge_multiplier}
)
and
(
sum(rate(run_googleapis_com:container_billable_instance_time{monitored_resource="cloud_run_revision", service_name="${each.key}"}[5m])) by (service_name, location) > ${var.billable_time_min_rate}
)
EOT
duration = "300s"
}
}
user_labels = {
severity = "warning"
service = each.key
tier = "finops"
"created-with-google-skill" = "cloud-monitoring-alert-configuration"
}
alert_strategy { auto_close = "604800s" }
notification_channels = var.notification_channels
}Diagnose Cold Start Latency
histogram_quantile(0.95,
sum(
rate(run_googleapis_com:container_startup_latencies_bucket{monitored_resource="cloud_run_revision", service_name="[SERVICE_NAME]"}[30m])
) by (le, service_name, location)
) > [STARTUP_LATENCY_SECONDS]Map Telemetry Data Sources
| Signal Indicator | Native Metric | SRE Observability Strategy |
|---|---|---|
| Availability (5xx) | run.googleapis.com/request_count | SLO Fast Burn (1h/5m) & Slow Burn (3d/6h) (response_code_class="5xx") |
| Request Latency (P50/P95/P99) | run.googleapis.com/request_latencies | SLO P95 & P99 Window Evaluation (request_latencies_bucket) |
| Performance / Client Errors (4xx) | run.googleapis.com/request_count | Client Error Spike Alert: 4xx ratio $> 10%$ with 5 req/s guard |
| Container Instance Saturation | run.googleapis.com/container/instance_count | Scaling Ceiling Warning (0.80) & Critical (0.95) relative to max_instances |
| Container Memory Utilization | run.googleapis.com/container/memory/utilizations | OOM Prevention Alert: P95 memory utilization $\ge 90%$ (Exit 137 guard) |
| Container CPU Utilization | run.googleapis.com/container/cpu/utilizations | Compute Throttling Alert: P95 CPU utilization $> 85%$ |
| Billable Instance Time | run.googleapis.com/container/billable_instance_time | FinOps Cost Spike Alert: 5m rate $> 3\times$ 1h moving average baseline |
| Container CPU Allocation | run.googleapis.com/container/cpu/allocation_time | Regional vCPU Quota Headroom Monitoring |
| Container Memory Allocation | run.googleapis.com/container/memory/allocation_time | Regional RAM Quota Headroom Monitoring |
| Traffic Drop Anomaly | run.googleapis.com/request_count | Guarded Anomaly Alert: Drop compared to 1h baseline (rate(...[1h]) > 1.0) |
| Cold Start Latency | run.googleapis.com/container/startup_latencies | Conditional Diagnostic: P95 startup latency over 30m window |
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs Python
Security
What our scanner noticed· Low risk
- Scripts that your agent may run
This skill includes scripts that your agent may run. Read them before you use it.
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- Publisher
- Google (Official)
- Source
- google/skills · skills/cloud/cloud-monitoring-alert-configuration · commit 2194d3b
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by Google or Google.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download cloud-monitoring-alert-configuration.zip