Cloud Monitoring Alert Configuration

GoogleOfficialPublished by Google

Skill
Workflow
Cloud and DevOps
Apache-2.0
Low risk
Has scripts

From the author

Configures Cloud Monitoring alerting policies using Terraform and PromQL across Cloud Run, Google Kubernetes Engine (GKE), Compute Engine, AI Hypercomputer reservations, Vertex AI Agent Engine, and generic Google Cloud resources. Use when creating, reviewing, migrating, or deduplicating Terraform alerting policies for latency, traffic, error rates, saturation, host or reservation maintenance, GPU XID insights, or AI agent telemetry. Don't use for standalone PromQL query authoring without alerting, log querying, or billing cost audits.

— the description by Google

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured workflow for defining, reviewing, and managing Cloud Monitoring alerting policies. It leverages Terraform for infrastructure-as-code and PromQL for metric conditions, ensuring consistent observability across Google Cloud resources including Cloud Run, GKE, Compute Engine, and AI Agent platforms. The skill enforces best practices like the Four Golden Signals, dynamic grouping, and proper labeling, while providing clear paths for delegating SLO-specific configurations and notification channel management.

What it does

Generates parameterized Terraform configurations for Cloud Monitoring alert policies.
Implements PromQL-based conditions for high-signal monitoring.
Provides resource-specific guidance for GKE, Cloud Run, Compute Engine, and AI agents.
Enforces standardized labeling for all generated alerting resources.
Integrates with specialized scripts to analyze traffic patterns for AI agent telemetry.
Delegates SLO and notification channel configuration to dedicated skills.

When to use it

When creating or migrating Terraform-based alerting policies for Google Cloud infrastructure.
When setting up monitoring for AI agent reliability, quality, and cost telemetry.
When deduplicating existing alert policies to improve signal-to-noise ratios.
When configuring alerts for latency, traffic, error rates, or saturation across cloud services.

Advantages

Ensures consistent observability standards across diverse cloud environments.
Reduces manual configuration errors through structured Terraform templates.
Promotes best practices like dynamic grouping and the Four Golden Signals.
Maintains clear separation of concerns by delegating SLO and notification tasks.
Provides data-driven traffic analysis for accurate AI agent alert thresholds.

Usage examples

“Create a Terraform alerting policy for high latency on my GKE service.”: The agent will guide you through selecting the appropriate GKE PromQL queries, generate the google_monitoring_alert_policy Terraform resource, and prompt you to attach notification channels.
“Configure a cost alert for my Vertex AI Agent based on token usage.”: The agent will run the traffic analysis script to determine your workload pattern and generate a PromQL-based alert policy tailored to your agent's historical token consumption.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/google/cloud-monitoring-alert-configuration/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip cloud-monitoring-alert-configuration.zip -d .agents/skills

Documentation by Google

Configure Cloud Monitoring Alerts

Configure high-signal alerting policies in Cloud Monitoring using Terraform (google_monitoring_alert_policy) and PromQL (condition_prometheus_query_language).

Usage Attribution

For all gcloud CLI, REST API, and MCP attribution requirements when executing commands with this skill, follow the instructions in Usage Attribution.

Enforce Core Alerting Rules

Apply the Four Golden Signals: Cover latency, traffic, errors, and saturation for the target workload.
Use Dynamic Grouping: Aggregate metrics by resource labels and parameterize thresholds with Terraform variables. Never hardcode ephemeral pod or instance identifiers.
Default to PromQL: Use Prometheus Query Language (PromQL) for metric conditions, and use log-based alerts (condition_matched_log or google_logging_metric) for lifecycle and audit events.
Attach Skill Attribution Labels: Always include user_labels with created-with-google-skill = "cloud-monitoring-alert-configuration" on every generated google_monitoring_alert_policy resource.
Require User Confirmation Before Mutating Resources: Never execute commands that modify infrastructure, enable telemetry, create online monitors, or apply Terraform without waiting for explicit user confirmation.
Prompt for Notification Channels: Alerting policies without notification channels incur charges without notifying anyone during an incident. Whenever notification channels are not specified in the user prompt across any workflow, explicitly inform the user that no notification channels are attached to the alerting policies, advise them to configure and attach notification channels before deploying, and ask at the end of your response if they want to configure notification channels.

Route by Target Resource Type

Identify the target resource type and read only the matching reference file directly. Never run directory listings or workspace file searches to look for queries or .tf files unless the prompt explicitly provides a file path. If a user asks to fix or validate a PromQL query without providing the query text or a file path, construct a representative broken-and-corrected query example directly.

Configure Cloud Run Alerts

HTTP Services: cloud_run_services.md
Batch Jobs: cloud_run_jobs.md
Worker Pools: cloud_run_worker_pools.md

Configure Google Kubernetes Engine Alerts

PromQL Queries and Golden Signals: GKE PromQL Queries
Metrics Catalog and Open-Source Alerts: GKE Metrics
Prerequisites and Setup: GKE Setup

Configure AI Agent Alerts

Reliability: Reliability
Quality: Quality
Cost: Cost
Safety: Safety
Security: Security
Telemetry Enablement: Telemetry
Traffic Profiles: Historical and No History

Configure Compute Engine and AI Hypercomputer Alerts

Compute Engine VMs, MIGs, GPU XID Insights, Host or Reservation Maintenance (type=scheduled, unscheduled, or multiple), and Machine or Domain Health: compute_engine_alerts.md

Configure Generic Resource Alerts

Generic Workloads, Custom Metrics, MQL Migration, and HCL Deduplication: generic_alert_policies.md

Delegate Service Level Objective Alerts

Delegate to the google-cloud-slo-alert-configuration skill. That skill owns the SLO setup wizard, the burn-rate PromQL templates (references/promql_templates.md), the SLI metric mapping (references/service_metrics.md), and the App Hub scoping labels (references/app_hub_labels.md). Do NOT duplicate those templates here or claim that this skill owns its own SLO PromQL template library.

Delegate whenever the user asks for an availability or latency SLO, an error budget or burn-rate alert, or a Service Level Objective on any resource. When SLOs are only one part of a broader request (for example, a full production alerting suite for a Cloud Run service), delegate the SLO policies and continue generating the remaining golden-signal policies from the resource-specific reference in the same response.

Every resource reference carries a Context to hand off list naming the SLI metric, its good or bad filter, and the grouping labels — under a ## Delegate Service Level Objective Alerts heading for Cloud Run and generic resources, and inside the SLO section itself for GKE and Agent Platform. Read it and pass that context on, so the SLO skill does not repeat discovery you have already done. When delegating, you MUST also:

State the SLO target and window explicitly if the user gave them, or let the SLO skill apply its SRE defaults (99.9% over a rolling 28 days) if they did not.
Require the returned policies to carry user_labels with created-with-google-skill = "cloud-monitoring-alert-configuration" so attribution stays consistent with the rest of the generated suite. This literal value does NOT change when you delegate: the label records the entry-point skill, not whichever skill supplied the template. Do NOT set created-with-google-skill = "google-cloud-slo-alert-configuration" on policies generated through this delegation.
Preserve the burn-rate duration contract, which google-cloud-slo-alert-configuration does not specify: Fast Burn (1h and 5m windows, factor 14.4) carries duration = "300s", and Slow Burn ([3d] and [6h] lookback, factor 1.0) omits the duration field entirely — do not emit duration = "0s" in its place. Cloud Monitoring rejects a non-zero retest buffer when the lookback window exceeds 25 hours.
When no notification channels are provided, inform the user that no notification channels are attached to the alerting policies, advise them to configure and attach notification channels, and ask if they want to configure notification channels.

[!IMPORTANT] Always tell the user about the referral, even on the fallback path. Some reference files in this skill include standalone burn-rate templates to use when the google-cloud-slo-alert-configuration skill is not installed. If you emit policies from those fallback templates, your response MUST still tell the user that google-cloud-slo-alert-configuration owns SLO configuration, state that the emitted policies use the local fallback templates, and list the handoff context. Never silently present SLO policies as if this skill owned them.

[!NOTE] Exception: Agent Platform (OpenTelemetry-instrumented AI agents on any runtime, including Vertex AI Agent Engine and Cloud Run) is the one resource whose Fast and Slow burn policies live in this skill and remain authoritative. Delegate only for SLO customization — see Reliability.

Notification Channels: Delegate to cloud-monitoring-notification-channels-configuration to configure email, PagerDuty, Pub/Sub, or Slack channels. When an alerting policy has no notification channels attached, inform the user that no notification channels are attached and that they should configure notification channels and attach them to the alerting policy.
PromQL Authoring: Delegate to cloud-monitoring-promql-query for standalone PromQL query authoring or metric exploration.

Follow the Alert Configuration Workflow

Discover: Identify target resources, metric schemas, and scoping projects using the lookup commands in the matching reference file (such as discover_metric_schema.py for Compute Engine metrics and other Google Cloud resource metrics, or gather_agent_info.py, check_telemetry.py, analyze_traffic.py, create_online_monitor.py, list_trace_scope_table_names.py, and list_log_scope_table_names.py for AI agents).
Route: Load the reference file for the target resource type or delegate SLOs to google-cloud-slo-alert-configuration.
Generate: Produce parameterized Terraform HCL configurations with required user_labels.
Validate: When the loaded reference file requires a validation script (such as lint_promql.py, validate_config.py, lint_syntax.py, or scan_duplicates.py), run or print that exact script command. Do not list or search the scripts/ directory.
Explain and Prompt: Explain each alert policy in plain English. When no notification channels are provided, tell the user that no notification channels are attached to the alerting policy, advise them to configure and attach notification channels, and ask if they want to configure notification channels.

Files

ATTRIBUTION.mdAdded by AutoGineer890 B
LICENSELicence file added by AutoGineer11.1 KiB
SKILL.md11.1 KiB
references/agent_platform_cost_alert_policies.md9.7 KiB
references/agent_platform_has_historical_traffic_data.md5.8 KiB
references/agent_platform_no_historical_traffic_data.md4.1 KiB
references/agent_platform_quality_alert_policies.md12.8 KiB
references/agent_platform_reliability_alert_policies.md28.5 KiB
references/agent_platform_safety_alert_policies.md7.0 KiB
references/agent_platform_security_alert_policies.md6.8 KiB
references/agent_platform_telemetry_enablement.md2.6 KiB
references/attribution.md1.8 KiB
references/cloud_run_jobs.md5.3 KiB
references/cloud_run_services.md37.8 KiB
references/cloud_run_worker_pools.md13.7 KiB
references/compute_engine_alerts.md40.5 KiB
references/generic_alert_policies.md18.9 KiB
references/kubernetes_gke_configuration_prerequisites.md13.3 KiB
references/kubernetes_metrics_and_alerts_catalog.md22.0 KiB
references/kubernetes_promql_queries.md18.2 KiB
scripts/agent_platform/analyze_traffic.pyScript12.8 KiB
scripts/agent_platform/analyze_traffic_test.pyScript7.8 KiB
scripts/agent_platform/check_telemetry.pyScript5.8 KiB
scripts/agent_platform/check_telemetry_test.pyScript5.4 KiB
scripts/agent_platform/config_utils.pyScript9.4 KiB
scripts/agent_platform/config_utils_test.pyScript10.2 KiB
scripts/agent_platform/create_online_monitor.pyScript5.8 KiB
scripts/agent_platform/create_online_monitor_test.pyScript5.7 KiB
scripts/agent_platform/gather_agent_info.pyScript14.8 KiB
scripts/agent_platform/gather_agent_info_test.pyScript11.0 KiB
scripts/agent_platform/lint_syntax.pyScript1.9 KiB
scripts/agent_platform/list_log_scope_table_names.pyScript4.7 KiB
scripts/agent_platform/list_log_scope_table_names_test.pyScript8.6 KiB
scripts/agent_platform/list_trace_scope_table_names.pyScript4.3 KiB
scripts/agent_platform/list_trace_scope_table_names_test.pyScript6.4 KiB
scripts/agent_platform/mock_traffic_data.pyScript100.4 KiB
scripts/agent_platform/requirements.txt102 B
scripts/agent_platform/scan_duplicates.pyScript3.7 KiB
scripts/agent_platform/scan_duplicates_test.pyScript3.9 KiB
scripts/discover_metric_schema.pyScript13.1 KiB
scripts/discover_metric_schema_test.pyScript10.4 KiB
scripts/kubernetes/validate_config.pyScript9.2 KiB
scripts/lint_promql.pyScript2.5 KiB

references/agent_platform_quality_alert_policies.md

Configure Quality Alert Policies

[!IMPORTANT] Quality alerting policies are specific to Vertex AI Agent Engine deployments and rely on metrics exported by Vertex AI Online Monitors.

These alert policies help users diagnose drops in response quality (intent drift) or hallucination rates from their agents. They also help identify when an agent gets stuck in an infinite reasoning or tool call loop.

Verify Prerequisites: Lines 25-117
- Warn About Billing Costs and Provision the Online Monitor: Lines 27-117
Follow Critical Rules: Lines 118-161
- Review Telemetry Metrics: Lines 134-146
- Construct the Metric Filter: Lines 147-161
Configure Policy Specifications: Lines 162-263
- Configure Final Response Quality: Lines 164-196
- Configure Tool Use Quality: Lines 197-229
- Configure Hallucination Alerts: Lines 230-263
Run Tooling Scripts: Lines 264-281
Validate and Finalize: Lines 282-292

Verify Prerequisites

Warn About Billing Costs and Provision the Online Monitor

Quality alerting policies rely on metrics exported by Online Monitors. If telemetry is disabled on the reasoning engine, no traces are sent, and the quality metrics remain empty. Ensure the Online Monitor is provisioned for the agent and telemetry is enabled:

[ ] Ask for Approval with Billing Warning: Both Online Monitors and Telemetry incur separate billing charges. If not pre-approved in the prompt:
- Your response MUST warn the user about the potential billing costs of Online Monitor and explicitly ask the user for approval before configuring or provisioning it.
- Your response MUST warn the user about the potential billing costs of Telemetry (Cloud Trace/Logging export) and explicitly ask the user for approval before configuring or enabling it.
- Important: Even when printing draft Terraform HCL or CLI commands without executing them, do not merely list billing warnings as passive bullet points — you MUST include a direct question asking the user to confirm and approve both Online Monitor and Telemetry billing costs before configuring them (for example, "Do you approve enabling Telemetry (Cloud Trace/Logging export) and provisioning the Vertex AI Online Monitor, given the additional billing costs they incur?").
[ ] Verify Telemetry First: Before generating any alerting policy plan or provisioning Online Monitors, you MUST always verify the telemetry status of the Reasoning Engine first using the check_telemetry.py script as detailed in Verify Telemetry Status below.

Verify Telemetry Status

Before generating any alerting policies, proposing a plan, or provisioning Online Monitors, you MUST always verify whether the agent is ready to export traces by running the telemetry verification script:

Mandatory Command: python3 scripts/agent_platform/check_telemetry.py --project-id "{project_id}" --agent-resource-name "{agent_resource_name}"
- Note on Parameters: The {agent_resource_name} parameter MUST be the full resource path format projects/<project_id>/locations/<location>/reasoningEngines/<agent_id> (for example, projects/gcp-prod/locations/us-central1/reasoningEngines/556677) and not just the agent ID itself.
- Dependency Failures: If package imports or dependency installation fails, resolve the issues (for example, by verifying package installation) and run the script again with the specific project-id and agent-resource-name parameters. If you cannot run the script successfully due to missing dependencies, you MUST still attempt to run it first and then include the complete, pre-populated execution command in your final response as a verification plan (this applies even if the user has pre-approved provisioning).
Detailed Enablement Instructions: For details on the required environment variables, Terraform setups, and project dependencies, you MUST read and follow: agent_platform_telemetry_enablement.md.

Provision the Online Monitor

Because Online Monitors cannot be configured using Terraform, follow the instructions in Run Tooling Scripts to run the create_online_monitor provisioning script and generate the required metrics. Proceed to the Validate and Finalize section below.

Format the Execution Plan

If execution fails (for example, due to sandbox restrictions or permissions), you MUST provide a clear, actionable plan for the user to follow as a next step. Your response MUST explicitly include a section containing the exact Python execution command with all parameter values (such as project ID, region, and agent resource name) fully populated. Do not merely state that the user must run it.

Format the plan as follows:

Provide the Online Monitor Provisioning Plan

Online Monitor Provisioning Command:

python3 scripts/agent_platform/create_online_monitor.py \
  --project-id "{project_id}" \
  --agent-resource-name "projects/{project_id}/locations/{location}/reasoningEngines/{agent_id}" \
  --sampling-percentage {percentage}

Verify Telemetry Command (Optional fallback):

python3 scripts/agent_platform/check_telemetry.py \
  --project-id "{project_id}" \
  --agent-resource-name "projects/{project_id}/locations/{location}/reasoningEngines/{agent_id}"

Follow Critical Rules

Alert policy configuration: You MUST configure the following Quality alerting policies (single-metric degradation policies, also referred to as Quality - Metric Degradation (Single Metric)). See Review Telemetry Metrics to get the required metrics and read the descriptions for each signal:
1. Final Response Quality (5-Minute window)
2. Tool Use Quality (5-Minute window)
3. Hallucination (5-Minute window)
Standard Threshold Filters for Agent Quality: For the 3 agent quality metrics, you MUST use standard condition_threshold filters matching the monitored resource type aiplatform.googleapis.com/OnlineEvaluator and metric type aiplatform.googleapis.com/online_evaluator/scores. Do NOT use PromQL or condition_sql.

Review Telemetry Metrics

Because the scores metric is of value type DISTRIBUTION, standard mean-based PromQL or arithmetic ALIGN_MEAN aligners are unsupported. You MUST use a percentile aligner (typically ALIGN_PERCENTILE_50 to evaluate the median score) within the aggregations block of your condition_threshold.

Signal	Metric Name (`evaluation_metric_name`)	Target Threshold	Recommended Aligner
Final Response Quality	`final_response_quality_v1`	`< 0.8` (or custom)	`ALIGN_PERCENTILE_50`
Tool Use Quality	`tool_use_quality_v1`	`< 0.8` (or custom)	`ALIGN_PERCENTILE_50`
Hallucination (Groundedness)	`hallucination_v1`	`< 0.9` (or custom)	`ALIGN_PERCENTILE_50`

Construct the Metric Filter

All agent quality evaluation metrics are exported by Online Monitors to the monitored resource type aiplatform.googleapis.com/OnlineEvaluator under the metric type aiplatform.googleapis.com/online_evaluator/scores.

When configuring a quality alert policy in Terraform, use the following filter expression structure:

resource.type="aiplatform.googleapis.com/OnlineEvaluator"
AND metric.type="aiplatform.googleapis.com/online_evaluator/scores"
AND metric.labels.evaluation_metric_name="{metric_name}"

Configure Policy Specifications

Configure Final Response Quality

Define Terraform HCL for Final Response Quality

resource "google_monitoring_alert_policy" "final_response_quality" {
  project      = var.project_id
  display_name = "Agent Quality - Final Response Quality Low"
  combiner     = "OR"
  conditions {
    display_name = "Final Response Quality Score < 0.8"
    condition_threshold {
      filter                  = "resource.type=\"aiplatform.googleapis.com/OnlineEvaluator\" AND metric.type=\"aiplatform.googleapis.com/online_evaluator/scores\" AND metric.labels.evaluation_metric_name=\"final_response_quality_v1\""
      comparison              = "COMPARISON_LT"
      threshold_value         = 0.8
      duration                = "300s" # 5m buffer
      evaluation_missing_data = "EVALUATION_MISSING_DATA_INACTIVE"
      trigger {
        count = 1
      }
      aggregations {
        alignment_period   = "300s" # Aligned with duration
        per_series_aligner = "ALIGN_PERCENTILE_50"
      }
    }
  }

  user_labels = {
    created-with-google-skill = "cloud-monitoring-alert-configuration"
  }
}

Configure Tool Use Quality

Define Terraform HCL for Tool Use Quality

resource "google_monitoring_alert_policy" "tool_use_quality" {
  project      = var.project_id
  display_name = "Agent Quality - Tool Use Quality Low"
  combiner     = "OR"
  conditions {
    display_name = "Tool Use Quality Score < 0.8"
    condition_threshold {
      filter                  = "resource.type=\"aiplatform.googleapis.com/OnlineEvaluator\" AND metric.type=\"aiplatform.googleapis.com/online_evaluator/scores\" AND metric.labels.evaluation_metric_name=\"tool_use_quality_v1\""
      comparison              = "COMPARISON_LT"
      threshold_value         = 0.8
      duration                = "300s"
      evaluation_missing_data = "EVALUATION_MISSING_DATA_INACTIVE"
      trigger {
        count = 1
      }
      aggregations {
        alignment_period   = "300s"
        per_series_aligner = "ALIGN_PERCENTILE_50"
      }
    }
  }

  user_labels = {
    created-with-google-skill = "cloud-monitoring-alert-configuration"
  }
}

Configure Hallucination Alerts

Define Terraform HCL for Hallucination Alerts

resource "google_monitoring_alert_policy" "hallucination" {
  project      = var.project_id
  display_name = "Agent Quality - Hallucination (Groundedness) Low"
  combiner     = "OR"

  conditions {
    display_name = "Groundedness Score < 0.9"
    condition_threshold {
      filter                  = "resource.type=\"aiplatform.googleapis.com/OnlineEvaluator\" AND metric.type=\"aiplatform.googleapis.com/online_evaluator/scores\" AND metric.labels.evaluation_metric_name=\"hallucination_v1\""
      comparison              = "COMPARISON_LT"
      threshold_value         = 0.9
      duration                = "300s"
      evaluation_missing_data = "EVALUATION_MISSING_DATA_INACTIVE"
      trigger {
        count = 1
      }
      aggregations {
        alignment_period   = "300s"
        per_series_aligner = "ALIGN_PERCENTILE_50"
      }
    }
  }

  user_labels = {
    created-with-google-skill = "cloud-monitoring-alert-configuration"
  }
}

Run Tooling Scripts

create_online_monitor: Use this script to provision the required Online Monitor and generate the telemetry metrics:
- Command: python3 scripts/agent_platform/create_online_monitor.py --project-id={gcp_project_id} --agent-resource-name={agent_resource_name} [--sampling-percentage={percentage}]
- Sampling Rate Recommendation: For production agents, configure a conservative sampling percentage (default: 10%) to control LLM evaluation costs. For details, refer to Continuous evaluation with online monitors.
check_telemetry: Verify whether telemetry is enabled on a Vertex AI Reasoning Engine using this script:
- Command: python3 scripts/agent_platform/check_telemetry.py --project-id={gcp_project_id} --agent-resource-name="projects/{project_id}/locations/{location}/reasoningEngines/{agent_id}"

Validate and Finalize

Apply Duration Buffers (Transient Glitches): To avoid alerts firing on transient spikes, always use a duration = "300s" (5 minutes) buffer to filter out transient scoring dips or evaluation outliers caused by temporary LLM judge congestion or edge-case query outliers.
Handle Script Failures: If check_telemetry.py or create_online_monitor.py fails unexpectedly, analyze the error message. Common issues include an invalid agent-resource-name format (must be the full path), missing permissions, or an unenabled API. Dynamically correct parameters and retry.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

What our scanner noticed· Low risk

Scripts that your agent may run

This skill includes scripts that your agent may run. Read them before you use it.

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

Google

Publisher

Google (Official)

Source

google/skills · skills/cloud/cloud-monitoring-alert-configuration · commit 2194d3b

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by Google or Google.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download cloud-monitoring-alert-configuration.zip