Generated banner for Agent Platform Troubleshooting
Generated logo for Agent Platform Troubleshooting

Agent Platform Troubleshooting

GoogleOfficialPublished by Google

  • Skill
  • Review and audit
  • Software development
  • Apache-2.0
  • No issues found

From the author

Troubleshoots Google Cloud Gemini Enterprise Agent Platform issues across Agent Gateway, Agent Connectivity Templates (ACT), Agent Registry, Agent Identity, Identity-Aware Proxy (IAP), Unified Access Policies (UAP), Model Armor, Semantic Governance Policies (SGP), Cloud DNS peering, Private CA TrustConfig, Next-Hop Secure Web Proxy (SWP), Explicit HTTPS Proxy (L4 AuthzPolicy), and Cloud Trace. Use when agent requests fail with 401/403/404/498/500/503 or Code 13, Egress request is not authorized, HTTP CONNECT fails with 404, Agent Runtime crashes on startup, or gateway/ACT/IAP/Model Armor/SGP configs fail validation. Don't use for general Google Cloud Identity and Access Management (IAM) debugging or networking issues unrelated to the Agent Platform stack.
— the description by Google

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured diagnostic framework for identifying and resolving complex issues within the Google Cloud Gemini Enterprise Agent Platform. It covers the entire stack, including Agent Gateway, Agent Connectivity Templates (ACT), Agent Registry, Identity-Aware Proxy (IAP), Model Armor, Semantic Governance Policies (SGP), and network-level configurations like DNS peering and proxy architectures. The skill enforces a strict, rule-based diagnostic flow to ensure efficient troubleshooting while preventing common misconfigurations, such as incorrect IAM assignments or unsupported multi-region registrations.

What it does

  • Analyzes Agent Platform logs to identify root causes for 401, 403, 404, 498, 500, and 503 errors.
  • Validates Agent Gateway, ACT, and registry configurations against platform best practices.
  • Provides specific remediation steps for IAP, Model Armor, and SGP policy enforcement issues.
  • Troubleshoots complex network egress scenarios including SWP, explicit proxies, and VPC-SC perimeters.
  • Generates a standardized diagnostic report with findings and actionable recommendations.
  • Enforces strict operational safety rules to prevent unauthorized mutations or excessive resource consumption.

When to use it

  • When agent requests fail with HTTP status codes like 403 (Egress not authorized) or 404 (HTTP CONNECT failure).
  • When Agent Runtime instances crash on startup due to connectivity or policy validation errors.
  • When troubleshooting configuration mismatches in Agent Gateway, ACT, or registry services.
  • When diagnosing VPC-SC perimeter blocks or issues with DNS peering and Private CA TrustConfig.
  • When investigating Model Armor or SGP verdict rationales for blocked requests.

Advantages

  • Reduces mean time to resolution by following a proven, rule-based diagnostic hierarchy.
  • Prevents common configuration pitfalls through mandatory pre-flight checks and strict usage rules.
  • Provides clear, actionable remediation steps tailored to the specific failure mode.
  • Ensures consistent diagnostic output across different project environments.
  • Maintains operational security by requiring the user to own all mutations.

Usage examples

“My agent is getting a 403 Egress request is not authorized error when calling an external API.”
The agent will initiate the diagnostic flow, checking IAP egressor bindings on the registry entry and verifying the relevant AuthzPolicy.
“Why is my Agent Gateway stuck in PROVISIONING state?”
The agent will check for PSC subnet IP exhaustion and provide guidance on expanding the CIDR range if necessary.
“My Agent Runtime crashes on startup with connection errors to telemetry.mtls.googleapis.com.”
The agent will verify if the required telemetry endpoints are registered and suggest updating the AuthorizationPolicy to allow access.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/google/agent-platform-troubleshooting/download
  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip agent-platform-troubleshooting.zip -d .agents/skills

Documentation by Google

Agent Platform Troubleshooting

Important: You MUST ONLY use the reference files located in this skill's references/ directory (references/field-manual.md, references/known-issues.md, references/agent-gateway.md, references/policies.md, references/agent-registry.md, references/agent-identity.md). Do NOT search for or read external playbooks outside this directory.

Diagnose issues across the Google Cloud Gemini Enterprise Agent Platform: Agent Gateway, Agent Connectivity Templates (ACT), Agent Registry (Services, Agents, MCP Servers, Endpoints), Agent Identity, IAP/UAP policies, Model Armor, Semantic Governance Policies (SGP), explicit/transparent proxies, and tracing. This skill produces a diagnostic report — findings and fix recommendations. It does not apply fixes; the user owns all mutations.

Usage Attribution

MUST READ: Before executing any gcloud, REST API, or MCP command, you MUST read and follow Usage Attribution. Prefix direct terminal gcloud calls (never generated scripts, Terraform, or user code) with CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:agent-platform-troubleshooting)" and append agent-skills/1.0 (skill:agent-platform-troubleshooting) to direct HTTP/REST/MCP User-Agent headers.

MANDATORY PRE-FLIGHT CHECKLIST (CHECK BEFORE RESPONDING OR CALLING TOOLS)

CRITICAL: Before generating ANY response or calling any tools, evaluate the user's prompt against Rules 1–14. When a Turn-0 rule matches, execute its directive immediately and STOP.

Rule 1: Out-of-Scope GCP IAM / GCS Queries

If the prompt asks about Compute Engine (GCE), Cloud Storage (GCS) buckets, or generic GCP IAM permissions unrelated to the Agent Platform stack:

  • IMMEDIATELY DECLINE ON TURN 0. DO NOT CALL ANY TOOLS OR GIVE IAM GUIDES.

  • Respond: "I decline to troubleshoot generic GCP IAM or GCS access issues, as they are out of scope for the Agent Platform Troubleshooting skill."

Rule 2: Strict Prohibition on Custom Discovery Scripts

If asked to write, generate, compile, or execute a custom Python or bash script to discover resources (e.g., active Agent Runtime instances):

  • DO NOT CALL ANY TOOLS OR WRITE/RUN ANY SCRIPTS.

  • RESPOND ON TURN 0: "I cannot write or execute custom Python or bash scripts for resource discovery. Custom discovery scripts are prohibited as they consume excessive turns and cause timeouts. Instead, use standard gcloud CLI commands (see Google Cloud SDK Installation) or curl REST API calls with application default credentials: gcloud agent-registry agents list --location=us-central1 or curl -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" "https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/reasoningEngines""

Rule 3: Consolidated Registry for Google APIs / Design Queries

If asked about registering multiple Agent Runtime, Cloud Resource Manager, or Google API interfaces, or structuring services in Agent Registry:

  • DO NOT CALL ANY TOOLS. RESPOND ON TURN 0 WITH:

    1. Consolidate ALL Google APIs under a single googleapis service entry in Agent Registry via gcloud agent-registry services create googleapis --endpoint-spec-type=no-spec.

    2. Explicitly state: "Do NOT register each Google API as a separate registry service entry, as separate entries cause resource clutter, complicate IAM policy management, and risk hitting registry quota limits."

    3. List all 8 required base FQDN interfaces (plus global https://aiplatform.googleapis.com when calling global Gemini models):

      • https://agentregistry.googleapis.com

      • https://aiplatform.mtls.googleapis.com

      • https://cloudresourcemanager.mtls.googleapis.com

      • https://iamcredentials.mtls.googleapis.com

      • https://telemetry.mtls.googleapis.com

      • https://{region}-aiplatform.mtls.googleapis.com

      • https://{region}-aiplatform.googleapis.com

      • https://aiplatform.{region}.rep.googleapis.com

    4. Refer to references/agent-registry.md §2 for the full command.

Rule 4: Cloud Run / Cloud Functions Egress 401/403 (OIDC vs IAP MCP)

If an agent gets a 401 or 403 egress error calling an MCP server or tool on Cloud Run or Cloud Functions:

  • DO NOT RUN LOG TOOLS. RESPOND ON TURN 0 WITH:

    1. Explain that direct Agent Identity (principalSet://... / principal://...) to standard Cloud Run OIDC authentication is not natively supported.

    2. Recommend Service Account impersonation in agent code to obtain an OIDC ID token, granting roles/iam.serviceAccountTokenCreator to the Agent Identity on the target SA (and roles/run.invoker to that SA on Cloud Run).

    3. Note that for Preview IAP-enabled Cloud Run MCP servers (--iap --functional-type=mcp-server, run.app short URL), authorize via IAP/UAP instead of roles/run.invoker (see references/known-issues.md BKI 21 and references/agent-identity.md).

Rule 5: Telemetry & Monitoring Endpoint Blocks on Startup

If Agent Runtime startup crashes (set_up()) with connection resets or handshake errors reaching telemetry.mtls.googleapis.com:

  • Evidence: Explicitly check whether all 4 monitoring/tracing endpoints are registered as Endpoints in Agent Registry (gcloud agent-registry endpoints list): telemetry.mtls.googleapis.com, monitoring.googleapis.com, trace.mtls.googleapis.com, and cloudtrace.googleapis.com.

  • Recommended Fix:

    1. Register missing telemetry/tracing interfaces via gcloud agent-registry services create --endpoint-spec-type=no-spec (which projects read-only Endpoints in Agent Registry for telemetry.mtls.googleapis.com, monitoring.googleapis.com, trace.mtls.googleapis.com, and cloudtrace.googleapis.com).

    2. Create or update an AuthorizationPolicy (AuthzPolicy / IAP / UAP) bound to the Gateway that allows the agent's principal set (identity) to access the telemetry endpoints (see references/known-issues.md BKI 23).

Rule 6: IAP Denial Troubleshooting (403 / Egress request is not authorized)

When logs or symptoms show 403 / Egress request is not authorized via IAP:

  1. Check IAP Egressor bindings on the registry entry FIRST (roles/iap.egressor via gcloud iap web get-iam-policy --resource-type=agent-registry bound to the server-generated child UUID from registryResource, not SERVICE_NAME).

  2. Create a registry entry (gcloud agent-registry services create) if none exists for the exact target hostname.

  3. Grant roles/iap.egressor on the registry entry (IAM v1 — set-iam-policy requires top-level {"bindings": [...]} JSON without a "policy" wrapper) or verify UAP AccessPolicy + CRM PolicyBinding (iapPolicyVersion: "V2", iap.googleapis.com/resources.egressViaIAP; see references/policies.md §2).

  4. Check IAP audit logs (protoPayload.serviceName="iap.googleapis.com") and verify an AuthorizationPolicy (AuthzPolicy) is bound to the Gateway.

  5. Explicitly warn: "Do NOT use roles/iap.tunnelResourceAccessor (which is for classic IAP TCP tunneling to GCE VMs, not Agent Gateway egress) and do NOT bypass IAP authentication."

Rule 7: PSC Subnet Exhaustion Speed Rule (PROVISIONING Hang)

When diagnosing gateway provisioning stuck or failing on subnet IP exhaustion:

  • DO NOT loop across regions. Run ONLY in the target region (default us-central1):

    1. gcloud network-services agent-gateways list --location=us-central1

    2. gcloud network-services agent-gateways describe <GW> --location=us-central1

    3. gcloud compute network-attachments describe <NA> --region=us-central1

    4. gcloud compute networks subnets describe <SUBNET> --region=us-central1

  • Check free IP count (Usable - Allocated = Free) before recommending any Network Attachment deletion; flag that a /28 subnet is easily exhausted and recommend expanding the CIDR range to at least /26 (known-issues.md BKI 19).

Rule 8: Multi-Region Manual Registration Prohibition (us / eu)

If asked about manually registering endpoints or services in us or eu:

  • DO NOT CALL ANY TOOLS. RESPOND ON TURN 0:

    1. "Manual endpoint/service registration is NOT supported in us or eu multi-region locations (us and eu are read-only roll-up locations for auto-discovered resources)." (Explicitly name BOTH us and eu.)

    2. "Register manually in a specific region (e.g., us-central1) or global."

Rule 9: VPC Service Controls (VPC-SC) Perimeter & ACT ALL_TRAFFIC

When diagnosing VPC-SC perimeter blocks or external API hangs under VPC-SC:

  1. Identify a VPC Service Controls perimeter block / boundary enforcement.

  2. Check that the Agent Connectivity Template (ACT) sets vpcEgress: ALL_TRAFFIC (native VPC-SC creation works out of the box since Sept 8, 2026 when vpcEgress: ALL_TRAFFIC is set) and that Cloud NAT is configured on the consumer VPC's PSC-I network attachment subnet.

  3. For legacy or strict custom perimeters blocking Google actuation identities, recommend creating ingress policies allowing:

  4. Explicitly state: "Do NOT disable VPC Service Controls or delete perimeters." Refer to references/known-issues.md BKI 18 & 28.

Rule 10: UAP Policy Binding Org Policy Blocker & Blanket DENY Crash
  • CUSTOM_ORG_POLICY_VIOLATION on gcloud iam policy-bindings create: Caused by constraints/iam.managed.disableAccessPolicyBinding (note: some docs misspell this plural). Apply an Org Policy override (enforce: false) at the target resource and wait 30 to 60 seconds for IAM propagation (BKI 24).

  • Startup crash after UAP destination.is_registered == false DENY: Explain that the blanket DENY blocked internal/unauthenticated SDK bootstrap and metadata calls (such as Cloud Resource Manager project lookups) during Reasoning Engine init; remove or scope the blanket DENY rule (BKI 5).

Rule 11: Agent Gateway Dual-Registry & Jurisdiction Invariants

When configuring or validating registries on an Agent Gateway:

  1. An Agent Gateway supports a maximum of two registries.

  2. When two registries are configured, exactly one must be global and the other must be regional or multi-regional.

  3. Binding two regional registries (e.g., us-central1 + europe-west1), two multi-regionals, two globals, or a mismatched jurisdiction (e.g., us-central1 gateway + eu registry) is invalid and rejected with an HTTP 400 validation error (references/agent-gateway.md §4, BKI 26).

Rule 12: Cross-Project Runtime-to-Gateway Binding Diagnosis

When Reasoning Engine in Project 1 fails to bind to a Gateway in Project 2:

  1. Deploying Identity: Needs networkservices.agentGateways.use and networkservices.agentGateways.get (or roles/networkservices.viewer) in Project 2.

  2. Vertex AI Service Agent: Grant roles/networkservices.viewer or custom role ae_agw_cross_project_sa (alias ar_agw_cross_project_sa) to service-PROJECT_1_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.com in Project 2.

  3. Regional Colocation: Verify both runtime and gateway are in the same region.

  4. CAA Opt-Out: Set GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICES to False (references/known-issues.md BKI 10, references/agent-gateway.md §11).

Rule 13: Downstream SWP + PBR vs. Explicit Forward HTTPS Proxy

Distinguish the two supported consumer-VPC proxy architectures:

  • Pattern A — Transparent Next-Hop SWP + PBR (no client HTTPS_PROXY needed):

    1. Explain that PSC endpoints cannot be route next-hops, SWP behind PSC Service Attachments only supports explicit proxy mode (not transparent next-hop mode), and static routes cannot be used because Network Attachments cannot have VM network tags.

    2. Deploy SWP directly in the consumer VPC with type: SECURE_WEB_GATEWAY and routingMode: NEXT_HOP_ROUTING_MODE on an internal IP on the PSC-I subnet (e.g., 10.20.1.250), with a /26 REGIONAL_MANAGED_PROXY ACTIVE subnet.

    3. Configure Policy-Based Routing (PBR) matching source CIDR 10.20.1.0/24 to --next-hop-ilb-ip=10.20.1.250 (priority 200) + fallback PBR (1000).

    4. Create Cloud NAT with ENDPOINT_TYPE_SWG (nats create ... --endpoint-types=ENDPOINT_TYPE_VM,ENDPOINT_TYPE_SWG), and allowlist <region>-aiplatform.mtls.googleapis.com in SWP CEL (BKI 33).

  • Pattern B — Explicit Forward HTTPS Proxy (HTTPS_PROXY + L4 AuthzPolicy):

    1. Without L4 bypass, HTTPS_PROXY=https://proxy.corp.internal:443 fails on HTTP CONNECT with HTTP 404 because Agent Gateway's L7 HTTP manager terminates TLS and rejects nested CONNECT.

    2. Import a v1beta1 L4 AuthzPolicy via gcloud beta network-security authz-policies import with action: ALLOW and networkRules matching to.operations[].snis (omit target.loadBalancingScheme and ports; max 5 rules; omit from or include 7.0.0.0/8 + PSC-I CIDR in from.sources[].ipBlocks).

    3. Listen with TLS on :443, allow the consumer PSC-I subnet CIDR on VPC firewall/proxy ACLs (never 7.0.0.0/8), set NO_PROXY/no_proxy to .googleapis.com,googleapis.com,metadata.google.internal,169.254.169.254,localhost,127.0.0.1, bundle agentGatewayCard.rootCertificates + Private CA in container env vars (SSL_CERT_FILE, REQUESTS_CA_BUNDLE), and warn L4 ALLOW bypasses L7 IAP/UAP, Model Armor, SGP, and httpRules for matched SNIs (BKI 34).

Rule 14: DNS Peering, TrustConfig, Model Armor, SGP, MCP, Identity, & Lifecycle
  • DNS Peering & Service Agent: Public FQDNs resolve via public recursive DNS. Private DNS requires dnsPeeringConfig (domains array with trailing .; ACT uses string targetNetwork only, whereas inline AgentGateway.networkConfig.dnsPeeringConfig also requires targetProject). Cloud DNS caps total peering hops at 2 (Gateway consumes Hop 1 → max 1 consumer peering hop; a 2nd hop fails preflight or returns SERVFAIL → 503, BKI 35). Grant roles/dns.peer, roles/dns.reader, roles/compute.networkViewer, and roles/compute.networkUser to service-GATEWAY_PROJECT_NUMBER@gcp-sa-agentgateway.iam.gserviceaccount.com (and compute.networkAttachments.use to caller, BKI 9). Never use authoritative "." private root zones ("." requires a Cloud DNS Private Forwarding Zone).

  • Private CA TrustConfig & Immutable ACT: Set egressNetworkConfig.tlsConfig.trustConfig with additionalRoots: PUBLICLY_TRUSTED_ROOTS (omitting breaks *.googleapis.com). ACT fields are immutable — create a new ACT and rebind via PATCH with updateMask=agentConnectivityTemplate (BKI 6).

  • Model Armor (CONTENT_AUTHZ): Ingress (CLIENT_TO_AGENT, max 1, ADK streamQuery only) uses service-AGENT_PROJECT_NUMBER@gcp-sa-aiplatform-re.iam.gserviceaccount.com; Egress (AGENT_TO_ANYWHERE, max 4) uses service-GATEWAY_PROJECT_NUMBER@gcp-sa-dep.iam.gserviceaccount.com. Scope httpRules when to exclude gRPC and streams >60 seconds (Stream removed, BKI 13).

  • Semantic Governance Policies (SGP): Uses gcp-sa-aiplatform, non-streaming LLM calls only, .contains() + !request.headers['content-type'].startsWith('application/grpc') in when, and full-map updates on PATCH ?updateMask=metadata so sgpEnforcementMode: "DRY_RUN" and sgpUserIntentMatching are preserved (BKI 38).

  • Native MCP, Identity, Trace, & Teardown: Set baseProtocolMethodsOption: MATCH_BASE_PROTOCOL_METHODS on MCP ALLOW rules (BKI 39). Orgless trust domains use agents.global.proj-PROJECT_NUMBER.system.id.goog (proj-, never project-); never grant roles/aiplatform.expressUser or project-wide roles/aiplatform.user to Agent Identities. For Cloud Trace (TelemetryPolicy), always recommend enabling cloudtrace.googleapis.com (gcloud services enable cloudtrace.googleapis.com) and granting roles/cloudtrace.agent to compute-system, gcp-sa-networksecurity, and gcp-sa-agentgateway (see references/agent-gateway.md §8.3 and references/field-manual.md Step 14.4). Tear down in strict 7-step order (workloads → UAP bindings → AuthzPolicy → extensions → AgentGateway → ACT → VPC) and use unique gateway IDs if deletion tombstones collide (BKIs 25, 30).

Quick Symptom Triage Matrix

Symptom / ErrorPrimary SuspectReference
403 / Egress request is not authorizedUnregistered FQDN, missing roles/iap.egressor / UAP binding, or PABRules 4, 6; references/policies.md §2; BKIs 4, 7, 8, 15
HTTP 404 on HTTP CONNECT (HTTPS_PROXY)Missing gcloud beta L4 AuthzPolicy (networkRules snis)Rule 13B; references/agent-gateway.md §6.3; BKI 34
HTTP 498 immediately after binding runtimeEnforcing gateway bound before allowlisting essential platform APIsreferences/agent-gateway.md §2; BKI 37
500 / PERMISSION_DENIED / Stream removedModel Armor SA roles (gcp-sa-dep vs gcp-sa-aiplatform-re) or 60s stream capRule 14; references/policies.md §12; BKI 13
503 / NXDOMAIN / SERVFAIL on private host>2-hop DNS peering, authoritative "." zone, Packet Intercept, or SWPRules 13A, 14; references/agent-gateway.md §3, §6; BKIs 33, 35, 36
Config validation failed / PERMISSION_DENIED on importDisabled agentregistry.googleapis.com, missing caller/gcp-sa-agentgateway IAM, or bad NA URIRule 14; BKIs 9, 20, 26
HTTP 409 ABORTED / 400 already being usedActive control-plane LRO mutex (~2 min), tombstone collision, or reverse teardownreferences/agent-gateway.md §9, §10; BKIs 25, 29, 30
Startup crash (reasoning_engine_stderr)Blocked telemetry/CRM bootstrap, UAP blanket DENY, or SGP gRPC interceptionRules 5, 10, 14; BKIs 1, 5, 17, 23, 38

When to use this skill

Trigger on Agent Gateway, Agent Connectivity Templates (ACT), Agent Registry, Agent Identity, IAP/UAP policies, Model Armor, Semantic Governance Policies (SGP), SWP/Explicit Proxy egress, Private CA TrustConfig, Cloud DNS peering, Cloud Trace TelemetryPolicy, or Gemini Enterprise / Agent Runtime / GKE / Cloud Run gateway bindings. Do not use for generic Google Cloud IAM, GCS, GCE, or networking issues unrelated to the Agent Platform stack.

Required context & Hypothesis Rules

Gather PROJECT_ID, PROJECT_NUMBER, LOCATION (default us-central1), AGENT_ID, AGENT_GATEWAY_NAME, Agent Identity (principal:// or principalSet://), exact error + timestamp, and target destination. If missing, ask or state unknowns using placeholders — never scan across projects or loop over regions. Before Step 1 queries, formulate at most 3 hypotheses anchored to "Why did it start failing now?" (e.g., recent Terraform/config changes).

Diagnostic flow

Follow in order (copy-paste commands in references/field-manual.md):

  • Step 0 — Pre-Flight & Access: Evaluate Rules 1–14. Verify project access (gcloud projects describe $PROJECT_ID).

  • Step 1 — Agent & Startup Logs: Classify error (403, 404 CONNECT, 498, 500, 503, crash, timeout). Check PSC subnet IPs, ACT ALL_TRAFFIC + Cloud NAT, and agentGatewayCard.rootCertificates CA trust.

  • Step 2 — Gateway Logs: Query networkservices.googleapis.com/Gateway (filtering -httpRequest.requestMethod="CONNECT" when inspecting L7 requests) to find the exact failing SNI/hostname and serviceExtensionsInfo.

  • Step 3 — IAP / Model Armor / SGP Logs: Check iapPolicyVersion (V1 vs V2), DRY_RUN vs enforced, Model Armor callout status, or logs/semantic-governance-policy verdicts and rationales.

  • Step 4 — Registry State: Verify exact hostname permutations in gcloud agent-registry {services,endpoints,mcp-servers,agents} list and confirm dual-registry + jurisdiction rules.

  • Step 5 — Identity & Policies: Check roles/iap.egressor via --resource-type=agent-registry on the child UUID (V1), UAP AccessPolicy/PolicyBinding + constraints/iam.managed.disableAccessPolicyBinding (V2), PAB overrides, and least-privilege Agent Identity roles (proj- for orgless).

  • Step 6 — AuthzPolicy & Extensions: Verify L4 networkRules vs L7 httpRules, MCP MATCH_BASE_PROTOCOL_METHODS, Model Armor / SGP when exclusions, ExtensionBinding scope, and policy caps (max 4 egress, max 1 ingress CONTENT_AUTHZ).

  • Step 7 — Connectivity, DNS, & TLS: Verify ACT TrustConfig (PUBLICLY_TRUSTED_ROOTS), gcp-sa-agentgateway IAM, <=2 DNS peering hops, no Packet Intercept on the consumer VPC, and SWP/Explicit Proxy settings.

  • Step 8 — PrincipalSet & Trace Verification: Test 1:1 principal:// binding if principalSet:// lags; for Cloud Trace, enable cloudtrace.googleapis.com, verify v1beta1 TelemetryPolicy (telemetryTarget referencing agentGateways/* and object parentBasedSampling: {enabled: true, samplingRate: 1.0}, never string ENABLED), and grant roles/cloudtrace.agent to service-<PROJECT_NUMBER>@compute-system.iam.gserviceaccount.com, service-<PROJECT_NUMBER>@gcp-sa-networksecurity.iam.gserviceaccount.com, and service-<PROJECT_NUMBER>@gcp-sa-agentgateway.iam.gserviceaccount.com.

Tools & Layered References

Use mcp__gcloud__run_gcloud_command (or gcloud --quiet), mcp__gcloud-observability__list_log_entries (or gcloud logging read), mcp__google-dev-knowledge__search_documents / get_documents / answer_query, and curl REST GETs, plus bundled references (load only the smallest set needed):

Output report

Always produce a read-only diagnostic report using this exact structure:

# Agent Platform Diagnostic — <one-line summary>

## Context
- Project: <id> (<number>) | Location: <region>
- Agent: <agent_id> | Gateway: <gateway_name>
- Symptom: <exact error message and when it started>

## Evidence gathered
- Agent log query: <filter, summary>
- Gateway log query: <filter, exact failing hostname / extension status>
- IAP / Policy log query: <filter, decision + enforcement mode>
- Registry & Policy state: <entries, AuthzPolicy bindings, roles/iap.egressor or UAP>

## Root cause hypothesis
<ranked root cause(s)>

## Why this fits the evidence
<connect log/config evidence to root cause>

## Recommended fix
<ordered gcloud / curl / YAML / Terraform changes for the user to apply>

## What to verify after the fix
<verification queries>

## Open questions / unknowns
<missing context or unverified items>

## Appendix: Raw Logs & Verified Links
- **Cloud Logging Filter**: <exact copy-pasteable filter or deep link>
- **Raw Logs**: <untruncated Agent, Gateway, and IAP/Policy log entries>

Principles

  • Exact hostname match first: Compare gateway log hostnames directly against Agent Registry.

  • Layered default-deny: Registry → Gateway AuthzPolicy → Extension → IAP/UAP/Model Armor/SGP → PAB (PAB overrides IAM Allow).

  • L4 bypass vs L7 governance: L4 networkRules enables HTTP CONNECT explicit proxies (:443) but bypasses L7 IAP, Model Armor, and SGP for matched SNIs.

  • Read-only, single-region, non-interactive: Pull evidence first, never mutate resources, check us-central1 unless logs indicate otherwise, and pass --quiet.

Files

references/agent-registry.md

Agent Registry — Debugging Reference

Curated reference for diagnosing why agents, MCP servers, and endpoints registered in Google Cloud Agent Registry are not accepting traffic through Agent Gateway / IAP.

Scope: Gemini Enterprise Agent Platform — Agent Registry (v1 GA). Core commands use gcloud agent-registry (gcloud beta agent-registry does not exist, and endpoints is read-only with no create subcommand; all manual writes go through gcloud agent-registry services). Standalone skills use gcloud agent-registry skills (list / describe).

Table of Contents

  • TL;DR for Debugging: Lines 29-149

  • 1. The Data Model in One Picture: Lines 150-197

  • 2. Default-Deny + Hostname Matching: Lines 198-338

  • 3. Resource Type — Agents: Lines 339-472

  • 4. Resource Type — MCP Servers: Lines 473-561

  • 5. Resource Type — Endpoints: Lines 562-705

  • 6. IAM Scope: Registry-Wide vs Per-Resource: Lines 706-845

  • 7. Common Registration Errors and Where to Look: Lines 846-877

  • 8. Discovery via the Agent Registry MCP Server: Lines 878-906

  • 9. Quick Reference — Every gcloud Subcommand: Lines 907-971

  • Source Index: Lines 972-998


TL;DR for Debugging

Read these first when something is broken.

  1. Agent Registry is default-deny at the gateway. Agent Gateway uses Identity-Aware Proxy (IAP) to enforce authorization:

    • UAP (Policy V2): When the gateway uses iapPolicyVersion: "V2", Gatekeeper queries the attached registries. If the destination matches, Gatekeeper sets destination.is_registered = true and exposes rich CEL metadata under destination.agent_registry.*. If not cataloged, destination.is_registered = false.

    • Legacy IAM v1: When using legacy v1, if a target hostname is not registered as a Service (and a matching roles/iap.egressor binding does not exist on the registry entry for the calling agent's principal), egress is blocked. Source: docs.cloud.google.com/iap/docs/agent-overview.

  2. There are exactly three discoverable resource types, and they are projections of the same writable Service:

    • Agent (read-only Agent)

    • MCP server (read-only McpServer)

    • Endpoint (read-only Endpoint) You always create/update/delete via services. You list/describe via agents, mcp-servers, endpoints, or services. Source: docs.cloud.google.com/agent-registry/concepts.

  3. Hostname matching is exact. A single Google API can be reached at five+ distinct hostnames (base, mTLS, locational, locational-mTLS, REP regional). Each hostname is a separate Service registration. If your agent calls bigquery.us-central1.rep.googleapis.com but you only registered bigquery.googleapis.com, the call is denied. See the "Hostname Matrix" section.

  4. Manual registration is not supported in us or eu multi-region locations. us and eu are read-only roll-up discovery locations for automatically registered Gemini Enterprise components and standalone skills. Always register Service entries and Binding resources in a specific region (e.g., us-central1, europe-west1) or global.

  5. Two predefined IAM roles control the registry itself:

    • roles/agentregistry.viewer (agentregistry.{agents,endpoints,mcpServers}.list) — discover entries and required by the caller importing an AgentGateway referencing the registry.

    • roles/agentregistry.editor — create/update/delete Service resources. These are project-scoped catalog permissions, not runtime traffic permissions.

  6. Traffic permissions live on IAP (Policy V1 vs. UAP Policy V2):

    • Policy V1 (gcloud iap web): Grant roles/iap.egressor (iap.webServiceVersions.egressViaIAP) with --resource-type=agent-registry always set, optionally scoped with --agent=AGENT_UUID, --mcp-server=MCP_UUID, or --endpoint=ENDPOINT_UUID (using the server-generated registryResource basename, never SERVICE_NAME).

    • UAP Policy V2 (iapPolicyVersion: "V2"): Grant iap.googleapis.com/resources.egressViaIAP via gcloud iam access-policies and gcloud iam policy-bindings.

  7. Identifiers matter:

    • SERVICE_NAME / SERVICE_ID — your chosen ID (4–63 chars [a-z0-9-]) for gcloud agent-registry services.

    • registryResource UUID — server-generated child ID under /endpoints/, /mcpServers/, or /agents/ required by gcloud iap web.

    • Agent / MCP server URN — immutable identifier (urn:mcp:..., urn:agent:...) used by discovery tools and ExtensionBinding matchers.

    • Resource URI — runtime URI embedded in the SPIFFE principal.

  8. Specs have a hard 10 KB cap and use plain file paths (no @). Pass --mcp-server-spec-content=toolspec.json or --agent-spec-content=agent-card.json (<= 10 KB; gcloud arg_json rejects curl-style @file.json).

  9. Automatic registration vs. manual: Agent Runtime (SDK), Google Workspace, Gemini Enterprise, annotated GKE (registry.gke.io/functional-type: "MCP_SERVER"), Cloud Run services with --functional-type={agent|mcp-server}, and enabled Google/Google Cloud remote MCP servers (created in global only) register automatically within the governance project only. Cross-project workloads, standard Cloud Run services, on-prem, and third-party APIs require manual gcloud agent-registry services create.

    • Cloud Run .run.app vs. .a.run.app trap: Automatic registration records the deterministic regional URL (https://<SERVICE>-<PROJECT_NUMBER>.<REGION>.run.app). Calling the legacy hash URL (https://<SERVICE>-<HASH>-uc.a.run.app) fails exact hostname matching unless manually registered.

  10. First debug command: confirm the exact target URL is registered (gcloud agent-registry endpoints list --project=$PROJECT_ID --location=$LOCATION and gcloud agent-registry services list --filter="interfaces.url:FRAGMENT").

  11. Governance project scope (no automatic cross-project discovery). Automatic registration only discovers resources inside the governance project where agentregistry.googleapis.com is enabled. Remote agents and MCP servers in separate Google Cloud projects must be manually registered in the central governance project's registry.

  12. Agent Gateway Dual-Registry & Jurisdiction Rules: An AgentGateway supports a maximum of two registries in registries[]. When two registries are configured, exactly one must be global and the other must be regional (e.g., us-central1) or multi-regional (us, eu).

    • Binding two regional registries (e.g., us-central1 and europe-west1) is invalid and rejected with an HTTP 400 validation error.

    • Jurisdiction alignment: A regional gateway can only reference a regional registry in the same region or a multi-region registry in the same data residency jurisdiction (e.g., us-central1 gateway + eu registry is rejected with InvalidArgument).

    • Gateway import preflight: If agent-gateways import fails with Config validation failed and missing permissions: agentregistry.agents.list, enable agentregistry.googleapis.com on the registry project and ensure the deploying caller holds roles/agentregistry.viewer (agentregistry.{agents,endpoints,mcpServers}.list).

  13. REST API Path Extensions & Explicit Forward Proxies: SDK clients such as Zendesk append .json (/api/v2/tickets.json) — account for .json, / subpaths, and ? query strings in UAP CEL rules. Note that an explicit forward HTTPS_PROXY tunnel (HTTP CONNECT) requires a v1beta1 L4 AuthzPolicy (networkRules matching SNIs), not an Agent Registry entry for the proxy host (see references/agent-gateway.md and BKI #34).


1. The Data Model in One Picture

                 WRITE side                          READ side (read-only projections)
              +--------------+                    +----------+
              |              |  endpoint-spec --> | Endpoint |
              |   Service    |  mcp-server-spec ->| McpServer|
              | (writable)   |  agent-spec -----> |  Agent   |
              +--------------+                    +----------+
                ^                                        ^
         create/update/delete                     get/list/search
         (services subcommand)                    (endpoints / mcp-servers /
                                                   agents subcommand)
  • The Service resource is the only thing you write.

  • Agent Registry automatically generates the corresponding Agent, McpServer, or Endpoint projection based on the *-spec-type you passed at create time.

  • The mutually exclusive *-spec-type flag at creation determines the read-only projection (spec types cannot be changed on services update):

    services create Spec FlagAllowed --interfacesRead-Only Projection
    --endpoint-spec-type=no-specRequired (1+ URLs)Endpoint
    --mcp-server-spec-type={tool-spec|no-spec}Required (protocolBinding=JSONRPC)McpServer
    --agent-spec-type=a2a-agent-cardMust be omitted (read from agent-card.json)Agent (with A2A skills)
    --agent-spec-type=no-specRequired (REST URL)Agent (non-A2A REST)

    (Standalone Skill resources do not use Service; inspect via gcloud agent-registry skills list --location=LOCATION --project=PROJECT_ID.)

Source: docs.cloud.google.com/agent-registry/overview, docs.cloud.google.com/agent-registry/concepts.

URN formats (memorize these)
  • Manually registered MCP server URN: urn:mcp:projects-PROJECT_NUMBER:projects:PROJECT_NUMBER:locations:REGION:agentregistry:SERVICE_ID

  • Google Cloud remote MCP server URN: urn:mcp:googleapis.com:projects:PROJECT_NUMBER:locations:global:SERVICE_NAME

If a discovery query returns no results for what you think is a registered server, check the URN format vs the location you registered against.

Source: docs.cloud.google.com/agent-registry/concepts.


2. Default-Deny + Hostname Matching

This is the single most common debugging failure mode. Read this whole section.

Why default-deny

Agent Gateway is the egress proxy for agents. It is built on IAP. IAP only forwards a request if there is an IAM allow policy granting roles/iap.egressor to the calling agent's principal on a registry resource that matches the request. If no Service matches the destination hostname, no policy can match, and the request is denied.

Source: docs.cloud.google.com/iap/docs/agent-overview.

Hostname Matrix — register every form an agent might call

A single Google Cloud API can be addressed via several distinct hostnames. Each one is a separate registry entry. The canonical set includes the following; if any are missing for an API your agent uses, expect denial.

VariantHostname patternExample (bigquery, us-central1)
Base / globalSERVICE.googleapis.combigquery.googleapis.com
mTLSSERVICE.mtls.googleapis.combigquery.mtls.googleapis.com
LocationalLOCATION-SERVICE.googleapis.comus-central1-bigquery.googleapis.com
Locational mTLSLOCATION-SERVICE.mtls.googleapis.comus-central1-bigquery.mtls.googleapis.com
REP (public)SERVICE.LOCATION.rep.googleapis.combigquery.us-central1.rep.googleapis.com
REP (private/PSC)SERVICE.LOCATION.p.rep.googleapis.combigquery.us-central1.p.rep.googleapis.com
  • Locational endpoints specify the region/multi-region in the URL prefix. Format: LOCATION-SERVICE.googleapis.com. Locational endpoints don't support data-residency-compliant connections from on-premises and don't support failure-domain isolation; REP is replacing them.

  • REP (Regional Endpoint) confines TLS termination and traffic to the specified region. Format: SERVICE.REGION.rep.googleapis.com. Private REP via PSC adds .p..

  • The base/global hostname terminates TLS at the nearest edge and provides no regional isolation.

Source: docs.cloud.google.com/docs/security/compliance/restrict-endpoint-usage.

Warning: MANDATORY GOOGLE APIS CONSOLIDATION RULE: NEVER recommend or register separate Service resources for individual Google APIs (such as Agent Runtime, Cloud Resource Manager, IAM, or Telemetry). Do NOT register each Google API as a separate service entry, as separate service entries cause resource clutter, complicate IAM policy management, and risk hitting registry quota limits. ALL Google APIs MUST ALWAYS be consolidated under a single Agent Registry Service named googleapis using multiple interfaces.

Instead of registering dozens of separate Service resources for each Google API and its regional/mTLS permutations, consolidate all Google APIs under a single Agent Registry Service named googleapis using multiple interfaces. This reduces resource clutter, simplifies IAM policy management (granting roles/iap.egressor once), and avoids registry quota limits.

The 8 Mandatory Base Interfaces + Essential Platform Families

For every Agent Runtime deployment behind an ALL_TRAFFIC Egress Agent Gateway, the consolidated googleapis service must include the 8 base FQDN interfaces:

  1. https://agentregistry.googleapis.com

  2. https://aiplatform.mtls.googleapis.com

  3. https://cloudresourcemanager.mtls.googleapis.com

  4. https://iamcredentials.mtls.googleapis.com

  5. https://telemetry.mtls.googleapis.com

  6. https://{region}-aiplatform.mtls.googleapis.com (e.g., us-central1)

  7. https://{region}-aiplatform.googleapis.com

  8. https://aiplatform.{region}.rep.googleapis.com

Depending on model availability and runtime features, include these additional interfaces from the 9 essential Google Cloud API families on the same googleapis service:

  • Global & Multi-Region Foundation Models: Many Gemini models only expose global or multi-region (us, eu) endpoints. Add https://aiplatform.googleapis.com (alongside aiplatform.mtls.googleapis.com) and, if targeting us/eu, https://us-aiplatform.googleapis.com, https://us-aiplatform.mtls.googleapis.com, and https://aiplatform.us.rep.googleapis.com. Calling an unsupported regional model returns HTTP 404 NOT_FOUND; calling global without registering aiplatform.googleapis.com drops TLS at 240.0.0.2:443 (UNEXPECTED_EOF).

  • Tracing, Logging, Monitoring, Storage, & Artifact Registry: Add https://cloudtrace.googleapis.com, https://cloudtrace.mtls.googleapis.com, https://trace.googleapis.com, https://trace.mtls.googleapis.com, https://monitoring.googleapis.com, https://monitoring.mtls.googleapis.com, https://logging.googleapis.com, https://logging.mtls.googleapis.com, https://storage.googleapis.com, https://storage.mtls.googleapis.com, https://artifactregistry.googleapis.com, and https://artifactregistry.mtls.googleapis.com.

  • Multi-Region .rep.mtls NXDOMAIN Pitfall: Google Cloud does not publish DNS records for aiplatform.us.rep.mtls.googleapis.com or aiplatform.eu.rep.mtls.googleapis.com. On older google-genai SDK versions with AGENT_IDENTITY enabled and location="us" or "eu", automatic mTLS URL rewriting fails with NXDOMAIN (Name or service not known). Fix: upgrade google-genai, target global or a single region, or set GOOGLE_API_USE_MTLS_ENDPOINT=never and GOOGLE_API_USE_CLIENT_CERTIFICATE=false.

Creating Consolidated Service via gcloud
gcloud agent-registry services create googleapis \
    --project=$PROJECT_ID --location=$LOCATION \
    --display-name="Google APIs" \
    --endpoint-spec-type=no-spec \
    --interfaces=url=https://agentregistry.googleapis.com,protocolBinding=JSONRPC \
    --interfaces=url=https://aiplatform.googleapis.com,protocolBinding=JSONRPC \
    --interfaces=url=https://aiplatform.mtls.googleapis.com,protocolBinding=JSONRPC \
    --interfaces=url=https://cloudresourcemanager.mtls.googleapis.com,protocolBinding=JSONRPC \
    --interfaces=url=https://iamcredentials.mtls.googleapis.com,protocolBinding=JSONRPC \
    --interfaces=url=https://telemetry.mtls.googleapis.com,protocolBinding=JSONRPC \
    --interfaces=url=https://cloudtrace.mtls.googleapis.com,protocolBinding=JSONRPC \
    --interfaces=url=https://trace.mtls.googleapis.com,protocolBinding=JSONRPC \
    --interfaces=url=https://monitoring.googleapis.com,protocolBinding=JSONRPC \
    --interfaces=url=https://$LOCATION-aiplatform.mtls.googleapis.com,protocolBinding=JSONRPC \
    --interfaces=url=https://$LOCATION-aiplatform.googleapis.com,protocolBinding=JSONRPC \
    --interfaces=url=https://aiplatform.$LOCATION.rep.googleapis.com,protocolBinding=JSONRPC
Quick "is the hostname registered?" check
gcloud agent-registry services list \
  --project=$PROJECT_ID \
  --location=$LOCATION \
  --format="table(name.basename(), interfaces.url)" \
  --filter="interfaces.url:HOSTNAME_FRAGMENT"

Replace HOSTNAME_FRAGMENT with e.g. bigquery.us-central1.rep.

Source: hostname patterns derived from docs.cloud.google.com/docs/security/compliance/restrict-endpoint-usage; registration flow from docs.cloud.google.com/agent-registry/register-endpoints.


3. Resource Type — Agents

Concept

Agents: Autonomous actors that possess specific skills. Skills represent an agent's high-level capabilities and are a primary mechanism for discovery. -- docs.cloud.google.com/agent-registry/concepts

Two kinds: - A2A-compliant agents -- implement Agent2Agent. Registry scans their agent-card.json to extract skills. - Standard REST agents -- registered as NO_SPEC type when no A2A spec exists.

Automatic registration sources: Agent Runtime (via SDK), Google Workspace built-ins, Gemini Enterprise built-ins, GKE deployments with the Agent Registry functional-type annotation. Everything else is manual.

List
gcloud agent-registry agents list \
  --project=PROJECT_ID \
  --location=REGION

Filter:

gcloud agent-registry agents list \
  --project=PROJECT_ID \
  --location=REGION \
  --filter="displayName='DISPLAY_NAME'"
Describe
gcloud agent-registry agents describe AGENT_NAME \
  --project=PROJECT_ID \
  --location=REGION

Output exposes core details, skills (if A2A), and Resource URI -- the runtime location of the agent, which is what gets embedded in the agent principal for IAM.

Update (via services, not agents)

Display name / description:

gcloud agent-registry services update AGENT_NAME \
  --project=PROJECT_ID \
  --location=REGION \
  --display-name="New name" \
  --description="Updated description"

Endpoint URL:

gcloud agent-registry services update AGENT_NAME \
  --project=PROJECT_ID \
  --location=REGION \
  --interfaces=url=ENDPOINT_URL,protocolBinding=PROTOCOL

Valid protocolBinding: HTTP_JSON, GRPC, JSONRPC.

Agent spec payload (max size 10 KB to satisfy gRPC control plane metadata header limits):

gcloud agent-registry services update AGENT_NAME \
  --project=PROJECT_ID \
  --location=REGION \
  --agent-spec-content=agent-card.json
Delete

For automatically registered agents: delete from the source runtime. For manually registered agents:

gcloud agent-registry services delete AGENT_NAME \
  --project=PROJECT_ID \
  --location=REGION
IAM model for agents
  • Registry permission to manage: roles/agentregistry.editor on the project.

  • Agent-to-agent egress (Policy V1): extract the generated Agent UUID (basename of registryResource) and bind roles/iap.egressor with --resource-type=agent-registry:

    AGENT_UUID=$(basename "$(gcloud agent-registry services describe AGENT_NAME \
      --project=PROJECT_ID --location=REGION --format='value(registryResource)')")
    gcloud iap web set-iam-policy agents-iap-policy.json \
      --project=PROJECT_ID \
      --resource-type=agent-registry \
      --agent="$AGENT_UUID" \
      --region=REGION

    The policy's members uses the calling agent's principal (principal://agents.global.org-123456789012.system.id.goog/resources/aiplatform/projects/9876543210/locations/us-central1/reasoningEngines/12345, orgless agents.global.proj-9876543210.system.id.goog/..., or WIF principal://iam.googleapis.com/...).

Common pitfalls
  • Agent in another project missing from central registry: automatic registration only discovers workloads inside the governance project. Manually register cross-project agents with gcloud agent-registry services create in the governance project (never in us or eu).

  • Agent appears in services list but not agents list: wrong spec type at creation (--agent-spec-type=a2a-agent-card or --agent-spec-type=no-spec required).

  • Cloud Run agent missing from registry: standard Cloud Run services do not auto-register unless deployed with gcloud beta run deploy --functional-type=agent --identity-type=agent-identity.

  • Gemini Enterprise egress ignores Agent Gateway: Gemini Enterprise requires two toggles — (1) PATCH the Engine on the regional endpoint (https://{global-|us-|eu-}discoveryengine.googleapis.com/v1/projects/PROJECT_NUMBER/locations/GE_LOCATION/collections/default_collection/engines/ENGINE_ID?updateMask=agentGatewaySetting.defaultEgressAgentGateway.name) and (2) set "use_agent_gateway_egress": true on each imported MCP/A2A DataConnector (see BKI #37).

  • A2A agent missing skills: agent-card.json exceeds 10 KB, or --interfaces was passed alongside --agent-spec-type=a2a-agent-card (forbidden).

Sources: docs.cloud.google.com/agent-registry/manage-agents, docs.cloud.google.com/agent-registry/register-agents, docs.cloud.google.com/gemini-enterprise-agent-platform/govern/policies/configure-iam-policies.


4. Resource Type — MCP Servers

Concept

MCP servers: Providers of standardized data resources and tools. Tools are deterministic functions exposed by an MCP server that agents can invoke to perform specific actions. -- docs.cloud.google.com/agent-registry/concepts

Official Google and Google Cloud remote MCP servers are auto-registered on API enable. External / custom MCP servers (e.g. Cloud Run-hosted FastMCP) need manual registration with a tool spec.

Register
gcloud agent-registry services create SERVER_NAME \
  --project=PROJECT_ID \
  --location=REGION \
  --display-name="DISPLAY_NAME" \
  --mcp-server-spec-type=tool-spec \
  --mcp-server-spec-content=toolspec.json \
  --interfaces=url=SERVER_URL,protocolBinding=JSONRPC
  • protocolBinding for MCP must be JSONRPC (jsonrpc), and toolspec.json must be <= 10 KB (do not prefix the path with @). Use --mcp-server-spec-type=no-spec (without --mcp-server-spec-content) when not uploading a static tool catalog.

List & Describe
gcloud agent-registry mcp-servers list --project=PROJECT_ID --location=REGION
gcloud agent-registry mcp-servers describe MCP_SERVER_UUID --project=PROJECT_ID --location=REGION
Update tool spec & Delete
gcloud agent-registry services update SERVER_NAME \
  --project=PROJECT_ID --location=REGION \
  --mcp-server-spec-content=toolspec.json

gcloud agent-registry services delete SERVER_NAME --project=PROJECT_ID --location=REGION
IAM model for MCP servers
  • Discover / manage catalog: roles/agentregistry.viewer / roles/agentregistry.editor.

  • Call Agent Registry remote MCP server: roles/mcp.toolUser + roles/agentregistry.viewer.

  • Agent-to-MCP egress (Policy V1): pass --resource-type=agent-registry and the server-generated McpServer UUID (registryResource basename):

    gcloud iap web set-iam-policy agents-iap-policy.json \
      --project=PROJECT_ID \
      --resource-type=agent-registry \
      --mcp-server=MCP_SERVER_UUID \
      --region=REGION

    Policy V1 CEL attributes: iap.googleapis.com/mcp.toolName, iap.googleapis.com/mcp.tool.isReadOnly, iap.googleapis.com/request.auth.type ('MCP').

Common pitfalls
  • Google/Google Cloud remote MCP server returns NOT_FOUND in a region: auto-registered Google remote MCP servers exist only in global (even when exposing regional .rep.googleapis.com/mcp URLs). Always query with --location=global and bind IAP policies with --region=global.

  • Cloud Run MCP server 403 unregistered mismatch (.a.run.app vs. .run.app): gcloud beta run deploy --functional-type=mcp-server registers https://<SERVICE>-<PROJECT_NUMBER>.<REGION>.run.app. Configure the agent to call that exact .run.app URL (not the legacy *-uc.a.run.app URL).

  • Native MCP AuthzPolicy blocks initialize / ping: when attaching a Network Security AuthzPolicy with mcp tool rules, set baseProtocolMethodsOption: MATCH_BASE_PROTOCOL_METHODS on ALLOW rules (see references/policies.md and BKI #39).

  • Tool spec rejected or stale: re-run services update --mcp-server-spec-content=toolspec.json (<= 10 KB, no @) after tool changes; never register manually in us or eu.

Sources: docs.cloud.google.com/agent-registry/register-mcp-servers, docs.cloud.google.com/agent-registry/manage-mcp-tools, docs.cloud.google.com/agent-registry/use-agentregistry-mcp, docs.cloud.google.com/gemini-enterprise-agent-platform/govern/policies/configure-iam-policies.


5. Resource Type — Endpoints

Concept

Endpoints: Target URLs, typically REST APIs, accessed by an agent. By abstracting these destinations into manageable resources, Agent Registry lets you centrally govern which external services your agents can access. -- docs.cloud.google.com/agent-registry/concepts

Endpoints are how you authorize an agent to reach anything that isn't an agent or an MCP server: Google APIs (BigQuery, Agent Runtime, etc.), third-party REST APIs, internal HTTP services. Always manually registered.

Register
gcloud agent-registry services create SERVICE_NAME \
  --project=PROJECT_ID \
  --location=REGION \
  --display-name="DISPLAY_NAME" \
  --endpoint-spec-type=no-spec \
  --interfaces=url=ENDPOINT_URL,protocolBinding=PROTOCOL
  • --endpoint-spec-type=no-spec is mandatory -- this is what causes the read-only Endpoint projection to appear.

  • After creation, Agent Registry auto-generates the read-only Endpoint resource that agents discover.

Hostname registration is exact-match. You must register every hostname variant the agent might dial. See "Hostname Matrix" in section 2. The fix for "endpoint registered but traffic denied" is almost always: register the missing variant.

List
gcloud agent-registry endpoints list \
  --project=PROJECT_ID \
  --location=REGION

Filter:

gcloud agent-registry endpoints list \
  --project=PROJECT_ID \
  --location=REGION \
  --filter="displayName='NAME'"

For URL-substring search, list services instead and filter on interfaces.url:

gcloud agent-registry services list \
  --project=PROJECT_ID \
  --location=REGION \
  --format="table(name.basename(), interfaces.url, interfaces.protocolBinding)" \
  --filter="interfaces.url:HOSTNAME_FRAGMENT"
Describe
gcloud agent-registry endpoints describe ENDPOINT_NAME \
  --project=PROJECT_ID \
  --location=REGION
Update
gcloud agent-registry services update SERVICE_NAME \
  --project=PROJECT_ID \
  --location=REGION \
  --interfaces=url=ENDPOINT_URL,protocolBinding=PROTOCOL
Delete
gcloud agent-registry services delete SERVICE_NAME \
  --project=PROJECT_ID \
  --location=REGION

"This action immediately removes the endpoint from discovery search results." Google Cloud console UI requires typing DELETE to confirm.

IAM model for endpoints
  • Discover endpoints: roles/agentregistry.viewer.

  • Register / update / delete: roles/agentregistry.editor.

  • Agent-to-endpoint traffic: bind roles/iap.egressor to the source agent's principal on the endpoint resource:

    gcloud iap web set-iam-policy agents-iap-policy.json \
      --project=PROJECT_ID \
      --resource-type=agent-registry \
      --endpoint=ENDPOINT_UUID \
      --region=REGION

    Per-endpoint policy bindings do not require a CEL condition; you can omit condition to grant unconditional egress to that one endpoint, or add a CEL condition for attribute-based scoping (e.g. by Name, ReadOnly, Destructive, Idempotent, OpenWorld properties).

Note on roles/iap.tunnelResourceAccessor

Some IAP-protected resources use roles/iap.tunnelResourceAccessor (iap.tunnelDestGroups.accessViaIAP, iap.tunnelInstances.accessViaIAP) -- that role governs classic IAP TCP tunnels to GCE instances and dest groups. Agent Gateway egress to Agent Registry resources does NOT use this role; it uses roles/iap.egressor (iap.webServiceVersions.egressViaIAP). If you see a tutorial suggesting tunnelResourceAccessor for agent egress, it's the wrong role -- confirm against docs.cloud.google.com/iam/docs/roles-permissions/iap.

Common pitfalls
  • Test connection fails in Google Cloud console for a private URL: expected. The Google Cloud console connection test only validates public URLs; it does not support private endpoints (Agent Runtime, internal HTTPS, etc.).

  • Agent gets denied on a Google API call: the API has a hostname variant you didn't register (commonly the REP regional form ${id}.${LOCATION}.rep.googleapis.com).

  • Endpoint registered in global but agent calls a regional URL: registry location doesn't have to match the destination URL location, but the IAM binding's --region must match where the resource is registered. Check that --region= on gcloud iap web set-iam-policy equals the location used at creation.

  • Endpoint not appearing in endpoints list: created with the wrong spec type. Must use --endpoint-spec-type=no-spec.

Sources: docs.cloud.google.com/agent-registry/register-endpoints, docs.cloud.google.com/agent-registry/manage-endpoints, docs.cloud.google.com/iam/docs/roles-permissions/iap, docs.cloud.google.com/gemini-enterprise-agent-platform/govern/policies/configure-iam-policies.


6. IAM Scope: Registry-Wide vs Per-Resource

There are two layers. Don't conflate them.

Layer A — Agent Registry permissions (manage the catalog)

Project-scoped, granted via standard IAM:

RoleUse
roles/agentregistry.viewerList / describe / search agents, MCP,
: : endpoints :
roles/agentregistry.editorCreate / update / delete Service resources
roles/mcp.toolUserCall tools via the Agent Registry remote MCP
: : server (mcp.tools.call) :

These let humans and CI register and inspect entries. They do not grant runtime traffic.

Layer B — Traffic Authorization (Legacy v1 vs. UAP Policy V2)

Depending on the gateway's iapPolicyVersion setting, registry resources authorize traffic through one of two mechanisms:

Mechanism B1: Unified Access Policy (UAP / Policy V2)

When the attached IAP Authz Extension specifies iapPolicyVersion: "V2", policies are decoupled from individual registry resources:

  • AccessPolicy and PolicyBinding (iam.googleapis.com/v3) attach to the Resource Manager hierarchy (Organization, Folder, Project) targeting numeric resource URIs (e.g. //cloudresourcemanager.googleapis.com/projects/${PROJECT_NUMBER}).

  • Rules grant FQDN permission iap.googleapis.com/resources.egressViaIAP.

  • Agent Registry entries dynamically populate the destination.agent_registry.* CEL attributes evaluated at runtime:

    • destination.is_registered (boolean): true if target matches a registry service.

    • destination.agent_registry.resource_type: 'ENDPOINT', 'MCP_SERVER', 'AGENT', 'SKILL'.

    • destination.agent_registry.location: region of registry resource.

    • destination.agent_registry.mcp_server.name: full MCP server resource name.

    • destination.agent_registry.mcp_server.method: 'tools', 'prompts', 'resources'.

    • destination.agent_registry.mcp_server.tool.name: specific tool name (e.g. 'updateTicket').

    • destination.agent_registry.mcp_server.tool.annotations.read_only_hint: boolean.

  • Mandatory CEL Guard: Always guard tool annotations with: (destination.agent_registry.mcp_server.method == 'tools') && (destination.agent_registry.mcp_server.tool.annotations.read_only_hint == true).

Mechanism B2: Legacy IAM v1 (Per-Resource Bindings)

In legacy v1, permissions are bound directly on Agent Registry resources via gcloud iap web set-iam-policy. The role is always roles/iap.egressor. The scope of the binding determines which target resources the source agent can reach:

ScopeRequired gcloud iap web Target FlagsGrants Egress To
Registry-wide--resource-type=agent-registryAll agents, MCP servers, and endpoints in that project/region registry
Single agent--resource-type=agent-registry --agent=AGENT_UUIDOne target agent (registryResource basename)
Single MCP server--resource-type=agent-registry --mcp-server=MCP_SERVER_UUIDOne target MCP server (registryResource basename)
Single endpoint--resource-type=agent-registry --endpoint=ENDPOINT_UUIDOne target endpoint (registryResource basename)

All four require --project=PROJECT_ID and --region=LOCATION (global or region). Never omit --resource-type=agent-registry: when --region is set without --resource-type, gcloud iap web targets IAPWebLocation (projects/PROJECT_NUMBER/locations/LOCATION/iap_web) and silently ignores --agent, --mcp-server, and --endpoint (mutating the regional iap_web container policy or failing on --region=global); when both --resource-type and --region are omitted, it targets project-wide IAPWeb (projects/PROJECT_NUMBER/iap_web).

Warning: Top-Level {"bindings": [...]} Required for gcloud iap web set-iam-policy: gcloud iap web set-iam-policy parses the file directly as an IAM Policy object (bindings, etag, version) and wraps SetIamPolicyRequest internally. Do NOT wrap "bindings" inside "policy": { ... } when using gcloud — apitools discards the outer "policy" key and submits an empty policy that silently wipes all existing IAP IAM bindings! (Only raw REST POST ...:setIamPolicy requests use the {"policy": {"bindings": [...]}} envelope.)

The Policy V1 JSON file for gcloud iap web set-iam-policy uses top-level "bindings":

{
  "bindings": [
    {
      "role": "roles/iap.egressor",
      "members": ["AGENT_PRINCIPAL"]
    }
  ],
  "version": 3
}
Agent principal formats (members / principals)
  • Agent Runtime / Gemini Enterprise (org-backed — ENGINE_ID is numeric): principal://agents.global.org-ORGANIZATION_ID.system.id.goog/resources/aiplatform/projects/PROJECT_NUMBER/locations/REGION/reasoningEngines/ENGINE_ID

  • Agent Runtime (orgless project — note proj-, NOT project-): principal://agents.global.proj-PROJECT_NUMBER.system.id.goog/resources/aiplatform/projects/PROJECT_NUMBER/locations/REGION/reasoningEngines/ENGINE_ID

  • Workload Identity Pool (DIY / external): principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/subject/SUBJECT

  • Project-wide Agent PrincipalSet: principalSet://agents.global.org-ORGANIZATION_ID.system.id.goog/attribute.platformContainer/aiplatform/projects/PROJECT_NUMBER

When debugging "wrong principal" errors: the principal is built from the agent's resource URI, not its display name and not its URN. View the resource URI via gcloud agent-registry agents describe.

Full IAP role reference (relevant subset)
RolePermission grants the role providesWhen to use
roles/iap.egressor (Beta)iap.webServiceVersions.egressViaIAP**Agent
: : : egress :
: : : through Agent :
: : : Gateway** :
roles/iap.httpsResourceAccessoriap.webServiceVersions.accessViaIAPHuman/service
: : : ingress to an :
: : : IAP-protected :
: : : web app :
roles/iap.tunnelResourceAccessoriap.tunnelDestGroups.accessViaIAP,Classic IAP
: : iap.tunnelInstances.accessViaIAP : TCP tunnels :
: : : to GCE -- :
: : : not for :
: : : agent gateway :
roles/iap.adminFull IAP admin (iap.tunnel.*, allManage IAP
: : getIam/setIam) : policies :
roles/iap.viewerRead IAP settingsAudit

Source: docs.cloud.google.com/iam/docs/roles-permissions/iap.


7. Common Registration Errors and Where to Look

SymptomLikely causeWhere to check
services create fails: "manualUsed --location=us orSwitch to a region (e.g.
: registration not supported" : --location=eu (multi-regions) : us-central1) or global :
services create succeeds butWrong *-spec-type.Re-create with the correct
: resource missing from endpoints : --endpoint-spec-type=no-spec : spec-type, or checkservices list` :
: list : produces anEndpoint`. : :
`services updateSpec > 10 KBTrim the tool spec
: --mcp-server-spec-content` rejects : : :
: file : : :
Agent calls API X, IAP returns 403Hostname variant not registered`services list
: : (commonly REP : --filter="interfaces.url:FRAGMENT"` :
: : *.LOCATION.rep.googleapis.com) : :
Agent calls API X, IAP returns 403 even though hostname is registeredMissing roles/iap.egressor binding for the source agent's principalgcloud iap web get-iam-policy --project=P --region=L --resource-type=agent-registry [--endpoint=UUID | --mcp-server=UUID | --agent=UUID]
MCP tool call succeeds for one tool, denied for anotherIAP policy condition restricts mcp.toolName or mcp.tool.isReadOnlyInspect the IAP policy on the MCP server resource
Auto-registered Agent Runtime agent missing from registryAgent deployed via custom code, not Agent Runtime SDKRe-deploy via Agent Runtime, or register manually as a Service
A2A agent has no skills in registryagent-card.json unreachable from Google's crawler or > 10 KBCurl the Agent Card URL from a public network; verify content type, schema, and <= 10 KB size
Agent principal mismatch in IAP policyPrincipal built from wrong resource URI / project number / project- typo instead of proj-gcloud agent-registry agents describe → use Resource URI and numeric PROJECT_NUMBER
roles/iap.tunnelResourceAccessor granted, still deniedWrong role — agent egress requires roles/iap.egressorReplace the binding with roles/iap.egressor
Endpoint Google Cloud console "Test connection" fails on private URLConnection test only works for public URLs (by design)Test via the agent itself or curl from a peered network
agents-iap-policy.json set, still deniedWrong --region flag on set-iam-policy, omitted --resource-type=agent-registry (wrote to IAPWebLocation iap_web), or wrapped JSON in {"policy": ...} (wiped bindings)Pass --resource-type=agent-registry, match --region (global for Google remote MCPs), and use top-level {"bindings": [...]}
Cannot find URN or child UUIDRun services describe for registryResource UUID; mcp-servers/agents describe for URNSee Section 1 URN & UUID rules
agent-gateways import: Config validation failed (missing permissions: agentregistry.agents.list)agentregistry.googleapis.com API disabled on registry project, or caller lacks roles/agentregistry.viewerEnable agentregistry.googleapis.com and grant roles/agentregistry.viewer (agentregistry.{agents,endpoints,mcpServers}.list)
agent-gateways import 400 / InvalidArgument on registriesTwo regional registries (us-central1 + europe-west1), > 2 registries, or cross-jurisdiction (us-central1 + eu)Configure max 2 registries: global + 1 same-region or same-jurisdiction multi-region registry
NXDOMAIN on aiplatform.{us,eu}.rep.mtls.googleapis.com.rep.mtls DNS does not exist for us/eu multi-regionsUpgrade google-genai, use global/regional endpoint, or set GOOGLE_API_USE_MTLS_ENDPOINT=never
gcloud iap web returns NOT_FOUND or ignores --endpoint/--mcp-server/--agentOmitted --resource-type=agent-registry (targets IAPWebLocation iap_web), passed SERVICE_NAME instead of UUID, or queried remote MCP in region instead of globalAlways pass --resource-type=agent-registry, use registryResource basename, and use --region=global for Google remote MCPs
Console loading error or gcloud error 13 with Workforce Identity Federation (WIF)Preview WIF limitation on Agent Registry & Agent Gateway control planesAuthenticate with a Cloud Identity or Google Workspace account (see BKI #36)

8. Discovery via the Agent Registry MCP Server

The registry itself is exposed as a remote MCP server at agentregistry.googleapis.com. Useful when debugging from inside an agent or via mcp inspector. Tools include:

Discovery: - search_agents, search_mcp_servers -- keyword/prefix search - get_agent, get_mcp_server, get_endpoint -- by resource name - get_service -- read the underlying writable spec - list_agents, list_mcp_servers, list_endpoints, list_services - list_bindings, get_binding, fetch_available_bindings - get_operation

Administrative: - create_service, update_service, delete_service - create_binding, update_binding, delete_binding

tools/list over MCP does not require auth:

POST /mcp HTTP/1.1
Host: agentregistry.googleapis.com
Content-Type: application/json

{ "jsonrpc": "2.0", "method": "tools/list" }

Source: docs.cloud.google.com/agent-registry/use-agentregistry-mcp.


9. Quick Reference — Every gcloud Subcommand

# READ
gcloud agent-registry agents      list   --project=P --location=L
gcloud agent-registry agents      describe NAME --project=P --location=L
gcloud agent-registry mcp-servers list   --project=P --location=L
gcloud agent-registry mcp-servers describe NAME --project=P --location=L
gcloud agent-registry endpoints   list   --project=P --location=L
gcloud agent-registry endpoints   describe NAME --project=P --location=L
gcloud agent-registry services    list   --project=P --location=L
gcloud agent-registry services    describe NAME --project=P --location=L
gcloud agent-registry skills      list   --project=P --location=L
gcloud agent-registry skills      describe NAME --project=P --location=L

# WRITE (always 'services'; 'endpoints' has no 'create' subcommand and there is no 'gcloud beta agent-registry')
gcloud agent-registry services create NAME \
    --project=P --location=L --display-name="..." \
    --endpoint-spec-type=no-spec \
    --interfaces=url=URL,protocolBinding=HTTP_JSON|GRPC|JSONRPC

gcloud agent-registry services create NAME \
    --project=P --location=L --display-name="..." \
    --mcp-server-spec-type=tool-spec \
    --mcp-server-spec-content=toolspec.json \
    --interfaces=url=URL,protocolBinding=JSONRPC

# A2A agent: omit --interfaces (read from agent-card.json)
gcloud agent-registry services create NAME \
    --project=P --location=L --display-name="..." \
    --agent-spec-type=a2a-agent-card \
    --agent-spec-content=agent-card.json

# Non-A2A REST agent: --agent-spec-type=no-spec + --interfaces
gcloud agent-registry services create NAME \
    --project=P --location=L --display-name="..." \
    --agent-spec-type=no-spec \
    --interfaces=url=URL,protocolBinding=HTTP_JSON

gcloud agent-registry services update NAME \
    --project=P --location=L \
    [ --display-name="..." | --description="..." \
      | --interfaces=url=URL,protocolBinding=... | --clear-interfaces \
      | --mcp-server-spec-content=toolspec.json \
      | --agent-spec-content=agent-card.json ]

gcloud agent-registry services delete NAME --project=P --location=L

# IAP Policy V1 egress bindings (--resource-type=agent-registry is ALWAYS required)
gcloud iap web set-iam-policy POLICY.json \
    --project=P --region=L --resource-type=agent-registry \
    [ --agent=AGENT_UUID | --mcp-server=MCP_SERVER_UUID | --endpoint=ENDPOINT_UUID ]

gcloud iap web get-iam-policy \
    --project=P --region=L --resource-type=agent-registry \
    [ --agent=AGENT_UUID | --mcp-server=MCP_SERVER_UUID | --endpoint=ENDPOINT_UUID ]

Spec size cap: 10 KB for --mcp-server-spec-content and --agent-spec-content (pass plain file path without @).

Manual registration locations: any region or global. Not us or eu.


Source Index

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an MCP server

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
Google
Publisher
Google (Official)
Source
google/skills · skills/cloud/agent-platform-troubleshooting · commit 2194d3b
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Google or Google.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download agent-platform-troubleshooting.zip