Agent Platform Troubleshooting
GoogleOfficialPublished by Google
- Skill
- Review and audit
- Software development
- Apache-2.0
- No issues found
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
This skill provides a structured diagnostic framework for identifying and resolving complex issues within the Google Cloud Gemini Enterprise Agent Platform. It covers the entire stack, including Agent Gateway, Agent Connectivity Templates (ACT), Agent Registry, Identity-Aware Proxy (IAP), Model Armor, Semantic Governance Policies (SGP), and network-level configurations like DNS peering and proxy architectures. The skill enforces a strict, rule-based diagnostic flow to ensure efficient troubleshooting while preventing common misconfigurations, such as incorrect IAM assignments or unsupported multi-region registrations.
What it does
- Analyzes Agent Platform logs to identify root causes for 401, 403, 404, 498, 500, and 503 errors.
- Validates Agent Gateway, ACT, and registry configurations against platform best practices.
- Provides specific remediation steps for IAP, Model Armor, and SGP policy enforcement issues.
- Troubleshoots complex network egress scenarios including SWP, explicit proxies, and VPC-SC perimeters.
- Generates a standardized diagnostic report with findings and actionable recommendations.
- Enforces strict operational safety rules to prevent unauthorized mutations or excessive resource consumption.
When to use it
- When agent requests fail with HTTP status codes like 403 (Egress not authorized) or 404 (HTTP CONNECT failure).
- When Agent Runtime instances crash on startup due to connectivity or policy validation errors.
- When troubleshooting configuration mismatches in Agent Gateway, ACT, or registry services.
- When diagnosing VPC-SC perimeter blocks or issues with DNS peering and Private CA TrustConfig.
- When investigating Model Armor or SGP verdict rationales for blocked requests.
Advantages
- Reduces mean time to resolution by following a proven, rule-based diagnostic hierarchy.
- Prevents common configuration pitfalls through mandatory pre-flight checks and strict usage rules.
- Provides clear, actionable remediation steps tailored to the specific failure mode.
- Ensures consistent diagnostic output across different project environments.
- Maintains operational security by requiring the user to own all mutations.
Usage examples
- “My agent is getting a 403 Egress request is not authorized error when calling an external API.”
- The agent will initiate the diagnostic flow, checking IAP egressor bindings on the registry entry and verifying the relevant AuthzPolicy.
- “Why is my Agent Gateway stuck in PROVISIONING state?”
- The agent will check for PSC subnet IP exhaustion and provide guidance on expanding the CIDR range if necessary.
- “My Agent Runtime crashes on startup with connection errors to telemetry.mtls.googleapis.com.”
- The agent will verify if the required telemetry endpoints are registered and suggest updating the AuthorizationPolicy to allow access.
Install
Download the skill as a ZIP:
https://autogineer.com/agent-skills/google/agent-platform-troubleshooting/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip agent-platform-troubleshooting.zip -d .agents/skills
Documentation by Google
Agent Platform Troubleshooting
Important: You MUST ONLY use the reference files located in this skill's
references/directory (references/field-manual.md,references/known-issues.md,references/agent-gateway.md,references/policies.md,references/agent-registry.md,references/agent-identity.md). Do NOT search for or read external playbooks outside this directory.
Diagnose issues across the Google Cloud Gemini Enterprise Agent Platform: Agent Gateway, Agent Connectivity Templates (ACT), Agent Registry (Services, Agents, MCP Servers, Endpoints), Agent Identity, IAP/UAP policies, Model Armor, Semantic Governance Policies (SGP), explicit/transparent proxies, and tracing. This skill produces a diagnostic report — findings and fix recommendations. It does not apply fixes; the user owns all mutations.
Usage Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read and follow Usage Attribution. Prefix
direct terminal gcloud calls (never generated scripts, Terraform, or user
code) with
CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:agent-platform-troubleshooting)"
and append agent-skills/1.0 (skill:agent-platform-troubleshooting) to direct
HTTP/REST/MCP User-Agent headers.
MANDATORY PRE-FLIGHT CHECKLIST (CHECK BEFORE RESPONDING OR CALLING TOOLS)
CRITICAL: Before generating ANY response or calling any tools, evaluate the user's prompt against Rules 1–14. When a Turn-0 rule matches, execute its directive immediately and STOP.
Rule 1: Out-of-Scope GCP IAM / GCS Queries
If the prompt asks about Compute Engine (GCE), Cloud Storage (GCS) buckets, or generic GCP IAM permissions unrelated to the Agent Platform stack:
IMMEDIATELY DECLINE ON TURN 0. DO NOT CALL ANY TOOLS OR GIVE IAM GUIDES.
Respond: "I decline to troubleshoot generic GCP IAM or GCS access issues, as they are out of scope for the Agent Platform Troubleshooting skill."
Rule 2: Strict Prohibition on Custom Discovery Scripts
If asked to write, generate, compile, or execute a custom Python or bash script to discover resources (e.g., active Agent Runtime instances):
DO NOT CALL ANY TOOLS OR WRITE/RUN ANY SCRIPTS.
RESPOND ON TURN 0: "I cannot write or execute custom Python or bash scripts for resource discovery. Custom discovery scripts are prohibited as they consume excessive turns and cause timeouts. Instead, use standard gcloud CLI commands (see Google Cloud SDK Installation) or curl REST API calls with application default credentials:
gcloud agent-registry agents list --location=us-central1orcurl -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" "https://us-central1-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/us-central1/reasoningEngines""
Rule 3: Consolidated Registry for Google APIs / Design Queries
If asked about registering multiple Agent Runtime, Cloud Resource Manager, or Google API interfaces, or structuring services in Agent Registry:
DO NOT CALL ANY TOOLS. RESPOND ON TURN 0 WITH:
Consolidate ALL Google APIs under a single
googleapisservice entry in Agent Registry viagcloud agent-registry services create googleapis --endpoint-spec-type=no-spec.Explicitly state: "Do NOT register each Google API as a separate registry service entry, as separate entries cause resource clutter, complicate IAM policy management, and risk hitting registry quota limits."
List all 8 required base FQDN interfaces (plus global
https://aiplatform.googleapis.comwhen calling global Gemini models):https://agentregistry.googleapis.comhttps://aiplatform.mtls.googleapis.comhttps://cloudresourcemanager.mtls.googleapis.comhttps://iamcredentials.mtls.googleapis.comhttps://telemetry.mtls.googleapis.comhttps://{region}-aiplatform.mtls.googleapis.comhttps://{region}-aiplatform.googleapis.comhttps://aiplatform.{region}.rep.googleapis.com
Refer to
references/agent-registry.md§2 for the full command.
Rule 4: Cloud Run / Cloud Functions Egress 401/403 (OIDC vs IAP MCP)
If an agent gets a 401 or 403 egress error calling an MCP server or tool on
Cloud Run or Cloud Functions:
DO NOT RUN LOG TOOLS. RESPOND ON TURN 0 WITH:
Explain that direct Agent Identity (
principalSet://.../principal://...) to standard Cloud Run OIDC authentication is not natively supported.Recommend Service Account impersonation in agent code to obtain an OIDC ID token, granting
roles/iam.serviceAccountTokenCreatorto the Agent Identity on the target SA (androles/run.invokerto that SA on Cloud Run).Note that for Preview IAP-enabled Cloud Run MCP servers (
--iap --functional-type=mcp-server,run.appshort URL), authorize via IAP/UAP instead ofroles/run.invoker(seereferences/known-issues.mdBKI 21 andreferences/agent-identity.md).
Rule 5: Telemetry & Monitoring Endpoint Blocks on Startup
If Agent Runtime startup crashes (set_up()) with connection resets or
handshake errors reaching telemetry.mtls.googleapis.com:
Evidence: Explicitly check whether all 4 monitoring/tracing endpoints are registered as Endpoints in Agent Registry (
gcloud agent-registry endpoints list):telemetry.mtls.googleapis.com,monitoring.googleapis.com,trace.mtls.googleapis.com, andcloudtrace.googleapis.com.Recommended Fix:
Register missing telemetry/tracing interfaces via
gcloud agent-registry services create --endpoint-spec-type=no-spec(which projects read-only Endpoints in Agent Registry fortelemetry.mtls.googleapis.com,monitoring.googleapis.com,trace.mtls.googleapis.com, andcloudtrace.googleapis.com).Create or update an
AuthorizationPolicy(AuthzPolicy/ IAP / UAP) bound to the Gateway that allows the agent's principal set (identity) to access the telemetry endpoints (seereferences/known-issues.mdBKI 23).
Rule 6: IAP Denial Troubleshooting (403 / Egress request is not authorized)
When logs or symptoms show 403 / Egress request is not authorized via IAP:
Check IAP Egressor bindings on the registry entry FIRST (
roles/iap.egressorviagcloud iap web get-iam-policy --resource-type=agent-registrybound to the server-generated child UUID fromregistryResource, notSERVICE_NAME).Create a registry entry (
gcloud agent-registry services create) if none exists for the exact target hostname.Grant
roles/iap.egressoron the registry entry (IAM v1 —set-iam-policyrequires top-level{"bindings": [...]}JSON without a"policy"wrapper) or verify UAPAccessPolicy+ CRMPolicyBinding(iapPolicyVersion: "V2",iap.googleapis.com/resources.egressViaIAP; seereferences/policies.md§2).Check IAP audit logs (
protoPayload.serviceName="iap.googleapis.com") and verify anAuthorizationPolicy(AuthzPolicy) is bound to the Gateway.Explicitly warn: "Do NOT use
roles/iap.tunnelResourceAccessor(which is for classic IAP TCP tunneling to GCE VMs, not Agent Gateway egress) and do NOT bypass IAP authentication."
Rule 7: PSC Subnet Exhaustion Speed Rule (PROVISIONING Hang)
When diagnosing gateway provisioning stuck or failing on subnet IP exhaustion:
DO NOT loop across regions. Run ONLY in the target region (default
us-central1):gcloud network-services agent-gateways list --location=us-central1gcloud network-services agent-gateways describe <GW> --location=us-central1gcloud compute network-attachments describe <NA> --region=us-central1gcloud compute networks subnets describe <SUBNET> --region=us-central1
Check free IP count (
Usable - Allocated = Free) before recommending any Network Attachment deletion; flag that a/28subnet is easily exhausted and recommend expanding the CIDR range to at least/26(known-issues.mdBKI 19).
Rule 8: Multi-Region Manual Registration Prohibition (us / eu)
If asked about manually registering endpoints or services in us or eu:
DO NOT CALL ANY TOOLS. RESPOND ON TURN 0:
"Manual endpoint/service registration is NOT supported in
usoreumulti-region locations (usandeuare read-only roll-up locations for auto-discovered resources)." (Explicitly name BOTHusandeu.)"Register manually in a specific region (e.g.,
us-central1) orglobal."
Rule 9: VPC Service Controls (VPC-SC) Perimeter & ACT ALL_TRAFFIC
When diagnosing VPC-SC perimeter blocks or external API hangs under VPC-SC:
Identify a VPC Service Controls perimeter block / boundary enforcement.
Check that the Agent Connectivity Template (ACT) sets
vpcEgress: ALL_TRAFFIC(native VPC-SC creation works out of the box since Sept 8, 2026 whenvpcEgress: ALL_TRAFFICis set) and that Cloud NAT is configured on the consumer VPC's PSC-I network attachment subnet.For legacy or strict custom perimeters blocking Google actuation identities, recommend creating ingress policies allowing:
Explicitly state: "Do NOT disable VPC Service Controls or delete perimeters." Refer to
references/known-issues.mdBKI 18 & 28.
Rule 10: UAP Policy Binding Org Policy Blocker & Blanket DENY Crash
CUSTOM_ORG_POLICY_VIOLATIONongcloud iam policy-bindings create: Caused byconstraints/iam.managed.disableAccessPolicyBinding(note: some docs misspell this plural). Apply an Org Policy override (enforce: false) at the target resource and wait 30 to 60 seconds for IAM propagation (BKI 24).Startup crash after UAP
destination.is_registered == falseDENY: Explain that the blanketDENYblocked internal/unauthenticated SDK bootstrap and metadata calls (such as Cloud Resource Manager project lookups) during Reasoning Engine init; remove or scope the blanketDENYrule (BKI 5).
Rule 11: Agent Gateway Dual-Registry & Jurisdiction Invariants
When configuring or validating registries on an Agent Gateway:
An Agent Gateway supports a maximum of two registries.
When two registries are configured, exactly one must be
globaland the other must beregionalormulti-regional.Binding two regional registries (e.g.,
us-central1+europe-west1), two multi-regionals, two globals, or a mismatched jurisdiction (e.g.,us-central1gateway +euregistry) is invalid and rejected with an HTTP 400 validation error (references/agent-gateway.md§4, BKI 26).
Rule 12: Cross-Project Runtime-to-Gateway Binding Diagnosis
When Reasoning Engine in Project 1 fails to bind to a Gateway in Project 2:
Deploying Identity: Needs
networkservices.agentGateways.useandnetworkservices.agentGateways.get(orroles/networkservices.viewer) in Project 2.Vertex AI Service Agent: Grant
roles/networkservices.vieweror custom roleae_agw_cross_project_sa(aliasar_agw_cross_project_sa) toservice-PROJECT_1_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount.comin Project 2.Regional Colocation: Verify both runtime and gateway are in the same region.
CAA Opt-Out: Set
GOOGLE_API_PREVENT_AGENT_TOKEN_SHARING_FOR_GCP_SERVICEStoFalse(references/known-issues.mdBKI 10,references/agent-gateway.md§11).
Rule 13: Downstream SWP + PBR vs. Explicit Forward HTTPS Proxy
Distinguish the two supported consumer-VPC proxy architectures:
Pattern A — Transparent Next-Hop SWP + PBR (no client
HTTPS_PROXYneeded):Explain that PSC endpoints cannot be route next-hops, SWP behind PSC Service Attachments only supports explicit proxy mode (not transparent next-hop mode), and static routes cannot be used because Network Attachments cannot have VM network tags.
Deploy SWP directly in the consumer VPC with
type: SECURE_WEB_GATEWAYandroutingMode: NEXT_HOP_ROUTING_MODEon an internal IP on the PSC-I subnet (e.g.,10.20.1.250), with a/26REGIONAL_MANAGED_PROXYACTIVEsubnet.Configure Policy-Based Routing (PBR) matching source CIDR
10.20.1.0/24to--next-hop-ilb-ip=10.20.1.250(priority 200) + fallback PBR (1000).Create Cloud NAT with
ENDPOINT_TYPE_SWG(nats create ... --endpoint-types=ENDPOINT_TYPE_VM,ENDPOINT_TYPE_SWG), and allowlist<region>-aiplatform.mtls.googleapis.comin SWP CEL (BKI 33).
Pattern B — Explicit Forward HTTPS Proxy (
HTTPS_PROXY+ L4AuthzPolicy):Without L4 bypass,
HTTPS_PROXY=https://proxy.corp.internal:443fails onHTTP CONNECTwith HTTP404because Agent Gateway's L7 HTTP manager terminates TLS and rejects nestedCONNECT.Import a
v1beta1L4AuthzPolicyviagcloud beta network-security authz-policies importwithaction: ALLOWandnetworkRulesmatchingto.operations[].snis(omittarget.loadBalancingSchemeandports; max 5 rules; omitfromor include7.0.0.0/8+ PSC-I CIDR infrom.sources[].ipBlocks).Listen with TLS on
:443, allow the consumer PSC-I subnet CIDR on VPC firewall/proxy ACLs (never7.0.0.0/8), setNO_PROXY/no_proxyto.googleapis.com,googleapis.com,metadata.google.internal,169.254.169.254,localhost,127.0.0.1, bundleagentGatewayCard.rootCertificates+ Private CA in container env vars (SSL_CERT_FILE,REQUESTS_CA_BUNDLE), and warn L4ALLOWbypasses L7 IAP/UAP, Model Armor, SGP, andhttpRulesfor matched SNIs (BKI 34).
Rule 14: DNS Peering, TrustConfig, Model Armor, SGP, MCP, Identity, & Lifecycle
DNS Peering & Service Agent: Public FQDNs resolve via public recursive DNS. Private DNS requires
dnsPeeringConfig(domainsarray with trailing.; ACT uses stringtargetNetworkonly, whereas inlineAgentGateway.networkConfig.dnsPeeringConfigalso requirestargetProject). Cloud DNS caps total peering hops at 2 (Gateway consumes Hop 1 → max 1 consumer peering hop; a 2nd hop fails preflight or returnsSERVFAIL→503, BKI 35). Grantroles/dns.peer,roles/dns.reader,roles/compute.networkViewer, androles/compute.networkUsertoservice-GATEWAY_PROJECT_NUMBER@gcp-sa-agentgateway.iam.gserviceaccount.com(andcompute.networkAttachments.useto caller, BKI 9). Never use authoritative"."private root zones ("."requires a Cloud DNS Private Forwarding Zone).Private CA
TrustConfig& Immutable ACT: SetegressNetworkConfig.tlsConfig.trustConfigwithadditionalRoots: PUBLICLY_TRUSTED_ROOTS(omitting breaks*.googleapis.com). ACT fields are immutable — create a new ACT and rebind viaPATCHwithupdateMask=agentConnectivityTemplate(BKI 6).Model Armor (
CONTENT_AUTHZ): Ingress (CLIENT_TO_AGENT, max 1, ADKstreamQueryonly) usesservice-AGENT_PROJECT_NUMBER@gcp-sa-aiplatform-re.iam.gserviceaccount.com; Egress (AGENT_TO_ANYWHERE, max 4) usesservice-GATEWAY_PROJECT_NUMBER@gcp-sa-dep.iam.gserviceaccount.com. ScopehttpRuleswhento exclude gRPC and streams >60 seconds (Stream removed, BKI 13).Semantic Governance Policies (SGP): Uses
gcp-sa-aiplatform, non-streaming LLM calls only,.contains()+!request.headers['content-type'].startsWith('application/grpc')inwhen, and full-map updates onPATCH ?updateMask=metadatasosgpEnforcementMode: "DRY_RUN"andsgpUserIntentMatchingare preserved (BKI 38).Native MCP, Identity, Trace, & Teardown: Set
baseProtocolMethodsOption: MATCH_BASE_PROTOCOL_METHODSon MCPALLOWrules (BKI 39). Orgless trust domains useagents.global.proj-PROJECT_NUMBER.system.id.goog(proj-, neverproject-); never grantroles/aiplatform.expressUseror project-wideroles/aiplatform.userto Agent Identities. For Cloud Trace (TelemetryPolicy), always recommend enablingcloudtrace.googleapis.com(gcloud services enable cloudtrace.googleapis.com) and grantingroles/cloudtrace.agenttocompute-system,gcp-sa-networksecurity, andgcp-sa-agentgateway(seereferences/agent-gateway.md§8.3 andreferences/field-manual.mdStep 14.4). Tear down in strict 7-step order (workloads → UAP bindings →AuthzPolicy→ extensions →AgentGateway→ ACT → VPC) and use unique gateway IDs if deletion tombstones collide (BKIs 25, 30).
Quick Symptom Triage Matrix
| Symptom / Error | Primary Suspect | Reference |
|---|---|---|
403 / Egress request is not authorized | Unregistered FQDN, missing roles/iap.egressor / UAP binding, or PAB | Rules 4, 6; references/policies.md §2; BKIs 4, 7, 8, 15 |
HTTP 404 on HTTP CONNECT (HTTPS_PROXY) | Missing gcloud beta L4 AuthzPolicy (networkRules snis) | Rule 13B; references/agent-gateway.md §6.3; BKI 34 |
HTTP 498 immediately after binding runtime | Enforcing gateway bound before allowlisting essential platform APIs | references/agent-gateway.md §2; BKI 37 |
500 / PERMISSION_DENIED / Stream removed | Model Armor SA roles (gcp-sa-dep vs gcp-sa-aiplatform-re) or 60s stream cap | Rule 14; references/policies.md §12; BKI 13 |
503 / NXDOMAIN / SERVFAIL on private host | >2-hop DNS peering, authoritative "." zone, Packet Intercept, or SWP | Rules 13A, 14; references/agent-gateway.md §3, §6; BKIs 33, 35, 36 |
Config validation failed / PERMISSION_DENIED on import | Disabled agentregistry.googleapis.com, missing caller/gcp-sa-agentgateway IAM, or bad NA URI | Rule 14; BKIs 9, 20, 26 |
HTTP 409 ABORTED / 400 already being used | Active control-plane LRO mutex (~2 min), tombstone collision, or reverse teardown | references/agent-gateway.md §9, §10; BKIs 25, 29, 30 |
Startup crash (reasoning_engine_stderr) | Blocked telemetry/CRM bootstrap, UAP blanket DENY, or SGP gRPC interception | Rules 5, 10, 14; BKIs 1, 5, 17, 23, 38 |
When to use this skill
Trigger on Agent Gateway, Agent Connectivity Templates (ACT), Agent Registry,
Agent Identity, IAP/UAP policies, Model Armor, Semantic Governance Policies
(SGP), SWP/Explicit Proxy egress, Private CA TrustConfig, Cloud DNS peering,
Cloud Trace TelemetryPolicy, or Gemini Enterprise / Agent Runtime / GKE /
Cloud Run gateway bindings. Do not use for generic Google Cloud IAM, GCS,
GCE, or networking issues unrelated to the Agent Platform stack.
Required context & Hypothesis Rules
Gather PROJECT_ID, PROJECT_NUMBER, LOCATION (default us-central1),
AGENT_ID, AGENT_GATEWAY_NAME, Agent Identity (principal:// or
principalSet://), exact error + timestamp, and target destination. If missing,
ask or state unknowns using placeholders — never scan across projects or loop
over regions. Before Step 1 queries, formulate at most 3 hypotheses anchored
to "Why did it start failing now?" (e.g., recent Terraform/config changes).
Diagnostic flow
Follow in order (copy-paste commands in references/field-manual.md):
Step 0 — Pre-Flight & Access: Evaluate Rules 1–14. Verify project access (
gcloud projects describe $PROJECT_ID).Step 1 — Agent & Startup Logs: Classify error (
403,404 CONNECT,498,500,503, crash, timeout). Check PSC subnet IPs, ACTALL_TRAFFIC+ Cloud NAT, andagentGatewayCard.rootCertificatesCA trust.Step 2 — Gateway Logs: Query
networkservices.googleapis.com/Gateway(filtering-httpRequest.requestMethod="CONNECT"when inspecting L7 requests) to find the exact failing SNI/hostname andserviceExtensionsInfo.Step 3 — IAP / Model Armor / SGP Logs: Check
iapPolicyVersion(V1vsV2),DRY_RUNvs enforced, Model Armor callout status, orlogs/semantic-governance-policyverdicts and rationales.Step 4 — Registry State: Verify exact hostname permutations in
gcloud agent-registry {services,endpoints,mcp-servers,agents} listand confirm dual-registry + jurisdiction rules.Step 5 — Identity & Policies: Check
roles/iap.egressorvia--resource-type=agent-registryon the child UUID (V1), UAPAccessPolicy/PolicyBinding+constraints/iam.managed.disableAccessPolicyBinding(V2), PAB overrides, and least-privilege Agent Identity roles (proj-for orgless).Step 6 — AuthzPolicy & Extensions: Verify L4
networkRulesvs L7httpRules, MCPMATCH_BASE_PROTOCOL_METHODS, Model Armor / SGPwhenexclusions,ExtensionBindingscope, and policy caps (max 4 egress, max 1 ingressCONTENT_AUTHZ).Step 7 — Connectivity, DNS, & TLS: Verify ACT
TrustConfig(PUBLICLY_TRUSTED_ROOTS),gcp-sa-agentgatewayIAM, <=2 DNS peering hops, no Packet Intercept on the consumer VPC, and SWP/Explicit Proxy settings.Step 8 — PrincipalSet & Trace Verification: Test 1:1
principal://binding ifprincipalSet://lags; for Cloud Trace, enablecloudtrace.googleapis.com, verifyv1beta1TelemetryPolicy(telemetryTargetreferencingagentGateways/*and objectparentBasedSampling: {enabled: true, samplingRate: 1.0}, never stringENABLED), and grantroles/cloudtrace.agenttoservice-<PROJECT_NUMBER>@compute-system.iam.gserviceaccount.com,service-<PROJECT_NUMBER>@gcp-sa-networksecurity.iam.gserviceaccount.com, andservice-<PROJECT_NUMBER>@gcp-sa-agentgateway.iam.gserviceaccount.com.
Tools & Layered References
Use mcp__gcloud__run_gcloud_command (or gcloud --quiet),
mcp__gcloud-observability__list_log_entries (or gcloud logging read),
mcp__google-dev-knowledge__search_documents / get_documents / answer_query,
and curl REST GETs, plus bundled references (load only the smallest set needed):
references/field-manual.md — read first for operational commands & log queries
references/known-issues.md — BKIs #1–#13, #15–#39 for fast pattern matches
references/agent-gateway.md — topologies, ACT, DNS, TLS, SWP/Explicit Proxy, Trace, teardown
references/policies.md — L4/L7
AuthzPolicy, MCP, UAP v3beta, Model Armor, SGPreferences/agent-registry.md —
servicesvs projectedendpoints, 9 API families, jurisdictionsreferences/agent-identity.md — 3-identity delegation, Cloud Run OIDC vs IAP MCP, WIF
Output report
Always produce a read-only diagnostic report using this exact structure:
# Agent Platform Diagnostic — <one-line summary>
## Context
- Project: <id> (<number>) | Location: <region>
- Agent: <agent_id> | Gateway: <gateway_name>
- Symptom: <exact error message and when it started>
## Evidence gathered
- Agent log query: <filter, summary>
- Gateway log query: <filter, exact failing hostname / extension status>
- IAP / Policy log query: <filter, decision + enforcement mode>
- Registry & Policy state: <entries, AuthzPolicy bindings, roles/iap.egressor or UAP>
## Root cause hypothesis
<ranked root cause(s)>
## Why this fits the evidence
<connect log/config evidence to root cause>
## Recommended fix
<ordered gcloud / curl / YAML / Terraform changes for the user to apply>
## What to verify after the fix
<verification queries>
## Open questions / unknowns
<missing context or unverified items>
## Appendix: Raw Logs & Verified Links
- **Cloud Logging Filter**: <exact copy-pasteable filter or deep link>
- **Raw Logs**: <untruncated Agent, Gateway, and IAP/Policy log entries>Principles
Exact hostname match first: Compare gateway log hostnames directly against Agent Registry.
Layered default-deny: Registry → Gateway
AuthzPolicy→ Extension → IAP/UAP/Model Armor/SGP → PAB (PAB overrides IAM Allow).L4 bypass vs L7 governance: L4
networkRulesenablesHTTP CONNECTexplicit proxies (:443) but bypasses L7 IAP, Model Armor, and SGP for matched SNIs.Read-only, single-region, non-interactive: Pull evidence first, never mutate resources, check
us-central1unless logs indicate otherwise, and pass--quiet.
Supporting Links
Files
- ATTRIBUTION.mdAdded by AutoGineer878 B
- LICENSELicence file added by AutoGineer11.1 KiB
- SKILL.md26.2 KiB
- references/agent-gateway.md55.2 KiB
- references/agent-identity.md33.6 KiB
- references/agent-registry.md49.7 KiB
- references/attribution.md1.9 KiB
- references/field-manual.md68.3 KiB
- references/known-issues.md87.6 KiB
- references/policies.md75.2 KiB
references/agent-registry.md
Agent Registry — Debugging Reference
Curated reference for diagnosing why agents, MCP servers, and endpoints registered in Google Cloud Agent Registry are not accepting traffic through Agent Gateway / IAP.
Scope: Gemini Enterprise Agent Platform — Agent Registry (v1 GA). Core commands
use gcloud agent-registry (gcloud beta agent-registry does not exist, and
endpoints is read-only with no create subcommand; all manual writes go through
gcloud agent-registry services). Standalone skills use gcloud agent-registry skills (list / describe).
Table of Contents
TL;DR for Debugging: Lines 29-149
1. The Data Model in One Picture: Lines 150-197
2. Default-Deny + Hostname Matching: Lines 198-338
3. Resource Type — Agents: Lines 339-472
4. Resource Type — MCP Servers: Lines 473-561
5. Resource Type — Endpoints: Lines 562-705
6. IAM Scope: Registry-Wide vs Per-Resource: Lines 706-845
7. Common Registration Errors and Where to Look: Lines 846-877
8. Discovery via the Agent Registry MCP Server: Lines 878-906
9. Quick Reference — Every gcloud Subcommand: Lines 907-971
Source Index: Lines 972-998
TL;DR for Debugging
Read these first when something is broken.
Agent Registry is default-deny at the gateway. Agent Gateway uses Identity-Aware Proxy (IAP) to enforce authorization:
UAP (Policy V2): When the gateway uses
iapPolicyVersion: "V2", Gatekeeper queries the attached registries. If the destination matches, Gatekeeper setsdestination.is_registered = trueand exposes rich CEL metadata underdestination.agent_registry.*. If not cataloged,destination.is_registered = false.Legacy IAM v1: When using legacy v1, if a target hostname is not registered as a
Service(and a matchingroles/iap.egressorbinding does not exist on the registry entry for the calling agent's principal), egress is blocked. Source:docs.cloud.google.com/iap/docs/agent-overview.
There are exactly three discoverable resource types, and they are projections of the same writable
Service:Agent (read-only
Agent)MCP server (read-only
McpServer)Endpoint (read-only
Endpoint) You always create/update/delete viaservices. You list/describe viaagents,mcp-servers,endpoints, orservices. Source:docs.cloud.google.com/agent-registry/concepts.
Hostname matching is exact. A single Google API can be reached at five+ distinct hostnames (base, mTLS, locational, locational-mTLS, REP regional). Each hostname is a separate
Serviceregistration. If your agent callsbigquery.us-central1.rep.googleapis.combut you only registeredbigquery.googleapis.com, the call is denied. See the "Hostname Matrix" section.Manual registration is not supported in
usoreumulti-region locations.usandeuare read-only roll-up discovery locations for automatically registered Gemini Enterprise components and standalone skills. Always registerServiceentries andBindingresources in a specific region (e.g.,us-central1,europe-west1) orglobal.Two predefined IAM roles control the registry itself:
roles/agentregistry.viewer(agentregistry.{agents,endpoints,mcpServers}.list) — discover entries and required by the caller importing anAgentGatewayreferencing the registry.roles/agentregistry.editor— create/update/deleteServiceresources. These are project-scoped catalog permissions, not runtime traffic permissions.
Traffic permissions live on IAP (Policy V1 vs. UAP Policy V2):
Policy V1 (
gcloud iap web): Grantroles/iap.egressor(iap.webServiceVersions.egressViaIAP) with--resource-type=agent-registryalways set, optionally scoped with--agent=AGENT_UUID,--mcp-server=MCP_UUID, or--endpoint=ENDPOINT_UUID(using the server-generatedregistryResourcebasename, neverSERVICE_NAME).UAP Policy V2 (
iapPolicyVersion: "V2"): Grantiap.googleapis.com/resources.egressViaIAPviagcloud iam access-policiesandgcloud iam policy-bindings.
Identifiers matter:
SERVICE_NAME/SERVICE_ID— your chosen ID (4–63 chars[a-z0-9-]) forgcloud agent-registry services.registryResourceUUID — server-generated child ID under/endpoints/,/mcpServers/, or/agents/required bygcloud iap web.Agent / MCP server URN — immutable identifier (
urn:mcp:...,urn:agent:...) used by discovery tools andExtensionBindingmatchers.Resource URI — runtime URI embedded in the SPIFFE principal.
Specs have a hard 10 KB cap and use plain file paths (no
@). Pass--mcp-server-spec-content=toolspec.jsonor--agent-spec-content=agent-card.json(<= 10 KB;gcloudarg_jsonrejectscurl-style@file.json).Automatic registration vs. manual: Agent Runtime (SDK), Google Workspace, Gemini Enterprise, annotated GKE (
registry.gke.io/functional-type: "MCP_SERVER"), Cloud Run services with--functional-type={agent|mcp-server}, and enabled Google/Google Cloud remote MCP servers (created inglobalonly) register automatically within the governance project only. Cross-project workloads, standard Cloud Run services, on-prem, and third-party APIs require manualgcloud agent-registry services create.Cloud Run
.run.appvs..a.run.apptrap: Automatic registration records the deterministic regional URL (https://<SERVICE>-<PROJECT_NUMBER>.<REGION>.run.app). Calling the legacy hash URL (https://<SERVICE>-<HASH>-uc.a.run.app) fails exact hostname matching unless manually registered.
First debug command: confirm the exact target URL is registered (
gcloud agent-registry endpoints list --project=$PROJECT_ID --location=$LOCATIONandgcloud agent-registry services list --filter="interfaces.url:FRAGMENT").Governance project scope (no automatic cross-project discovery). Automatic registration only discovers resources inside the governance project where
agentregistry.googleapis.comis enabled. Remote agents and MCP servers in separate Google Cloud projects must be manually registered in the central governance project's registry.Agent Gateway Dual-Registry & Jurisdiction Rules: An
AgentGatewaysupports a maximum of two registries inregistries[]. When two registries are configured, exactly one must beglobaland the other must be regional (e.g.,us-central1) or multi-regional (us,eu).Binding two regional registries (e.g.,
us-central1andeurope-west1) is invalid and rejected with an HTTP400validation error.Jurisdiction alignment: A regional gateway can only reference a regional registry in the same region or a multi-region registry in the same data residency jurisdiction (e.g.,
us-central1gateway +euregistry is rejected withInvalidArgument).Gateway import preflight: If
agent-gateways importfails withConfig validation failedandmissing permissions: agentregistry.agents.list, enableagentregistry.googleapis.comon the registry project and ensure the deploying caller holdsroles/agentregistry.viewer(agentregistry.{agents,endpoints,mcpServers}.list).
REST API Path Extensions & Explicit Forward Proxies: SDK clients such as Zendesk append
.json(/api/v2/tickets.json) — account for.json,/subpaths, and?query strings in UAP CEL rules. Note that an explicit forwardHTTPS_PROXYtunnel (HTTP CONNECT) requires av1beta1L4AuthzPolicy(networkRulesmatching SNIs), not an Agent Registry entry for the proxy host (seereferences/agent-gateway.mdand BKI #34).
1. The Data Model in One Picture
WRITE side READ side (read-only projections)
+--------------+ +----------+
| | endpoint-spec --> | Endpoint |
| Service | mcp-server-spec ->| McpServer|
| (writable) | agent-spec -----> | Agent |
+--------------+ +----------+
^ ^
create/update/delete get/list/search
(services subcommand) (endpoints / mcp-servers /
agents subcommand)The
Serviceresource is the only thing you write.Agent Registry automatically generates the corresponding
Agent,McpServer, orEndpointprojection based on the*-spec-typeyou passed at create time.The mutually exclusive
*-spec-typeflag at creation determines the read-only projection (spec types cannot be changed onservices update):services createSpec FlagAllowed --interfacesRead-Only Projection --endpoint-spec-type=no-specRequired (1+ URLs) Endpoint--mcp-server-spec-type={tool-spec|no-spec}Required ( protocolBinding=JSONRPC)McpServer--agent-spec-type=a2a-agent-cardMust be omitted (read from agent-card.json)Agent(with A2A skills)--agent-spec-type=no-specRequired (REST URL) Agent(non-A2A REST)(Standalone
Skillresources do not useService; inspect viagcloud agent-registry skills list --location=LOCATION --project=PROJECT_ID.)
Source: docs.cloud.google.com/agent-registry/overview,
docs.cloud.google.com/agent-registry/concepts.
URN formats (memorize these)
Manually registered MCP server URN:
urn:mcp:projects-PROJECT_NUMBER:projects:PROJECT_NUMBER:locations:REGION:agentregistry:SERVICE_IDGoogle Cloud remote MCP server URN:
urn:mcp:googleapis.com:projects:PROJECT_NUMBER:locations:global:SERVICE_NAME
If a discovery query returns no results for what you think is a registered server, check the URN format vs the location you registered against.
Source: docs.cloud.google.com/agent-registry/concepts.
2. Default-Deny + Hostname Matching
This is the single most common debugging failure mode. Read this whole section.
Why default-deny
Agent Gateway is the egress proxy for agents. It is built on IAP. IAP only
forwards a request if there is an IAM allow policy granting roles/iap.egressor
to the calling agent's principal on a registry resource that matches the
request. If no Service matches the destination hostname, no policy can
match, and the request is denied.
Source: docs.cloud.google.com/iap/docs/agent-overview.
Hostname Matrix — register every form an agent might call
A single Google Cloud API can be addressed via several distinct hostnames. Each one is a separate registry entry. The canonical set includes the following; if any are missing for an API your agent uses, expect denial.
| Variant | Hostname pattern | Example (bigquery, us-central1) |
|---|---|---|
| Base / global | SERVICE.googleapis.com | bigquery.googleapis.com |
| mTLS | SERVICE.mtls.googleapis.com | bigquery.mtls.googleapis.com |
| Locational | LOCATION-SERVICE.googleapis.com | us-central1-bigquery.googleapis.com |
| Locational mTLS | LOCATION-SERVICE.mtls.googleapis.com | us-central1-bigquery.mtls.googleapis.com |
| REP (public) | SERVICE.LOCATION.rep.googleapis.com | bigquery.us-central1.rep.googleapis.com |
| REP (private/PSC) | SERVICE.LOCATION.p.rep.googleapis.com | bigquery.us-central1.p.rep.googleapis.com |
Locational endpoints specify the region/multi-region in the URL prefix. Format:
LOCATION-SERVICE.googleapis.com. Locational endpoints don't support data-residency-compliant connections from on-premises and don't support failure-domain isolation; REP is replacing them.REP (Regional Endpoint) confines TLS termination and traffic to the specified region. Format:
SERVICE.REGION.rep.googleapis.com. Private REP via PSC adds.p..The base/global hostname terminates TLS at the nearest edge and provides no regional isolation.
Source:
docs.cloud.google.com/docs/security/compliance/restrict-endpoint-usage.
Recommended Pattern: Consolidated Google APIs Service
Warning: MANDATORY GOOGLE APIS CONSOLIDATION RULE: NEVER recommend or register separate
Serviceresources for individual Google APIs (such as Agent Runtime, Cloud Resource Manager, IAM, or Telemetry). Do NOT register each Google API as a separate service entry, as separate service entries cause resource clutter, complicate IAM policy management, and risk hitting registry quota limits. ALL Google APIs MUST ALWAYS be consolidated under a single Agent Registry Service namedgoogleapisusing multiple interfaces.
Instead of registering dozens of separate Service resources for each Google
API and its regional/mTLS permutations, consolidate all Google APIs under a
single Agent Registry Service named googleapis using multiple interfaces.
This reduces resource clutter, simplifies IAM policy management (granting
roles/iap.egressor once), and avoids registry quota limits.
The 8 Mandatory Base Interfaces + Essential Platform Families
For every Agent Runtime deployment behind an ALL_TRAFFIC Egress Agent Gateway,
the consolidated googleapis service must include the 8 base FQDN
interfaces:
https://agentregistry.googleapis.comhttps://aiplatform.mtls.googleapis.comhttps://cloudresourcemanager.mtls.googleapis.comhttps://iamcredentials.mtls.googleapis.comhttps://telemetry.mtls.googleapis.comhttps://{region}-aiplatform.mtls.googleapis.com(e.g.,us-central1)https://{region}-aiplatform.googleapis.comhttps://aiplatform.{region}.rep.googleapis.com
Depending on model availability and runtime features, include these additional
interfaces from the 9 essential Google Cloud API families on the same
googleapis service:
Global & Multi-Region Foundation Models: Many Gemini models only expose
globalor multi-region (us,eu) endpoints. Addhttps://aiplatform.googleapis.com(alongsideaiplatform.mtls.googleapis.com) and, if targetingus/eu,https://us-aiplatform.googleapis.com,https://us-aiplatform.mtls.googleapis.com, andhttps://aiplatform.us.rep.googleapis.com. Calling an unsupported regional model returnsHTTP 404 NOT_FOUND; callingglobalwithout registeringaiplatform.googleapis.comdrops TLS at240.0.0.2:443(UNEXPECTED_EOF).Tracing, Logging, Monitoring, Storage, & Artifact Registry: Add
https://cloudtrace.googleapis.com,https://cloudtrace.mtls.googleapis.com,https://trace.googleapis.com,https://trace.mtls.googleapis.com,https://monitoring.googleapis.com,https://monitoring.mtls.googleapis.com,https://logging.googleapis.com,https://logging.mtls.googleapis.com,https://storage.googleapis.com,https://storage.mtls.googleapis.com,https://artifactregistry.googleapis.com, andhttps://artifactregistry.mtls.googleapis.com.Multi-Region
.rep.mtlsNXDOMAINPitfall: Google Cloud does not publish DNS records foraiplatform.us.rep.mtls.googleapis.comoraiplatform.eu.rep.mtls.googleapis.com. On oldergoogle-genaiSDK versions withAGENT_IDENTITYenabled andlocation="us"or"eu", automatic mTLS URL rewriting fails withNXDOMAIN(Name or service not known). Fix: upgradegoogle-genai, targetglobalor a single region, or setGOOGLE_API_USE_MTLS_ENDPOINT=neverandGOOGLE_API_USE_CLIENT_CERTIFICATE=false.
Creating Consolidated Service via gcloud
gcloud agent-registry services create googleapis \
--project=$PROJECT_ID --location=$LOCATION \
--display-name="Google APIs" \
--endpoint-spec-type=no-spec \
--interfaces=url=https://agentregistry.googleapis.com,protocolBinding=JSONRPC \
--interfaces=url=https://aiplatform.googleapis.com,protocolBinding=JSONRPC \
--interfaces=url=https://aiplatform.mtls.googleapis.com,protocolBinding=JSONRPC \
--interfaces=url=https://cloudresourcemanager.mtls.googleapis.com,protocolBinding=JSONRPC \
--interfaces=url=https://iamcredentials.mtls.googleapis.com,protocolBinding=JSONRPC \
--interfaces=url=https://telemetry.mtls.googleapis.com,protocolBinding=JSONRPC \
--interfaces=url=https://cloudtrace.mtls.googleapis.com,protocolBinding=JSONRPC \
--interfaces=url=https://trace.mtls.googleapis.com,protocolBinding=JSONRPC \
--interfaces=url=https://monitoring.googleapis.com,protocolBinding=JSONRPC \
--interfaces=url=https://$LOCATION-aiplatform.mtls.googleapis.com,protocolBinding=JSONRPC \
--interfaces=url=https://$LOCATION-aiplatform.googleapis.com,protocolBinding=JSONRPC \
--interfaces=url=https://aiplatform.$LOCATION.rep.googleapis.com,protocolBinding=JSONRPCQuick "is the hostname registered?" check
gcloud agent-registry services list \
--project=$PROJECT_ID \
--location=$LOCATION \
--format="table(name.basename(), interfaces.url)" \
--filter="interfaces.url:HOSTNAME_FRAGMENT"Replace HOSTNAME_FRAGMENT with e.g. bigquery.us-central1.rep.
Source: hostname patterns derived from
docs.cloud.google.com/docs/security/compliance/restrict-endpoint-usage;
registration flow from
docs.cloud.google.com/agent-registry/register-endpoints.
3. Resource Type — Agents
Concept
Agents: Autonomous actors that possess specific skills. Skills represent an agent's high-level capabilities and are a primary mechanism for discovery. --
docs.cloud.google.com/agent-registry/concepts
Two kinds: - A2A-compliant agents -- implement Agent2Agent. Registry scans
their agent-card.json to extract skills. - Standard REST agents --
registered as NO_SPEC type when no A2A spec exists.
Automatic registration sources: Agent Runtime (via SDK), Google Workspace built-ins, Gemini Enterprise built-ins, GKE deployments with the Agent Registry functional-type annotation. Everything else is manual.
List
gcloud agent-registry agents list \
--project=PROJECT_ID \
--location=REGIONFilter:
gcloud agent-registry agents list \
--project=PROJECT_ID \
--location=REGION \
--filter="displayName='DISPLAY_NAME'"Describe
gcloud agent-registry agents describe AGENT_NAME \
--project=PROJECT_ID \
--location=REGIONOutput exposes core details, skills (if A2A), and Resource URI -- the runtime
location of the agent, which is what gets embedded in the agent principal for
IAM.
Update (via services, not agents)
Display name / description:
gcloud agent-registry services update AGENT_NAME \
--project=PROJECT_ID \
--location=REGION \
--display-name="New name" \
--description="Updated description"Endpoint URL:
gcloud agent-registry services update AGENT_NAME \
--project=PROJECT_ID \
--location=REGION \
--interfaces=url=ENDPOINT_URL,protocolBinding=PROTOCOLValid protocolBinding: HTTP_JSON, GRPC, JSONRPC.
Agent spec payload (max size 10 KB to satisfy gRPC control plane metadata header limits):
gcloud agent-registry services update AGENT_NAME \
--project=PROJECT_ID \
--location=REGION \
--agent-spec-content=agent-card.jsonDelete
For automatically registered agents: delete from the source runtime. For manually registered agents:
gcloud agent-registry services delete AGENT_NAME \
--project=PROJECT_ID \
--location=REGIONIAM model for agents
Registry permission to manage:
roles/agentregistry.editoron the project.Agent-to-agent egress (Policy V1): extract the generated
AgentUUID (basenameofregistryResource) and bindroles/iap.egressorwith--resource-type=agent-registry:AGENT_UUID=$(basename "$(gcloud agent-registry services describe AGENT_NAME \ --project=PROJECT_ID --location=REGION --format='value(registryResource)')") gcloud iap web set-iam-policy agents-iap-policy.json \ --project=PROJECT_ID \ --resource-type=agent-registry \ --agent="$AGENT_UUID" \ --region=REGIONThe policy's
membersuses the calling agent's principal (principal://agents.global.org-123456789012.system.id.goog/resources/aiplatform/projects/9876543210/locations/us-central1/reasoningEngines/12345, orglessagents.global.proj-9876543210.system.id.goog/..., or WIFprincipal://iam.googleapis.com/...).
Common pitfalls
Agent in another project missing from central registry: automatic registration only discovers workloads inside the governance project. Manually register cross-project agents with
gcloud agent-registry services createin the governance project (never inusoreu).Agent appears in
services listbut notagents list: wrong spec type at creation (--agent-spec-type=a2a-agent-cardor--agent-spec-type=no-specrequired).Cloud Run agent missing from registry: standard Cloud Run services do not auto-register unless deployed with
gcloud beta run deploy --functional-type=agent --identity-type=agent-identity.Gemini Enterprise egress ignores Agent Gateway: Gemini Enterprise requires two toggles — (1)
PATCHtheEngineon the regional endpoint (https://{global-|us-|eu-}discoveryengine.googleapis.com/v1/projects/PROJECT_NUMBER/locations/GE_LOCATION/collections/default_collection/engines/ENGINE_ID?updateMask=agentGatewaySetting.defaultEgressAgentGateway.name) and (2) set"use_agent_gateway_egress": trueon each imported MCP/A2ADataConnector(see BKI #37).A2A agent missing skills:
agent-card.jsonexceeds 10 KB, or--interfaceswas passed alongside--agent-spec-type=a2a-agent-card(forbidden).
Sources: docs.cloud.google.com/agent-registry/manage-agents,
docs.cloud.google.com/agent-registry/register-agents,
docs.cloud.google.com/gemini-enterprise-agent-platform/govern/policies/configure-iam-policies.
4. Resource Type — MCP Servers
Concept
MCP servers: Providers of standardized data resources and tools. Tools are deterministic functions exposed by an MCP server that agents can invoke to perform specific actions. --
docs.cloud.google.com/agent-registry/concepts
Official Google and Google Cloud remote MCP servers are auto-registered on API enable. External / custom MCP servers (e.g. Cloud Run-hosted FastMCP) need manual registration with a tool spec.
Register
gcloud agent-registry services create SERVER_NAME \
--project=PROJECT_ID \
--location=REGION \
--display-name="DISPLAY_NAME" \
--mcp-server-spec-type=tool-spec \
--mcp-server-spec-content=toolspec.json \
--interfaces=url=SERVER_URL,protocolBinding=JSONRPCprotocolBindingfor MCP must beJSONRPC(jsonrpc), andtoolspec.jsonmust be<= 10 KB(do not prefix the path with@). Use--mcp-server-spec-type=no-spec(without--mcp-server-spec-content) when not uploading a static tool catalog.
List & Describe
gcloud agent-registry mcp-servers list --project=PROJECT_ID --location=REGION
gcloud agent-registry mcp-servers describe MCP_SERVER_UUID --project=PROJECT_ID --location=REGIONUpdate tool spec & Delete
gcloud agent-registry services update SERVER_NAME \
--project=PROJECT_ID --location=REGION \
--mcp-server-spec-content=toolspec.json
gcloud agent-registry services delete SERVER_NAME --project=PROJECT_ID --location=REGIONIAM model for MCP servers
Discover / manage catalog:
roles/agentregistry.viewer/roles/agentregistry.editor.Call Agent Registry remote MCP server:
roles/mcp.toolUser+roles/agentregistry.viewer.Agent-to-MCP egress (Policy V1): pass
--resource-type=agent-registryand the server-generatedMcpServerUUID (registryResourcebasename):gcloud iap web set-iam-policy agents-iap-policy.json \ --project=PROJECT_ID \ --resource-type=agent-registry \ --mcp-server=MCP_SERVER_UUID \ --region=REGIONPolicy V1 CEL attributes:
iap.googleapis.com/mcp.toolName,iap.googleapis.com/mcp.tool.isReadOnly,iap.googleapis.com/request.auth.type('MCP').
Common pitfalls
Google/Google Cloud remote MCP server returns
NOT_FOUNDin a region: auto-registered Google remote MCP servers exist only inglobal(even when exposing regional.rep.googleapis.com/mcpURLs). Always query with--location=globaland bind IAP policies with--region=global.Cloud Run MCP server
403unregistered mismatch (.a.run.appvs..run.app):gcloud beta run deploy --functional-type=mcp-serverregistershttps://<SERVICE>-<PROJECT_NUMBER>.<REGION>.run.app. Configure the agent to call that exact.run.appURL (not the legacy*-uc.a.run.appURL).Native MCP
AuthzPolicyblocksinitialize/ping: when attaching a Network SecurityAuthzPolicywithmcptool rules, setbaseProtocolMethodsOption: MATCH_BASE_PROTOCOL_METHODSonALLOWrules (seereferences/policies.mdand BKI #39).Tool spec rejected or stale: re-run
services update --mcp-server-spec-content=toolspec.json(<= 10 KB, no@) after tool changes; never register manually inusoreu.
Sources: docs.cloud.google.com/agent-registry/register-mcp-servers,
docs.cloud.google.com/agent-registry/manage-mcp-tools,
docs.cloud.google.com/agent-registry/use-agentregistry-mcp,
docs.cloud.google.com/gemini-enterprise-agent-platform/govern/policies/configure-iam-policies.
5. Resource Type — Endpoints
Concept
Endpoints: Target URLs, typically REST APIs, accessed by an agent. By abstracting these destinations into manageable resources, Agent Registry lets you centrally govern which external services your agents can access. --
docs.cloud.google.com/agent-registry/concepts
Endpoints are how you authorize an agent to reach anything that isn't an agent or an MCP server: Google APIs (BigQuery, Agent Runtime, etc.), third-party REST APIs, internal HTTP services. Always manually registered.
Register
gcloud agent-registry services create SERVICE_NAME \
--project=PROJECT_ID \
--location=REGION \
--display-name="DISPLAY_NAME" \
--endpoint-spec-type=no-spec \
--interfaces=url=ENDPOINT_URL,protocolBinding=PROTOCOL--endpoint-spec-type=no-specis mandatory -- this is what causes the read-onlyEndpointprojection to appear.After creation, Agent Registry auto-generates the read-only
Endpointresource that agents discover.
Hostname registration is exact-match. You must register every hostname variant the agent might dial. See "Hostname Matrix" in section 2. The fix for "endpoint registered but traffic denied" is almost always: register the missing variant.
List
gcloud agent-registry endpoints list \
--project=PROJECT_ID \
--location=REGIONFilter:
gcloud agent-registry endpoints list \
--project=PROJECT_ID \
--location=REGION \
--filter="displayName='NAME'"For URL-substring search, list services instead and filter on
interfaces.url:
gcloud agent-registry services list \
--project=PROJECT_ID \
--location=REGION \
--format="table(name.basename(), interfaces.url, interfaces.protocolBinding)" \
--filter="interfaces.url:HOSTNAME_FRAGMENT"Describe
gcloud agent-registry endpoints describe ENDPOINT_NAME \
--project=PROJECT_ID \
--location=REGIONUpdate
gcloud agent-registry services update SERVICE_NAME \
--project=PROJECT_ID \
--location=REGION \
--interfaces=url=ENDPOINT_URL,protocolBinding=PROTOCOLDelete
gcloud agent-registry services delete SERVICE_NAME \
--project=PROJECT_ID \
--location=REGION"This action immediately removes the endpoint from discovery search results." Google Cloud console UI requires typing
DELETEto confirm.
IAM model for endpoints
Discover endpoints:
roles/agentregistry.viewer.Register / update / delete:
roles/agentregistry.editor.Agent-to-endpoint traffic: bind
roles/iap.egressorto the source agent's principal on the endpoint resource:gcloud iap web set-iam-policy agents-iap-policy.json \ --project=PROJECT_ID \ --resource-type=agent-registry \ --endpoint=ENDPOINT_UUID \ --region=REGIONPer-endpoint policy bindings do not require a CEL condition; you can omit
conditionto grant unconditional egress to that one endpoint, or add a CEL condition for attribute-based scoping (e.g. byName,ReadOnly,Destructive,Idempotent,OpenWorldproperties).
Note on roles/iap.tunnelResourceAccessor
Some IAP-protected resources use roles/iap.tunnelResourceAccessor
(iap.tunnelDestGroups.accessViaIAP, iap.tunnelInstances.accessViaIAP) --
that role governs classic IAP TCP tunnels to GCE instances and dest groups.
Agent Gateway egress to Agent Registry resources does NOT use this role; it
uses roles/iap.egressor (iap.webServiceVersions.egressViaIAP). If you see a
tutorial suggesting tunnelResourceAccessor for agent egress, it's the wrong
role -- confirm against docs.cloud.google.com/iam/docs/roles-permissions/iap.
Common pitfalls
Test connectionfails in Google Cloud console for a private URL: expected. The Google Cloud console connection test only validates public URLs; it does not support private endpoints (Agent Runtime, internal HTTPS, etc.).Agent gets denied on a Google API call: the API has a hostname variant you didn't register (commonly the REP regional form
${id}.${LOCATION}.rep.googleapis.com).Endpoint registered in
globalbut agent calls a regional URL: registry location doesn't have to match the destination URL location, but the IAM binding's--regionmust match where the resource is registered. Check that--region=ongcloud iap web set-iam-policyequals the location used at creation.Endpoint not appearing in
endpoints list: created with the wrong spec type. Must use--endpoint-spec-type=no-spec.
Sources: docs.cloud.google.com/agent-registry/register-endpoints,
docs.cloud.google.com/agent-registry/manage-endpoints,
docs.cloud.google.com/iam/docs/roles-permissions/iap,
docs.cloud.google.com/gemini-enterprise-agent-platform/govern/policies/configure-iam-policies.
6. IAM Scope: Registry-Wide vs Per-Resource
There are two layers. Don't conflate them.
Layer A — Agent Registry permissions (manage the catalog)
Project-scoped, granted via standard IAM:
| Role | Use |
|---|---|
roles/agentregistry.viewer | List / describe / search agents, MCP, |
| : : endpoints : | |
roles/agentregistry.editor | Create / update / delete Service resources |
roles/mcp.toolUser | Call tools via the Agent Registry remote MCP |
: : server (mcp.tools.call) : |
These let humans and CI register and inspect entries. They do not grant runtime traffic.
Layer B — Traffic Authorization (Legacy v1 vs. UAP Policy V2)
Depending on the gateway's iapPolicyVersion setting, registry resources
authorize traffic through one of two mechanisms:
Mechanism B1: Unified Access Policy (UAP / Policy V2)
When the attached IAP Authz Extension specifies iapPolicyVersion: "V2",
policies are decoupled from individual registry resources:
AccessPolicyandPolicyBinding(iam.googleapis.com/v3) attach to the Resource Manager hierarchy (Organization, Folder, Project) targeting numeric resource URIs (e.g.//cloudresourcemanager.googleapis.com/projects/${PROJECT_NUMBER}).Rules grant FQDN permission
iap.googleapis.com/resources.egressViaIAP.Agent Registry entries dynamically populate the
destination.agent_registry.*CEL attributes evaluated at runtime:destination.is_registered(boolean):trueif target matches a registry service.destination.agent_registry.resource_type:'ENDPOINT','MCP_SERVER','AGENT','SKILL'.destination.agent_registry.location: region of registry resource.destination.agent_registry.mcp_server.name: full MCP server resource name.destination.agent_registry.mcp_server.method:'tools','prompts','resources'.destination.agent_registry.mcp_server.tool.name: specific tool name (e.g.'updateTicket').destination.agent_registry.mcp_server.tool.annotations.read_only_hint: boolean.
Mandatory CEL Guard: Always guard tool annotations with:
(destination.agent_registry.mcp_server.method == 'tools') && (destination.agent_registry.mcp_server.tool.annotations.read_only_hint == true).
Mechanism B2: Legacy IAM v1 (Per-Resource Bindings)
In legacy v1, permissions are bound directly on Agent Registry resources via
gcloud iap web set-iam-policy. The role is always roles/iap.egressor. The
scope of the binding determines which target resources the source agent can
reach:
| Scope | Required gcloud iap web Target Flags | Grants Egress To |
|---|---|---|
| Registry-wide | --resource-type=agent-registry | All agents, MCP servers, and endpoints in that project/region registry |
| Single agent | --resource-type=agent-registry --agent=AGENT_UUID | One target agent (registryResource basename) |
| Single MCP server | --resource-type=agent-registry --mcp-server=MCP_SERVER_UUID | One target MCP server (registryResource basename) |
| Single endpoint | --resource-type=agent-registry --endpoint=ENDPOINT_UUID | One target endpoint (registryResource basename) |
All four require --project=PROJECT_ID and --region=LOCATION (global or
region). Never omit --resource-type=agent-registry: when --region is set
without --resource-type, gcloud iap web targets IAPWebLocation
(projects/PROJECT_NUMBER/locations/LOCATION/iap_web) and silently ignores
--agent, --mcp-server, and --endpoint (mutating the regional iap_web
container policy or failing on --region=global); when both --resource-type
and --region are omitted, it targets project-wide IAPWeb
(projects/PROJECT_NUMBER/iap_web).
Warning: Top-Level
{"bindings": [...]}Required forgcloud iap web set-iam-policy:gcloud iap web set-iam-policyparses the file directly as an IAMPolicyobject (bindings,etag,version) and wrapsSetIamPolicyRequestinternally. Do NOT wrap"bindings"inside"policy": { ... }when usinggcloud—apitoolsdiscards the outer"policy"key and submits an empty policy that silently wipes all existing IAP IAM bindings! (Only raw RESTPOST ...:setIamPolicyrequests use the{"policy": {"bindings": [...]}}envelope.)
The Policy V1 JSON file for gcloud iap web set-iam-policy uses top-level "bindings":
{
"bindings": [
{
"role": "roles/iap.egressor",
"members": ["AGENT_PRINCIPAL"]
}
],
"version": 3
}Agent principal formats (members / principals)
Agent Runtime / Gemini Enterprise (org-backed —
ENGINE_IDis numeric):principal://agents.global.org-ORGANIZATION_ID.system.id.goog/resources/aiplatform/projects/PROJECT_NUMBER/locations/REGION/reasoningEngines/ENGINE_IDAgent Runtime (orgless project — note
proj-, NOTproject-):principal://agents.global.proj-PROJECT_NUMBER.system.id.goog/resources/aiplatform/projects/PROJECT_NUMBER/locations/REGION/reasoningEngines/ENGINE_IDWorkload Identity Pool (DIY / external):
principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/subject/SUBJECTProject-wide Agent PrincipalSet:
principalSet://agents.global.org-ORGANIZATION_ID.system.id.goog/attribute.platformContainer/aiplatform/projects/PROJECT_NUMBER
When debugging "wrong principal" errors: the principal is built from the agent's
resource URI, not its display name and not its URN. View the resource URI
via gcloud agent-registry agents describe.
Full IAP role reference (relevant subset)
| Role | Permission grants the role provides | When to use |
|---|---|---|
roles/iap.egressor (Beta) | iap.webServiceVersions.egressViaIAP | **Agent |
| : : : egress : | ||
| : : : through Agent : | ||
| : : : Gateway** : | ||
roles/iap.httpsResourceAccessor | iap.webServiceVersions.accessViaIAP | Human/service |
| : : : ingress to an : | ||
| : : : IAP-protected : | ||
| : : : web app : | ||
roles/iap.tunnelResourceAccessor | iap.tunnelDestGroups.accessViaIAP, | Classic IAP |
: : iap.tunnelInstances.accessViaIAP : TCP tunnels : | ||
| : : : to GCE -- : | ||
| : : : not for : | ||
| : : : agent gateway : | ||
roles/iap.admin | Full IAP admin (iap.tunnel.*, all | Manage IAP |
| : : getIam/setIam) : policies : | ||
roles/iap.viewer | Read IAP settings | Audit |
Source: docs.cloud.google.com/iam/docs/roles-permissions/iap.
7. Common Registration Errors and Where to Look
| Symptom | Likely cause | Where to check |
|---|---|---|
services create fails: "manual | Used --location=us or | Switch to a region (e.g. |
: registration not supported" : --location=eu (multi-regions) : us-central1) or global : | ||
services create succeeds but | Wrong *-spec-type. | Re-create with the correct |
: resource missing from endpoints : --endpoint-spec-type=no-spec : spec-type, or checkservices list` : | ||
: list : produces anEndpoint`. : : | ||
| `services update | Spec > 10 KB | Trim the tool spec |
| : --mcp-server-spec-content` rejects : : : | ||
| : file : : : | ||
| Agent calls API X, IAP returns 403 | Hostname variant not registered | `services list |
| : : (commonly REP : --filter="interfaces.url:FRAGMENT"` : | ||
: : *.LOCATION.rep.googleapis.com) : : | ||
| Agent calls API X, IAP returns 403 even though hostname is registered | Missing roles/iap.egressor binding for the source agent's principal | gcloud iap web get-iam-policy --project=P --region=L --resource-type=agent-registry [--endpoint=UUID | --mcp-server=UUID | --agent=UUID] |
| MCP tool call succeeds for one tool, denied for another | IAP policy condition restricts mcp.toolName or mcp.tool.isReadOnly | Inspect the IAP policy on the MCP server resource |
| Auto-registered Agent Runtime agent missing from registry | Agent deployed via custom code, not Agent Runtime SDK | Re-deploy via Agent Runtime, or register manually as a Service |
| A2A agent has no skills in registry | agent-card.json unreachable from Google's crawler or > 10 KB | Curl the Agent Card URL from a public network; verify content type, schema, and <= 10 KB size |
| Agent principal mismatch in IAP policy | Principal built from wrong resource URI / project number / project- typo instead of proj- | gcloud agent-registry agents describe → use Resource URI and numeric PROJECT_NUMBER |
roles/iap.tunnelResourceAccessor granted, still denied | Wrong role — agent egress requires roles/iap.egressor | Replace the binding with roles/iap.egressor |
| Endpoint Google Cloud console "Test connection" fails on private URL | Connection test only works for public URLs (by design) | Test via the agent itself or curl from a peered network |
agents-iap-policy.json set, still denied | Wrong --region flag on set-iam-policy, omitted --resource-type=agent-registry (wrote to IAPWebLocation iap_web), or wrapped JSON in {"policy": ...} (wiped bindings) | Pass --resource-type=agent-registry, match --region (global for Google remote MCPs), and use top-level {"bindings": [...]} |
| Cannot find URN or child UUID | Run services describe for registryResource UUID; mcp-servers/agents describe for URN | See Section 1 URN & UUID rules |
agent-gateways import: Config validation failed (missing permissions: agentregistry.agents.list) | agentregistry.googleapis.com API disabled on registry project, or caller lacks roles/agentregistry.viewer | Enable agentregistry.googleapis.com and grant roles/agentregistry.viewer (agentregistry.{agents,endpoints,mcpServers}.list) |
agent-gateways import 400 / InvalidArgument on registries | Two regional registries (us-central1 + europe-west1), > 2 registries, or cross-jurisdiction (us-central1 + eu) | Configure max 2 registries: global + 1 same-region or same-jurisdiction multi-region registry |
NXDOMAIN on aiplatform.{us,eu}.rep.mtls.googleapis.com | .rep.mtls DNS does not exist for us/eu multi-regions | Upgrade google-genai, use global/regional endpoint, or set GOOGLE_API_USE_MTLS_ENDPOINT=never |
gcloud iap web returns NOT_FOUND or ignores --endpoint/--mcp-server/--agent | Omitted --resource-type=agent-registry (targets IAPWebLocation iap_web), passed SERVICE_NAME instead of UUID, or queried remote MCP in region instead of global | Always pass --resource-type=agent-registry, use registryResource basename, and use --region=global for Google remote MCPs |
Console loading error or gcloud error 13 with Workforce Identity Federation (WIF) | Preview WIF limitation on Agent Registry & Agent Gateway control planes | Authenticate with a Cloud Identity or Google Workspace account (see BKI #36) |
8. Discovery via the Agent Registry MCP Server
The registry itself is exposed as a remote MCP server at
agentregistry.googleapis.com. Useful when debugging from inside an agent or
via mcp inspector. Tools include:
Discovery: - search_agents, search_mcp_servers -- keyword/prefix search -
get_agent, get_mcp_server, get_endpoint -- by resource name -
get_service -- read the underlying writable spec - list_agents,
list_mcp_servers, list_endpoints, list_services - list_bindings,
get_binding, fetch_available_bindings - get_operation
Administrative: - create_service, update_service, delete_service -
create_binding, update_binding, delete_binding
tools/list over MCP does not require auth:
POST /mcp HTTP/1.1
Host: agentregistry.googleapis.com
Content-Type: application/json
{ "jsonrpc": "2.0", "method": "tools/list" }Source: docs.cloud.google.com/agent-registry/use-agentregistry-mcp.
9. Quick Reference — Every gcloud Subcommand
# READ
gcloud agent-registry agents list --project=P --location=L
gcloud agent-registry agents describe NAME --project=P --location=L
gcloud agent-registry mcp-servers list --project=P --location=L
gcloud agent-registry mcp-servers describe NAME --project=P --location=L
gcloud agent-registry endpoints list --project=P --location=L
gcloud agent-registry endpoints describe NAME --project=P --location=L
gcloud agent-registry services list --project=P --location=L
gcloud agent-registry services describe NAME --project=P --location=L
gcloud agent-registry skills list --project=P --location=L
gcloud agent-registry skills describe NAME --project=P --location=L
# WRITE (always 'services'; 'endpoints' has no 'create' subcommand and there is no 'gcloud beta agent-registry')
gcloud agent-registry services create NAME \
--project=P --location=L --display-name="..." \
--endpoint-spec-type=no-spec \
--interfaces=url=URL,protocolBinding=HTTP_JSON|GRPC|JSONRPC
gcloud agent-registry services create NAME \
--project=P --location=L --display-name="..." \
--mcp-server-spec-type=tool-spec \
--mcp-server-spec-content=toolspec.json \
--interfaces=url=URL,protocolBinding=JSONRPC
# A2A agent: omit --interfaces (read from agent-card.json)
gcloud agent-registry services create NAME \
--project=P --location=L --display-name="..." \
--agent-spec-type=a2a-agent-card \
--agent-spec-content=agent-card.json
# Non-A2A REST agent: --agent-spec-type=no-spec + --interfaces
gcloud agent-registry services create NAME \
--project=P --location=L --display-name="..." \
--agent-spec-type=no-spec \
--interfaces=url=URL,protocolBinding=HTTP_JSON
gcloud agent-registry services update NAME \
--project=P --location=L \
[ --display-name="..." | --description="..." \
| --interfaces=url=URL,protocolBinding=... | --clear-interfaces \
| --mcp-server-spec-content=toolspec.json \
| --agent-spec-content=agent-card.json ]
gcloud agent-registry services delete NAME --project=P --location=L
# IAP Policy V1 egress bindings (--resource-type=agent-registry is ALWAYS required)
gcloud iap web set-iam-policy POLICY.json \
--project=P --region=L --resource-type=agent-registry \
[ --agent=AGENT_UUID | --mcp-server=MCP_SERVER_UUID | --endpoint=ENDPOINT_UUID ]
gcloud iap web get-iam-policy \
--project=P --region=L --resource-type=agent-registry \
[ --agent=AGENT_UUID | --mcp-server=MCP_SERVER_UUID | --endpoint=ENDPOINT_UUID ]Spec size cap: 10 KB for --mcp-server-spec-content and
--agent-spec-content (pass plain file path without @).
Manual registration locations: any region or global. Not us or eu.
Source Index
Agent Registry overview: https://docs.cloud.google.com/agent-registry/overview.md.txt
Agent Registry concepts: https://docs.cloud.google.com/agent-registry/concepts.md.txt
Register agents (manual): https://docs.cloud.google.com/agent-registry/register-agents.md.txt
Manage agents: https://docs.cloud.google.com/agent-registry/manage-agents.md.txt
Register MCP servers: https://docs.cloud.google.com/agent-registry/register-mcp-servers.md.txt
Manage MCP servers/tools: https://docs.cloud.google.com/agent-registry/manage-mcp-tools.md.txt
Register endpoints: https://docs.cloud.google.com/agent-registry/register-endpoints.md.txt
Manage endpoints: https://docs.cloud.google.com/agent-registry/manage-endpoints.md.txt
Agent Registry MCP server: https://docs.cloud.google.com/agent-registry/use-agentregistry-mcp.md.txt
Govern (Gemini Enterprise Agent Platform): https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/agent-registry.md.txt
IAP for agents: https://docs.cloud.google.com/iap/docs/agent-overview.md.txt
IAP IAM roles: https://docs.cloud.google.com/iam/docs/roles-permissions/iap.md.txt
Configure IAM policies (IAP egressor examples): https://docs.cloud.google.com/gemini-enterprise-agent-platform/govern/policies/configure-iam-policies.md.txt
Regional service endpoint hostname formats: https://docs.cloud.google.com/docs/security/compliance/restrict-endpoint-usage.md.txt
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs an MCP server
Security
No issues found by our static scan· No issues found
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- Publisher
- Google (Official)
- Source
- google/skills · skills/cloud/agent-platform-troubleshooting · commit 2194d3b
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by Google or Google.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download agent-platform-troubleshooting.zip