Setting Up Cloudwatch Observability

AWSOfficialPublished by AWS

Skill
Tool integration
Cloud and DevOps
Apache-2.0
Medium risk

From the author

Sets up CloudWatch observability for the first time - Omni (CloudWatch Application Observability) and classic CloudWatch. Omni: creating a Space or Domain; access grants (who has access, at what level), access profiles bounding alerts, integrations, agents; instrumenting an app or AI agent with the plain ADOT SDK so traces reach Omni (Python/Node/Java/.NET on EC2/ECS/EKS/Lambda), incl. no-image-rebuild, CoreCLR vars; ingesting Azure or Google Cloud telemetry via the CloudWatch agent (VM/AKS, GCE/GKE); connecting Slack or the AWS DevOps Agent to a Space; whether GitHub or a custom MCP tool server (HTTP/stdio) can connect. CloudWatch: onboarding a service to Application Signals: ADOT auto-instrumentation, amazon-cloudwatch-observability add-on, monitored service, ServiceEvents, CI/CD metadata, Terraform. For using what is set up - queries, dashboards, alarms, Omni alerts, X-Ray, synthetics, Dynamic Instrumentation, AI agent evaluation, detecting Omni vs classic - use aws-observability.

— the description by AWS

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides the end-to-end setup procedures for two distinct CloudWatch observability products: CloudWatch Application Observability (Omni) and classic CloudWatch Application Signals. It acts as a central routing hub, helping users identify the correct product based on their requirements and guiding them through the specific onboarding, instrumentation, and configuration steps for each. It covers resource creation, access management, and telemetry ingestion, while clearly delineating the boundaries between the two products to ensure correct implementation.

What it does

Routes setup requests to the correct product path based on user intent and environment.
Guides users through CloudWatch Omni Domain and Space creation, access grants, and profile configuration.
Provides instructions for instrumenting applications and AI agents with the ADOT SDK for Omni.
Details the onboarding process for classic CloudWatch Application Signals, including auto-instrumentation and add-ons.
Supports telemetry ingestion configuration for Azure and Google Cloud environments.
Manages integration setups for Slack and the AWS DevOps Agent.

When to use it

When you need to set up CloudWatch Application Observability (Omni) for the first time.
When you need to onboard a service to classic CloudWatch Application Signals.
When you are unsure which CloudWatch observability product is appropriate for your workload.
When you need to configure access grants, profiles, or integrations for an existing Space.
When you need to instrument an application or AI agent to export telemetry to CloudWatch.

Advantages

Prevents configuration conflicts by enforcing mutually exclusive setup paths for Omni and Application Signals.
Provides a structured, step-by-step sequence for complex resource hierarchies like Domains and Spaces.
Clarifies the distinction between setup tasks and day-to-day operational tasks.
Ensures correct principal resolution and permission scoping for access grants and profiles.

Usage examples

“How do I start setting up CloudWatch Application Observability?”: The agent will explain the setup order (Domain, Space, grants, telemetry) and direct you to the app-basics reference to begin.
“I need to onboard my EKS service to Application Signals.”: The agent will route you to the Application Signals onboarding guide and the specific EKS-language instrumentation steps.
“How do I give a team access to my CloudWatch Omni Space?”: The agent will guide you through resolving the principal and using the access-grants reference to configure the appropriate permission level.
“Can I connect my GitHub repository to CloudWatch Omni?”: The agent will clarify that Omni does not support GitHub integration and stop the setup process.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/aws/setting-up-cloudwatch-observability/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip setting-up-cloudwatch-observability.zip -d .agents/skills

Documentation by AWS

Setting Up CloudWatch Observability

Scope: First-time CloudWatch observability setup, for both products — a CloudWatch Application Observability (Omni) Space from creation through first traces flowing, and onboarding a service to classic CloudWatch Application Signals. For using what is already set up (queries, dashboards, alarms, Omni alerts, evaluations, live debugging), route to aws-observability.

Two products, two reference folders

This skill owns setup for two products that share the CloudWatch name but are separate services. The folder a reference lives in is the signal for which product it belongs to — the same convention aws-observability uses:

Folder	Product	What setup means there
`references/cloudwatch-omni/`	CloudWatch Omni (Application / Agent Observability)	Domain → Space → grants → telemetry in → plain ADOT SDK instrumentation. No add-on, no CloudWatch Agent `application_signals` config, no port 4316.
`references/cloudwatch/`	Classic CloudWatch	Onboarding a service to Application Signals: ADOT auto-instrumentation, the `amazon-cloudwatch-observability` EKS add-on, the CloudWatch Agent, monitored service, ServiceEvents, CI/CD git/deployment metadata, port 4316.

The two instrumentation paths are mutually exclusive on a given workload — the Omni path explicitly forbids the Application Signals env vars and the add-on, and vice versa. Decide the product first (Routing Rules 1–4 below), then stay inside that folder. Enabling one does not replace the other; an account may run both, on different services.

Works best with the AWS MCP server — enables running AWS CLI commands directly. All guidance also works with standard AWS CLI access (aws cloudwatchomni ... on the Omni path; aws eks, aws iam, and aws application-signals on the Application Signals path).

Concepts — Omni (`references/cloudwatch-omni/`)

These are Omni's resources. None of them exist on the Application Signals path — that one has no Domain, Space, grant, or Dataset; its unit is a monitored service, and access is plain IAM. Do not ask an Application Signals onboarding customer for a spaceId.

Term	What it is
Domain	The identity boundary. Carries the authorization provider (IAM or Identity Center) and owns the endpoint URL customers reach Omni through. One per account, or one shared across an AWS Organization.
Space	A workspace holding telemetry, in exactly one account and one Region. Created under a Domain. At most one per account per Region. Region rule: under an IAM-only Domain a Space may sit in a Region other than the Domain's; under an account-scoped Identity Center Domain the Space must be in the Domain's own Region; under an org-scoped Identity Center Domain it must be in a Region the instance is replicated to.
Access grant	Attaches a principal — person, group, IAM identity, or async workload — to one Space at a permission level. The only way anyone reaches data through a Space — it does not restrict the source CloudWatch log groups, which stay readable under their own IAM.
Access Profile	A named boundary for async workloads (alerts, integrations, agents) that act without a person in the loop. It is only a named container: `create-access-profile` takes a Space, a name, and a description and nothing else — no permission, action, or scope input. It does something only once two separate sets of grants exist (what the profile may do; which workloads may assume it) and a workload names it.
Dataset	What queries run against. Telemetry arrives through the CloudWatch OTLP endpoints, or by forwarding what is already in CloudWatch log groups.

Omni setup order from nothing: Domain → Space → grants → telemetry in → instrumentation. (The Application Signals equivalent is much shorter and has no prerequisite resources — add-on/agent, IAM, then the per-platform enablement change; see references/cloudwatch/application-signals-onboarding.md.) "Telemetry in" (a collector exporting to CloudWatch's OTLP endpoints, plus dataset forwarding for what is already in CloudWatch) is its own step, separate from instrumenting the workloads. Access Profiles are conditional, not a step in the sequence — only when async workloads (alerts, integrations, agents) are involved. Whenever you give this sequence, also say that instrumentation or forwarding started before a Space exists appears to succeed while delivering telemetry nowhere the customer can see — a customer who checks Omni first reads a working, empty Space as a failure. For the concept relationships and the full arc, see references/cloudwatch-omni/app-basics.md.

This is a routing skill. Classify the user's setup request and delegate to the correct reference. Omni references live under references/cloudwatch-omni/; classic-CloudWatch (Application Signals) references live under references/cloudwatch/.

User intent	Reference
Onboard a service to Application Signals (auto-instrumentation, the `amazon-cloudwatch-observability` EKS add-on, CloudWatch Agent IAM for Application Signals, monitored service, reporting telemetry to Application Signals, ServiceEvents, the two onboarding tiers)	`references/cloudwatch/application-signals-onboarding.md`
Propagate ServiceEvents git/deployment metadata through CI/CD (the 5 `OTEL_AWS_SERVICE_EVENTS_*` vars, per-provider patterns)	`references/cloudwatch/application-signals-cicd-metadata.md`
Per-platform × per-language Application Signals enablement steps once platform and language are known	The matching `references/cloudwatch/appsignals-guides/<platform>-<language>.md` (e.g. `references/cloudwatch/appsignals-guides/eks-python.md`)
Turn on Dynamic Instrumentation for a service at onboarding time (the `OTEL_AWS_DYNAMIC_INSTRUMENTATION_*` vars and their IAM)	`references/cloudwatch/application-signals-onboarding.md` (Step 5d). Using it to debug is aws-observability
Understand what Omni is, its concepts, or where to start	`references/cloudwatch-omni/app-basics.md`
Instrument an AI agent (ADOT, OpenInference, framework detection, trace verification), or deploy an agent to production and get traces flowing to CloudWatch — env vars per platform (AgentCore, Lambda, or other platforms such as ECS/EC2/EKS), routing spans to a custom trace log group, IAM permissions needed, how the export authenticates (AWS credentials, no API token), ADOT version requirements	`references/cloudwatch-omni/omni-agents-instrumentation/omni-agents-instrumentation.md` (§ Production deployment for the deploy case)
Per-framework OpenInference guide (LangChain, LangGraph, Strands, CrewAI, OpenAI Agents, Vercel AI) once the agent framework is known	`references/cloudwatch-omni/omni-agents-instrumentation/openinference-framework-guide.md`, then the matching `references/cloudwatch-omni/omni-agents-instrumentation/instrument-<framework>.md`
Instrument an application (ADOT SDK on EC2/ECS/EKS/Lambda — Python, Node.js, Java, .NET)	`references/cloudwatch-omni/instrumentation/instrumentation.md`
Emit a custom application or agent metric so it is queryable in Omni (why OTLP and not `PutMetricData`/EMF)	`references/cloudwatch-omni/instrumentation/instrumentation.md` (§ Custom metrics)
Create an account-scoped Space or Domain	`references/cloudwatch-omni/spaces-and-domains.md`
Whether a Space can be in a different Region from its Domain	`references/cloudwatch-omni/spaces-and-domains.md` (Prerequisites → Region rules)
The AgentCore evaluation role that `create-space` asks for — what it is, whether to create one	`references/cloudwatch-omni/spaces-and-domains.md` (Step 3 → Also resolve the AgentCore evaluation role)
Let Omni run Bedrock models for a Space (Agent Playground, evaluations run from Omni), or a Bedrock `AccessDeniedException` from those features	`references/cloudwatch-omni/spaces-and-domains.md` (Step 3 → Optional: let Omni invoke Bedrock models)
A Space that was created successfully but returns an authorization error when used	`references/cloudwatch-omni/spaces-and-domains.md` (Troubleshooting → If the Space was created but cannot be used) — a space access role trust-policy problem, not a grant problem
Create a Domain shared across an AWS Organization	`references/cloudwatch-omni/org-domains.md`
Configure access grants for people or IAM identities or alerts	`references/cloudwatch-omni/access-grants.md`
An `aws cloudwatchomni` command reporting that the service is not supported, or a request to repair the customer's CLI or SDK so Omni commands work	Not a setup task — a local client-version problem. STOP and route to aws-observability
A programmatic caller denied on a Space it already had working — a CI role, SDK client, CloudFormation role, coding agent, or a workload under an Access Profile	`references/cloudwatch-omni/access-grants.md` (Grants, IAM, and Access Profiles), and aws-observability for the API-side view
Bound an alert, integration, or agent with an Access Profile	`references/cloudwatch-omni/access-profiles.md`
Deploy an OTel Collector so an instrumented app has somewhere to export to (EC2/ECS/EKS) — it exports to CloudWatch's own per-signal OTLP endpoints	`references/cloudwatch-omni/instrumentation/collector.md`
An OTLP collector already running (a CloudWatch Agent with an `opentelemetry` section or OTLP on 4317/4318, or `otelcol-contrib`) whose telemetry does not arrive, in whole or for one signal such as traces	`references/cloudwatch-omni/instrumentation/collector.md` (Verify), then the platform collector guide it links for the troubleshooting table
Enable Transaction Search so traces reach a Space (spans land in `aws/spans` only once it is on — per account, per Region)	`references/cloudwatch-omni/instrumentation/collector.md` (Step 1)
Forward telemetry already in CloudWatch into the Dataset	`references/cloudwatch-omni/data-forwarding-and-centralization.md`
Send your application's own telemetry from Azure (the logs/metrics/traces your service emits, via the CloudWatch agent on an Azure VM/AKS)	`references/cloudwatch-omni/azure-ingestion/azure-ingestion.md` (intent triage), then `references/cloudwatch-omni/azure-ingestion/custom-telemetry.md` (the CloudWatch-agent-on-VM/AKS procedure)
Send your application's own telemetry from Google Cloud (the logs/metrics/traces your service emits, via the CloudWatch agent on a GCE instance/GKE cluster)	`references/cloudwatch-omni/gcp-ingestion/gcp-ingestion.md` (intent triage), then `references/cloudwatch-omni/gcp-ingestion/custom-telemetry.md` (the CloudWatch-agent-on-GCE/GKE procedure)
Connect Slack to a Space for the first time	`references/cloudwatch-omni/slack-integration.md`
Connect the AWS DevOps Agent to a Space (Settings → Integrations → AWS DevOps Agent), an empty agent-space picker, or Investigations reporting the AWS DevOps Agent is not connected	`references/cloudwatch-omni/devops-agent-integration.md`. Creating the DevOps Agent space itself and running or chatting with investigations belong to the `aws-agents-for-devsecops` plugin
Whether a custom MCP tool server can be registered (Omni has none)	`references/cloudwatch-omni/custom-mcp-integration.md`
Enable or configure CloudWatch Omni Intelligence for a Space — attach an optional customer-managed KMS key (`put-intelligence-configuration`), connect an intelligence integration (`AWS_INTEGRATION`, `AWS_CONFIG_SLREC`, `EXTERNAL_AGENT`), check current config, or troubleshoot enablement (404, AccessDenied, empty graph)	`references/cloudwatch-omni/intelligence-setup.md`

Routing Rules

If the user is asking what Omni is, what a Domain or Space or grant means, where to start, or the end-to-end order of steps from nothing — rather than asking to perform one setup step — route to references/cloudwatch-omni/app-basics.md and answer the sequence from its "Setup order" section (not from a single procedure file such as collector.md, which covers one step). It also carries the boundary against aws-observability. A question about one concept's rules — a Space's Region relative to its Domain, the roles create-space needs — belongs to the procedure file for that concept (spaces-and-domains.md), which the routing table names.
If the request is about instrumenting an AI agent project (adding OTel to a framework such as LangChain, LangGraph, Strands, CrewAI, OpenAI Agents, or Vercel AI), it is Omni — route to references/cloudwatch-omni/omni-agents-instrumentation/omni-agents-instrumentation.md. There is no Application Signals path for an agent framework.
If the request is about instrumenting a general application (a service on EC2, ECS, EKS, or Lambda in Python, Node.js, Java, or .NET — not an AI-agent framework), decide the product before opening any procedure file — the two paths are mutually exclusive on a workload:
- Names Application Signals, ServiceEvents, a "monitored service", the service map, the amazon-cloudwatch-observability add-on, the CloudWatch Agent's application_signals configuration, or port 4316 → the CloudWatch path: references/cloudwatch/application-signals-onboarding.md, then the matching references/cloudwatch/appsignals-guides/<platform>-<language>.md.
- Names Omni, Application Observability, a Space, a Domain, or a Dataset — or explicitly rules Application Signals out ("plain OTel", "no add-on", "no agent sidecar") → the Omni path: references/cloudwatch-omni/instrumentation/instrumentation.md.
- Naming the CloudWatch Agent alone decides nothing: it is also the Omni path's default collector. An agent with an opentelemetry section or OTLP receivers on 4317/4318 is the Omni collector, and its delivery problems belong to rule 6.
- Names neither → rule 4.
If an instrumentation, ADOT, or collector request names neither product (neither Omni or a Space, nor Application Signals, ServiceEvents, or the amazon-cloudwatch-observability add-on), probe for a Space in the target Region before choosing. list-spaces is account-global: the --region flag only selects the endpoint, and the response lists every Space in the account, each with its own region, so filter to the target Region rather than trusting a non-empty list: aws cloudwatchomni list-spaces --region <region> --query "items[?region=='<region>']". A Space in that Region → the Omni path, references/cloudwatch-omni/instrumentation/instrumentation.md. An empty filtered list → the customer has not adopted Omni in that Region, so Application Signals is the live product for them: references/cloudwatch/application-signals-onboarding.md. Either way the request stays in this skill — the probe picks the folder, not the skill. If the probe errors with an unknown service, that is the CLI model, not evidence Omni is absent — fall back to the customer's wording and ask only if still inconclusive.
If the request is about Space/Domain creation or configuration, route to the matching setup reference. An authorization error that appears the moment a just-created Space is used is part of this — it is a space access role trust-policy problem (create-space never verifies the role can be assumed), so route to references/cloudwatch-omni/spaces-and-domains.md → "If the Space was created but cannot be used", not to the access-grants reference.
If the user asks about getting telemetry to a destination — telemetry not yet reaching CloudWatch — deploy an OTel Collector on EC2/ECS/EKS so an instrumented workload has somewhere to send OTLP, and wire the app's OTEL_EXPORTER_OTLP_ENDPOINT to it: references/cloudwatch-omni/instrumentation/collector.md. The same file covers an OTLP collector that is already deployed but whose telemetry does not arrive, in whole or for one signal (its Verify section and the platform guide's troubleshooting table). The collector exports straight to CloudWatch's own per-signal OTLP endpoints. If instead the telemetry is already in CloudWatch log groups and needs forwarding into the Dataset, route to references/cloudwatch-omni/data-forwarding-and-centralization.md. Both the collector's OTLP export and CloudWatch's OTLP endpoints authenticate with SigV4 (AWS credentials), including OIDC-federated credentials for a workload outside AWS (see references/cloudwatch-omni/azure-ingestion/custom-telemetry.md or references/cloudwatch-omni/gcp-ingestion/custom-telemetry.md). A purely token-based ingestion path for a sender that cannot obtain AWS credentials at all is not covered by these skills; do not improvise one, and say so plainly.
If the request is about getting Azure or Google Cloud telemetry into CloudWatch, route by cloud — references/cloudwatch-omni/azure-ingestion/azure-ingestion.md for Azure, references/cloudwatch-omni/gcp-ingestion/gcp-ingestion.md for Google Cloud — and decide by intent. If the customer wants the telemetry their own application produces (the logs/metrics/traces from their code), that is supported via the CloudWatch agent on an Azure VM or AKS cluster, or on a GCE instance or GKE cluster — follow the reference. If they want telemetry their cloud's resources emit on their own (the equivalent of AWS VPC flow logs / Route 53 logs: Azure resource and platform logs, or Google Cloud Audit Logs), that is not supported — say so and do not attempt a setup. If the request names no cloud, ask which one before giving any steps.
If the user asks to connect, enable, or authorize Slack for a Space for the first time (including granting the operator permission to use it), route to references/cloudwatch-omni/slack-integration.md. Using Slack after it is connected (posting findings to a channel, mentioning the assistant, or searching Slack) happens in Slack and the console and is not covered by these skills; Slack as an alert notification target is covered by aws-observability's Omni alerts reference.
Application Signals, ServiceEvents, and Dynamic Instrumentation are CloudWatch features, and they split by setup versus use — not by skill. Enabling them on a service that does not have them yet is setup and belongs here, under references/cloudwatch/: Application Signals onboarding and ServiceEvents in references/cloudwatch/application-signals-onboarding.md (with the CI/CD metadata chain in references/cloudwatch/application-signals-cicd-metadata.md), and switching Dynamic Instrumentation on at instrumentation time in that same file's Step 5d. They must never appear in an Omni instrumentation change — for that they remain out of scope, and rule 3 is how you tell the two paths apart. Using them once the service reports — reading the service map, alarming on Application Signals metrics, or placing breakpoints and reading snapshots with Dynamic Instrumentation — is day-to-day work: STOP and route to aws-observability.
If the user already has a working Space, or a service already reporting to Application Signals, and is asking about queries/dashboards/alarms/alerts/investigation, STOP and route to aws-observability.
If the user asks to register their own MCP tool server, connect a custom MCP server, or set up MCP authentication: CloudWatch Omni does not let customers add a custom MCP server. There is no console screen for it and the integration type is not enabled for customer use, so the attempt is refused. Say so plainly and stop; do not describe a setup flow, and do not offer a substitute. See references/cloudwatch-omni/custom-mcp-integration.md.
If the user asks to connect GitHub, link a GitHub organization or repositories, or enrich the application map from source code: CloudWatch Omni has no GitHub integration. Say so plainly and stop — do not walk through a setup. Do not offer a substitute: the Application Signals GitHub Action and the CloudWatch Logs GitHub audit-log source are separate CloudWatch features, not a way to connect GitHub to Omni, and a custom MCP server is not a GitHub integration. A GitHub-related feature found in public documentation is not evidence of an Omni GitHub integration.
If the user asks to enable or configure CloudWatch Omni Intelligence — attach or remove a customer-managed KMS key for intelligence data, add an intelligence integration (AWS_INTEGRATION, AWS_CONFIG_SLREC, or EXTERNAL_AGENT), read the current intelligence configuration, or troubleshoot why it will not turn on (a 404 UnknownOperationException, an AccessDeniedException, or an empty context graph) — route to references/cloudwatch-omni/intelligence-setup.md. That reference is the source of truth for the KMS ARN rules (full key ARN only, kms:Decrypt/kms:GenerateDataKey, exactly one of --kms-key-arn/--remove-kms-key, and that revoking the key makes intelligence data unrecoverable), the per-integration extra IAM, the two AccessDenied causes, and why "not enabled" is a 404 rather than an AccessDenied. Reading or interpreting the intelligence data once it is on is day-to-day use: STOP and route to the aws-observability skill — triaging insights, observations, or anomalies goes to its intelligence.md reference, which covers the ListInsights / ListObservations / GetContextGraph operations and the 404-vs-403-vs-local error rule; a pure topology or blast-radius question goes to its context-graph.md reference. For Slack specifically, route to references/cloudwatch-omni/slack-integration.md; for a custom MCP server, route to references/cloudwatch-omni/custom-mcp-integration.md (not supported). Do not offer GitHub, PagerDuty, Teams, MODEL_PROVIDER, or CUSTOM_MCP as intelligence integration types.
If the request names neither product — "add observability to my API", "how should I begin monitoring my workload", with no mention of Omni, a Space, a Domain, or of a CloudWatch feature — do not assume either one. Which product a customer wants to set up is their intent, not something the account's current state tells you, so ask, framed product versus product: CloudWatch Omni (Application/Agent Observability — Domains, Spaces, grants, a Dataset) or CloudWatch (log groups, alarms, Log Insights, Application Signals). Both answers are served here, from different reference folders, so this is a question about which product to set up — not about which skill to use, and not a reason to hand the request to aws-observability. In the same message, explain the boundary so they can choose — it is setup versus use: creating or configuring a Domain, Space, grant, or Access Profile, getting telemetry flowing for the first time, instrumenting an application or agent for Omni, and onboarding a service to Application Signals are all first-time setup and belong here; querying telemetry, building dashboards, configuring alarms or Omni alerts, investigating a live problem, and debugging with Dynamic Instrumentation are day-to-day use and belong to aws-observability. Do not start walking through Domain/Space creation or Application Signals onboarding until they confirm which product; having asked, stop and wait.
If the user asks to connect the AWS DevOps Agent to a Space, or the connect card or Investigations shows it is not connected, route to references/cloudwatch-omni/devops-agent-integration.md. These skills cover only that Omni-side connection. Creating or configuring the DevOps Agent space, and running, following, or chatting with its investigations from a coding agent, belong to the agent's dedicated aws-agents-for-devsecops plugin; point the customer there rather than reconstructing that workflow.

Files

references/cloudwatch-omni/instrumentation/collector-eks.md

OTel Collector on EKS

Read collector.md first — it holds the destination choice, the config, and the endpoint limits. This guide covers the in-cluster deployment and, the part that trips people up, how the application pod finds the collector pod.

The app addresses the collector by Kubernetes DNS. Its shape depends on which collector you deploy: the primary CloudWatch Agent runs as a DaemonSet with hostNetwork: true in the amazon-cloudwatch namespace, while the backup otelcol-contrib runs as a Deployment fronted by a ClusterIP Service in its own namespace.

How the app pod reaches the collector pod

This is the same mechanism the CloudWatch Agent operator uses. With the amazon-cloudwatch-observability add-on installed, instrumented pods export to:

http://cloudwatch-agent.amazon-cloudwatch:4316

That is http://<service-name>.<namespace>:<port> — a Service DNS name, resolvable from any pod in the cluster because kube-dns is in every pod's resolver search path. Nothing about it is CloudWatch-specific except the names and the port.

Our equivalent, with our own Service and the standard OTLP/HTTP port:

http://otel-collector.opentelemetry.svc.cluster.local:4318

Two deliberate differences from the add-on's form:

4318, not 4316. 4316 is the CloudWatch Agent's Application Signals port. 4318 is the OTLP standard for HTTP.
The fully-qualified name. otel-collector.opentelemetry resolves too, via the search suffixes in /etc/resolv.conf. The FQDN skips the search list and resolves in one query — use it, but either works.

We deploy the Service ourselves rather than installing the add-on, because the add-on brings an operator and hardcodes CloudWatch Agent endpoints and Application Signals.

Step 1: Give the collector an AWS identity

The collector calls AWS; the app does not. Two ways, pick based on what the cluster already has:

IRSA (preferred; scopes permissions to this one workload). Requires an IAM OIDC provider on the cluster — check with aws eks describe-cluster --name <cluster> --query 'cluster.identity.oidc.issuer' and aws iam list-open-id-connect-providers. If there is no provider, creating one is a prerequisite, not part of this change.

If the cluster is built with the common terraform-aws-modules/eks module and enable_irsa = true, the OIDC provider already exists and the module exports what you need — use module.eks.oidc_provider_arn and module.eks.oidc_provider rather than declaring your own provider resource:

resource "aws_iam_role" "otel_collector" {
  # IAM role names are GLOBAL. Include cluster and region, or a second cluster with the same
  # name in another region collides on apply.
  name = "${var.cluster_name}-${var.region}-otel-collector"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Federated = module.eks.oidc_provider_arn }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          # CloudWatch Agent (primary). For the otelcol-contrib backup below this is instead
          # "system:serviceaccount:opentelemetry:otel-collector".
          "${module.eks.oidc_provider}:sub" = "system:serviceaccount:amazon-cloudwatch:cloudwatch-agent"
          "${module.eks.oidc_provider}:aud" = "sts.amazonaws.com"
        }
      }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "otel_collector" {
  role       = aws_iam_role.otel_collector.name
  policy_arn = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
}

CloudWatchAgentServerPolicy is the grant for the destination — CloudWatch's own per-signal OTLP endpoints.

The sub condition must match system:serviceaccount:<namespace>:<serviceaccount> exactly. A typo here produces AccessDenied on AssumeRoleWithWebIdentity at collector startup.

Node role (simpler, coarser — what the Application Signals Terraform guide does): attach CloudWatchAgentServerPolicy to the node group role. Every pod on the node inherits it. Say that trade-off out loud when recommending it.

Step 2: Deploy the collector

Primary: the CloudWatch Agent, installed standalone

Install the agent yourself rather than through the EKS add-on, which turns on Application Signals and Container Insights. Use the amazon-cloudwatch-observability Helm chart — the same chart the add-on wraps — configured down to the opentelemetry section.

Prefer the IaC form. Express the release as Terraform so this step stays a reviewable diff like every other platform in this guide set, and terraform plan is the review:

# Pin the provider. Without this, `terraform init` takes the newest major version and the required
# syntax changes with it -- v3 wants `kubernetes = { ... }`, v2 wanted a `kubernetes { ... }` block.
terraform {
  required_providers {
    helm = {
      source  = "hashicorp/helm"
      version = "~> 3.0"
    }
  }
}

# Helm provider v3 syntax: `kubernetes` and `exec` are ATTRIBUTES (`= {`), not blocks.
# On a stack pinned to helm v2, drop the `=` on both.
provider "helm" {
  kubernetes = {
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
    exec = {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
    }
  }
}

resource "helm_release" "cwagent" {
  name             = "cwagent"
  repository       = "https://aws-observability.github.io/helm-charts"
  chart            = "amazon-cloudwatch-observability"
  version          = "6.6.0"
  namespace        = "amazon-cloudwatch"
  create_namespace = true
  values           = [file("${path.module}/cwagent-values.yaml")]
}

This produces the same result as the CLI: identical resolved values, an identical AmazonCloudWatchAgent config, and no restart of the agent pods. It also leaves the out-of-band IRSA annotation alone (see below) — a helm_release in-place update does not clobber it. A helm_release replacement would, since that recreates the ServiceAccount: changing name or namespace, or a terraform destroy and re-apply, means re-annotating.

If a release already exists — for instance it was installed by the CLI first — import it rather than uninstalling, so the collector never goes away:

terraform import helm_release.cwagent amazon-cloudwatch/cwagent   # <namespace>/<name>

Expect the first plan after import to be non-empty: the provider does not read repository or values back off the release, and create_namespace reads back as false, so you get one in-place update. That is a helm upgrade and a revision bump, not a reinstall and not an outage. The second plan is clean.

One cosmetic difference worth knowing on a Helm 4 CLI host: taking a v4-created release over with the Terraform provider changes its apply method from server-side to client-side apply, because the provider embeds an older Helm SDK. It changed nothing about the rendered resources here.

Alternative: the Helm CLI. Use this when the stack has no Helm provider, or for a one-off. It is imperative, so there is no diff to present — write the values file for the user to review and hand them the commands rather than running them yourself:

helm repo add aws-observability https://aws-observability.github.io/helm-charts
helm install cwagent aws-observability/amazon-cloudwatch-observability \
  --namespace amazon-cloudwatch --create-namespace \
  --version 6.6.0 -f cwagent-values.yaml

Either way, one imperative step survives: the IRSA annotation cannot be set through the chart at all (see below), so it remains a kubectl annotate plus a pod restart, or EKS Pod Identity.

# values.yaml — validated against chart 6.6.0
clusterName: <cluster-name>     # REQUIRED by the chart; it will not render without this
region: <region>                # REQUIRED; agent.config.agent.region does NOT satisfy it
agent:
  config:
    agent:
      region: <region>
    opentelemetry:
      collect:
        otlp:
          grpc_endpoint: 0.0.0.0:4317
          http_endpoint: 0.0.0.0:4318
# These default to TRUE and deploy machinery unrelated to an OTLP collector. (`kubeStateMetrics`
# and `nodeExporter` also default TRUE, and the chart's `agents` list has a second entry,
# `cloudwatch-agent-cluster-scraper` -- all three render only when `otelContainerInsights` is on,
# so leaving that false keeps them dormant and they need no entry here.)
containerLogs:
  enabled: false                # fluent-bit DaemonSets shipping ALL container logs to CW Logs
dcgmExporter:
  enabled: false
neuronMonitor:
  enabled: false
otelContainerInsights:
  enabled: false               # already the chart default; set it to pin the behaviour
containerInsights:
  enabled: false
applicationSignals:
  enabled: false

The operator comes with this chart and cannot be turned off. There is no manager.enabled key — setting one is silently ignored. More importantly the agent ships as an AmazonCloudWatchAgent custom resource that only the operator reconciles into the DaemonSet and Services, so disabling it would leave nothing running. That is acceptable here: with applicationSignals.enabled: false the operator creates no Instrumentation CR and annotates nothing, and its admission webhooks are failurePolicy: Ignore, so a sick operator cannot wedge pod creation. If you need a genuinely operator-free collector, use a plain manifest (mount the JSON at /etc/cwagentconfig/cwagentconfig.json or pass it via CW_CONFIG_CONTENT, and give the ServiceAccount a ClusterRole allowing list/watch on replicasets — without it the agent floods its own log with RBAC denials).

The chart also renders two Windows agents you cannot switch off, and one of them carries Application Signals. cloudwatch-agent-windows and cloudwatch-agent-windows-container-insights are gated only on agent.enabled, and their configs are hardcoded in the chart — the Windows one contains logs.metrics_collected.application_signals and traces.traces_collected.application_signals, and the operator gives it a Service exposing 4316. No values setting suppresses them; there is no windows key at all. On a Linux-only cluster they are inert (nodeSelector: kubernetes.io/os=windows, 0 desired, no endpoints) and can be left alone — but expect to find 4316 in the namespace, and on a cluster with Windows nodes delete both CRs after install or Application Signals will run there.

The agent's Service does not exist at template time, because the operator creates it. Do not try to read it from helm template — the only Service there is the operator's webhook on 443. The name is deterministic from agents[].name; after install confirm with kubectl -n amazon-cloudwatch get svc. The app endpoint is:

http://cloudwatch-agent.amazon-cloudwatch.svc.cluster.local:4318

Note this is the same host the add-on exposes — the difference is the port: 4318 (OTLP), never the Application Signals port 4316.

The agent runs as a DaemonSet with hostNetwork: true, not the Deployment-plus-ClusterIP shape described for the backup below. 4317/4318 are therefore bound on every node IP — worth saying out loud to the customer, for the same reason the guide forbids a LoadBalancer.

IRSA cannot be set through the chart. The agent ServiceAccount template accepts no annotations (agent.serviceAccount exposes only name; the chart's roleArn value is AKS-only), so Step 1's annotation mechanism does not apply unchanged. Either annotate out of band — noting a later helm upgrade does not revert it, but a ServiceAccount recreation would —

kubectl -n amazon-cloudwatch annotate serviceaccount cloudwatch-agent \
  eks.amazonaws.com/role-arn=<role-arn> --overwrite
kubectl -n amazon-cloudwatch delete pod -l app.kubernetes.io/name=cloudwatch-agent

— or use EKS Pod Identity, which needs no annotation. Either way the trust policy's sub is system:serviceaccount:amazon-cloudwatch:cloudwatch-agent, not the opentelemetry:otel-collector value used by the backup path below.

Backup: upstream otelcol-contrib

apiVersion: v1
kind: Namespace
metadata:
  name: opentelemetry
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-collector
  namespace: opentelemetry
  annotations:
    # IRSA only; omit if using the node role
    eks.amazonaws.com/role-arn: arn:aws:iam::<account>:role/otel-collector-irsa
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: opentelemetry
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          # Both protocols, matching collector.md: the Java agent defaults to gRPC on 4317 while
          # Python/Node/.NET default to HTTP on 4318. Expose both ports on the Service below too.
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      batch:
        send_batch_size: 200
        timeout: 10s
    exporters:
      otlphttp/traces:
        traces_endpoint: https://xray.<region>.amazonaws.com/v1/traces
        compression: gzip
        auth:
          authenticator: sigv4auth/traces
      otlphttp/metrics:
        metrics_endpoint: https://monitoring.<region>.amazonaws.com/v1/metrics
        compression: gzip
        auth:
          authenticator: sigv4auth/metrics
    extensions:
      sigv4auth/traces:
        region: "<region>"
        service: "xray"
      sigv4auth/metrics:
        region: "<region>"
        service: "monitoring"
    service:
      extensions: [sigv4auth/traces, sigv4auth/metrics]
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlphttp/traces]
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlphttp/metrics]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: opentelemetry
spec:
  replicas: 1
  selector:
    matchLabels: { app: otel-collector }
  template:
    metadata:
      labels: { app: otel-collector }
    spec:
      serviceAccountName: otel-collector
      containers:
        - name: otel-collector
          # upstream CONTRIB (sigv4auth is a contrib component); pin a real version, never :latest
          image: otel/opentelemetry-collector-contrib:<version>
          args: ["--config=/conf/config.yaml"]
          ports:
            - containerPort: 4318
              name: otlp-http
          volumeMounts:
            - name: config
              mountPath: /conf
          resources:
            requests: { cpu: 100m, memory: 128Mi }
            limits:   { memory: 512Mi }
      volumes:
        - name: config
          configMap:
            name: otel-collector-config
---
apiVersion: v1
kind: Service
metadata:
  name: otel-collector
  namespace: opentelemetry
spec:
  type: ClusterIP
  selector:
    app: otel-collector
  ports:
    - name: otlp-http
      port: 4318
      targetPort: 4318
    - name: otlp-grpc
      port: 4317
      targetPort: 4317

type: ClusterIP — in-cluster only. Never LoadBalancer; that publishes an unauthenticated OTLP receiver to the internet.

The Service selector must match the pod template labels. When they disagree the Service exists, DNS resolves, and every connection is refused because the Service has no endpoints.

Step 3: Point the application at the collector

On the application container in its Deployment, alongside the loader variable and volumeMounts the eks-*.md instrumentation guides add:

env:
  # ... existing, including the language's loader variable ...
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    # CloudWatch Agent (primary). For the otelcol-contrib backup this is instead
    # http://otel-collector.opentelemetry.svc.cluster.local:4318
    value: "http://cloudwatch-agent.amazon-cloudwatch.svc.cluster.local:4318"
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: "http/protobuf"

A stale value here fails loudly and unmistakably — UnknownHostException on the collector name — so if you see that, check which of the two paths you actually deployed.

The Service DNS name is not node-local, despite the agent being a DaemonSet. A ClusterIP Service is load-balanced by kube-proxy across every ready endpoint, so an app pod may well send to the agent on a different node — and in a multi-AZ cluster, across an AZ boundary. That is correct and works; it is simply not node-local, and the DaemonSet does not make it so.

If node-local delivery matters — for cross-AZ data-transfer cost, or to keep the failure domain on one node — pick one of:

Target the node IP. Use the downward-API pattern from the DaemonSet alternative below (status.hostIP into NODE_IP, then http://$(NODE_IP):4318). This works with the primary agent as-is, because hostNetwork: true already binds 4318 on the node.
Keep the Service and add internalTrafficPolicy: Local, which tells kube-proxy to route only to endpoints on the calling node. Simpler to reason about, but traffic fails rather than falls back if the local agent pod is unhealthy.

Otherwise leave the Service name and drop the node-local expectation.

OTEL_EXPORTER_OTLP_PROTOCOL is not optional — see collector.md.

Alternative: DaemonSet

A DaemonSet keeps telemetry node-local, which avoids cross-AZ traffic charges and removes the single collector as a bottleneck. The app then addresses the node it is running on:

env:
  - name: NODE_IP
    valueFrom:
      fieldRef: { fieldPath: status.hostIP }
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://$(NODE_IP):4318"

This needs hostPort: 4318 on the DaemonSet container, and the $(NODE_IP) interpolation only works because NODE_IP is declared earlier in the same env list — Kubernetes substitutes in order. Start with the Deployment; move to a DaemonSet when volume or cost justifies it.

Verify

These inspect a running collector, so they belong to the conversation after the user applies.

Primary — the CloudWatch Agent. Note the namespace, workload kind and Service name all differ from the backup path below:

kubectl -n amazon-cloudwatch get pods,svc
kubectl -n amazon-cloudwatch get daemonset cloudwatch-agent

# THE receiver-listening check, and the EKS substitute for `ss -lntp` -- you cannot exec into the
# agent container to run that: the image is distroless (no ss, no sh, no printenv). This also carries
# per-signal export health. Read it from stdout; there is no log FILE in a container.
kubectl -n amazon-cloudwatch logs -l app.kubernetes.io/name=cloudwatch-agent \
  | grep -E 'Starting HTTP server|Starting GRPC server|Everything is ready|Exporting failed|dropped_items'

# NOT a listening check, despite appearances. Endpoints are populated from ready pods and the
# Service's port spec, so a too-old agent listening on nothing still shows 4317/4318 here. Run it
# only for the *selector* mismatch case: empty output means the Service selects no agent pods at all.
kubectl -n amazon-cloudwatch get endpoints cloudwatch-agent

# IRSA actually injected? Read it from the API server, not from inside the pod
kubectl -n amazon-cloudwatch get pod -l app.kubernetes.io/name=cloudwatch-agent \
  -o jsonpath='{range .items[0].spec.containers[0].env[*]}{.name}={.value}{"\n"}{end}' \
  | grep -E 'AWS_ROLE_ARN|AWS_WEB_IDENTITY_TOKEN_FILE'

# DNS and the receiver, from the app's own namespace
kubectl -n <app-ns> exec deploy/<app> -- \
  curl -sS -o /dev/null -w '%{http_code}\n' \
  http://cloudwatch-agent.amazon-cloudwatch.svc.cluster.local:4318/v1/traces   # 405 = listening

Two caveats on that last check: many slim app images have no curl (python:3.12-slim, mcr.microsoft.com/dotnet/aspnet:*), so prefer the agent-side listener log lines, or run a throwaway kubectl run --rm --image=curlimages/curl pod in the app's namespace. And kubectl get svc will show an otlp-grpc port on 4317 even when grpc_endpoint is not configured — only the agent log's Starting GRPC server line tells the truth.

Backup — otelcol-contrib:

kubectl -n opentelemetry get pods,svc
kubectl -n opentelemetry logs deploy/otel-collector

# Service actually has endpoints (empty = selector/label mismatch)
kubectl -n opentelemetry get endpoints otel-collector

# DNS and the receiver, from the app's own namespace
kubectl -n <app-ns> exec deploy/<app> -- \
  curl -sS -o /dev/null -w '%{http_code}\n' \
  http://otel-collector.opentelemetry.svc.cluster.local:4318/v1/traces   # 405 = listening

Symptom	Cause
`no such host` from the app	Wrong namespace in the DNS name, or Service not created
DNS resolves, connection refused	Service selector doesn't match pod labels — check `get endpoints`
Collector logs `AccessDenied` on `AssumeRoleWithWebIdentity`	Four causes, in order of likelihood: the ServiceAccount was never annotated (on the primary path the chart cannot do it — see Step 2, and remember the pod delete); the IRSA `sub` condition doesn't match — it is `system:serviceaccount:amazon-cloudwatch:cloudwatch-agent` on the primary path and `system:serviceaccount:opentelemetry:otel-collector` on the backup; the role was created seconds ago and IAM has not propagated (retry); or the manifest was applied to a different cluster, so the token is signed by another OIDC issuer — check `kubectl config current-context`
Collector logs `403` on export	Read the 403's `Message=` body first. A denial naming the principal and action means the role lacks `CloudWatchAgentServerPolicy`; one describing account enablement is not IAM and no policy change will fix it
Receiver reachable, nothing exported	`OTEL_EXPORTER_OTLP_PROTOCOL` unset, SDK defaulted to gRPC
Metrics arrive, spans do not	Transaction Search not enabled on the account
Spans arrive, metrics do not	The metrics endpoint rejected the publish. Read the 403 body — this is the inverse case and has nothing to do with Transaction Search

Completion

Tell the user what was added. On the primary path that is a Helm release in amazon-cloudwatch (which brings the operator, an AmazonCloudWatchAgent custom resource, the agent DaemonSet and its Services, plus two inert Windows agents), an IRSA role, and the out-of-band ServiceAccount annotation plus pod restart. On the backup path it is a namespace, ServiceAccount, ConfigMap, Deployment and Service, and the IAM role or node-role attachment. Either way: the app now exports to the collector's DNS name, an OIDC provider may still be a prerequisite they need, and no application source or image changed.

Present the values file plus whichever install form you used — the helm_release resource as part of the Terraform diff, or the helm commands handed over for the user to run. Either way the IRSA annotation and pod restart are commands for them, not for you.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an MCP server, Needs Python

Security

What our scanner noticed· Medium risk

References to credential files
Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

AWS

Publisher

AWS (Official)

Source

aws/agent-toolkit-for-aws · skills/specialized-skills/operations-skills/setting-up-cloudwatch-observability · commit 0a218f8

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
NOTICE (added by AutoGineer from NOTICE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by AWS or AWS.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download setting-up-cloudwatch-observability.zip