Setting Up Cloudwatch Observability

AWSOfficialPublished by AWS

Skill
Tool integration
Cloud and DevOps
Apache-2.0
Medium risk

From the author

Sets up CloudWatch observability for the first time - Omni (CloudWatch Application Observability) and classic CloudWatch. Omni: creating a Space or Domain; access grants (who has access, at what level), access profiles bounding alerts, integrations, agents; instrumenting an app or AI agent with the plain ADOT SDK so traces reach Omni (Python/Node/Java/.NET on EC2/ECS/EKS/Lambda), incl. no-image-rebuild, CoreCLR vars; ingesting Azure or Google Cloud telemetry via the CloudWatch agent (VM/AKS, GCE/GKE); connecting Slack or the AWS DevOps Agent to a Space; whether GitHub or a custom MCP tool server (HTTP/stdio) can connect. CloudWatch: onboarding a service to Application Signals: ADOT auto-instrumentation, amazon-cloudwatch-observability add-on, monitored service, ServiceEvents, CI/CD metadata, Terraform. For using what is set up - queries, dashboards, alarms, Omni alerts, X-Ray, synthetics, Dynamic Instrumentation, AI agent evaluation, detecting Omni vs classic - use aws-observability.

— the description by AWS

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides the end-to-end setup procedures for two distinct CloudWatch observability products: CloudWatch Application Observability (Omni) and classic CloudWatch Application Signals. It acts as a central routing hub, helping users identify the correct product based on their requirements and guiding them through the specific onboarding, instrumentation, and configuration steps for each. It covers resource creation, access management, and telemetry ingestion, while clearly delineating the boundaries between the two products to ensure correct implementation.

What it does

Routes setup requests to the correct product path based on user intent and environment.
Guides users through CloudWatch Omni Domain and Space creation, access grants, and profile configuration.
Provides instructions for instrumenting applications and AI agents with the ADOT SDK for Omni.
Details the onboarding process for classic CloudWatch Application Signals, including auto-instrumentation and add-ons.
Supports telemetry ingestion configuration for Azure and Google Cloud environments.
Manages integration setups for Slack and the AWS DevOps Agent.

When to use it

When you need to set up CloudWatch Application Observability (Omni) for the first time.
When you need to onboard a service to classic CloudWatch Application Signals.
When you are unsure which CloudWatch observability product is appropriate for your workload.
When you need to configure access grants, profiles, or integrations for an existing Space.
When you need to instrument an application or AI agent to export telemetry to CloudWatch.

Advantages

Prevents configuration conflicts by enforcing mutually exclusive setup paths for Omni and Application Signals.
Provides a structured, step-by-step sequence for complex resource hierarchies like Domains and Spaces.
Clarifies the distinction between setup tasks and day-to-day operational tasks.
Ensures correct principal resolution and permission scoping for access grants and profiles.

Usage examples

“How do I start setting up CloudWatch Application Observability?”: The agent will explain the setup order (Domain, Space, grants, telemetry) and direct you to the app-basics reference to begin.
“I need to onboard my EKS service to Application Signals.”: The agent will route you to the Application Signals onboarding guide and the specific EKS-language instrumentation steps.
“How do I give a team access to my CloudWatch Omni Space?”: The agent will guide you through resolving the principal and using the access-grants reference to configure the appropriate permission level.
“Can I connect my GitHub repository to CloudWatch Omni?”: The agent will clarify that Omni does not support GitHub integration and stop the setup process.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/aws/setting-up-cloudwatch-observability/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip setting-up-cloudwatch-observability.zip -d .agents/skills

Documentation by AWS

Setting Up CloudWatch Observability

Scope: First-time CloudWatch observability setup, for both products — a CloudWatch Application Observability (Omni) Space from creation through first traces flowing, and onboarding a service to classic CloudWatch Application Signals. For using what is already set up (queries, dashboards, alarms, Omni alerts, evaluations, live debugging), route to aws-observability.

Two products, two reference folders

This skill owns setup for two products that share the CloudWatch name but are separate services. The folder a reference lives in is the signal for which product it belongs to — the same convention aws-observability uses:

Folder	Product	What setup means there
`references/cloudwatch-omni/`	CloudWatch Omni (Application / Agent Observability)	Domain → Space → grants → telemetry in → plain ADOT SDK instrumentation. No add-on, no CloudWatch Agent `application_signals` config, no port 4316.
`references/cloudwatch/`	Classic CloudWatch	Onboarding a service to Application Signals: ADOT auto-instrumentation, the `amazon-cloudwatch-observability` EKS add-on, the CloudWatch Agent, monitored service, ServiceEvents, CI/CD git/deployment metadata, port 4316.

The two instrumentation paths are mutually exclusive on a given workload — the Omni path explicitly forbids the Application Signals env vars and the add-on, and vice versa. Decide the product first (Routing Rules 1–4 below), then stay inside that folder. Enabling one does not replace the other; an account may run both, on different services.

Works best with the AWS MCP server — enables running AWS CLI commands directly. All guidance also works with standard AWS CLI access (aws cloudwatchomni ... on the Omni path; aws eks, aws iam, and aws application-signals on the Application Signals path).

Concepts — Omni (`references/cloudwatch-omni/`)

These are Omni's resources. None of them exist on the Application Signals path — that one has no Domain, Space, grant, or Dataset; its unit is a monitored service, and access is plain IAM. Do not ask an Application Signals onboarding customer for a spaceId.

Term	What it is
Domain	The identity boundary. Carries the authorization provider (IAM or Identity Center) and owns the endpoint URL customers reach Omni through. One per account, or one shared across an AWS Organization.
Space	A workspace holding telemetry, in exactly one account and one Region. Created under a Domain. At most one per account per Region. Region rule: under an IAM-only Domain a Space may sit in a Region other than the Domain's; under an account-scoped Identity Center Domain the Space must be in the Domain's own Region; under an org-scoped Identity Center Domain it must be in a Region the instance is replicated to.
Access grant	Attaches a principal — person, group, IAM identity, or async workload — to one Space at a permission level. The only way anyone reaches data through a Space — it does not restrict the source CloudWatch log groups, which stay readable under their own IAM.
Access Profile	A named boundary for async workloads (alerts, integrations, agents) that act without a person in the loop. It is only a named container: `create-access-profile` takes a Space, a name, and a description and nothing else — no permission, action, or scope input. It does something only once two separate sets of grants exist (what the profile may do; which workloads may assume it) and a workload names it.
Dataset	What queries run against. Telemetry arrives through the CloudWatch OTLP endpoints, or by forwarding what is already in CloudWatch log groups.

Omni setup order from nothing: Domain → Space → grants → telemetry in → instrumentation. (The Application Signals equivalent is much shorter and has no prerequisite resources — add-on/agent, IAM, then the per-platform enablement change; see references/cloudwatch/application-signals-onboarding.md.) "Telemetry in" (a collector exporting to CloudWatch's OTLP endpoints, plus dataset forwarding for what is already in CloudWatch) is its own step, separate from instrumenting the workloads. Access Profiles are conditional, not a step in the sequence — only when async workloads (alerts, integrations, agents) are involved. Whenever you give this sequence, also say that instrumentation or forwarding started before a Space exists appears to succeed while delivering telemetry nowhere the customer can see — a customer who checks Omni first reads a working, empty Space as a failure. For the concept relationships and the full arc, see references/cloudwatch-omni/app-basics.md.

This is a routing skill. Classify the user's setup request and delegate to the correct reference. Omni references live under references/cloudwatch-omni/; classic-CloudWatch (Application Signals) references live under references/cloudwatch/.

User intent	Reference
Onboard a service to Application Signals (auto-instrumentation, the `amazon-cloudwatch-observability` EKS add-on, CloudWatch Agent IAM for Application Signals, monitored service, reporting telemetry to Application Signals, ServiceEvents, the two onboarding tiers)	`references/cloudwatch/application-signals-onboarding.md`
Propagate ServiceEvents git/deployment metadata through CI/CD (the 5 `OTEL_AWS_SERVICE_EVENTS_*` vars, per-provider patterns)	`references/cloudwatch/application-signals-cicd-metadata.md`
Per-platform × per-language Application Signals enablement steps once platform and language are known	The matching `references/cloudwatch/appsignals-guides/<platform>-<language>.md` (e.g. `references/cloudwatch/appsignals-guides/eks-python.md`)
Turn on Dynamic Instrumentation for a service at onboarding time (the `OTEL_AWS_DYNAMIC_INSTRUMENTATION_*` vars and their IAM)	`references/cloudwatch/application-signals-onboarding.md` (Step 5d). Using it to debug is aws-observability
Understand what Omni is, its concepts, or where to start	`references/cloudwatch-omni/app-basics.md`
Instrument an AI agent (ADOT, OpenInference, framework detection, trace verification), or deploy an agent to production and get traces flowing to CloudWatch — env vars per platform (AgentCore, Lambda, or other platforms such as ECS/EC2/EKS), routing spans to a custom trace log group, IAM permissions needed, how the export authenticates (AWS credentials, no API token), ADOT version requirements	`references/cloudwatch-omni/omni-agents-instrumentation/omni-agents-instrumentation.md` (§ Production deployment for the deploy case)
Per-framework OpenInference guide (LangChain, LangGraph, Strands, CrewAI, OpenAI Agents, Vercel AI) once the agent framework is known	`references/cloudwatch-omni/omni-agents-instrumentation/openinference-framework-guide.md`, then the matching `references/cloudwatch-omni/omni-agents-instrumentation/instrument-<framework>.md`
Instrument an application (ADOT SDK on EC2/ECS/EKS/Lambda — Python, Node.js, Java, .NET)	`references/cloudwatch-omni/instrumentation/instrumentation.md`
Emit a custom application or agent metric so it is queryable in Omni (why OTLP and not `PutMetricData`/EMF)	`references/cloudwatch-omni/instrumentation/instrumentation.md` (§ Custom metrics)
Create an account-scoped Space or Domain	`references/cloudwatch-omni/spaces-and-domains.md`
Whether a Space can be in a different Region from its Domain	`references/cloudwatch-omni/spaces-and-domains.md` (Prerequisites → Region rules)
The AgentCore evaluation role that `create-space` asks for — what it is, whether to create one	`references/cloudwatch-omni/spaces-and-domains.md` (Step 3 → Also resolve the AgentCore evaluation role)
Let Omni run Bedrock models for a Space (Agent Playground, evaluations run from Omni), or a Bedrock `AccessDeniedException` from those features	`references/cloudwatch-omni/spaces-and-domains.md` (Step 3 → Optional: let Omni invoke Bedrock models)
A Space that was created successfully but returns an authorization error when used	`references/cloudwatch-omni/spaces-and-domains.md` (Troubleshooting → If the Space was created but cannot be used) — a space access role trust-policy problem, not a grant problem
Create a Domain shared across an AWS Organization	`references/cloudwatch-omni/org-domains.md`
Configure access grants for people or IAM identities or alerts	`references/cloudwatch-omni/access-grants.md`
An `aws cloudwatchomni` command reporting that the service is not supported, or a request to repair the customer's CLI or SDK so Omni commands work	Not a setup task — a local client-version problem. STOP and route to aws-observability
A programmatic caller denied on a Space it already had working — a CI role, SDK client, CloudFormation role, coding agent, or a workload under an Access Profile	`references/cloudwatch-omni/access-grants.md` (Grants, IAM, and Access Profiles), and aws-observability for the API-side view
Bound an alert, integration, or agent with an Access Profile	`references/cloudwatch-omni/access-profiles.md`
Deploy an OTel Collector so an instrumented app has somewhere to export to (EC2/ECS/EKS) — it exports to CloudWatch's own per-signal OTLP endpoints	`references/cloudwatch-omni/instrumentation/collector.md`
An OTLP collector already running (a CloudWatch Agent with an `opentelemetry` section or OTLP on 4317/4318, or `otelcol-contrib`) whose telemetry does not arrive, in whole or for one signal such as traces	`references/cloudwatch-omni/instrumentation/collector.md` (Verify), then the platform collector guide it links for the troubleshooting table
Enable Transaction Search so traces reach a Space (spans land in `aws/spans` only once it is on — per account, per Region)	`references/cloudwatch-omni/instrumentation/collector.md` (Step 1)
Forward telemetry already in CloudWatch into the Dataset	`references/cloudwatch-omni/data-forwarding-and-centralization.md`
Send your application's own telemetry from Azure (the logs/metrics/traces your service emits, via the CloudWatch agent on an Azure VM/AKS)	`references/cloudwatch-omni/azure-ingestion/azure-ingestion.md` (intent triage), then `references/cloudwatch-omni/azure-ingestion/custom-telemetry.md` (the CloudWatch-agent-on-VM/AKS procedure)
Send your application's own telemetry from Google Cloud (the logs/metrics/traces your service emits, via the CloudWatch agent on a GCE instance/GKE cluster)	`references/cloudwatch-omni/gcp-ingestion/gcp-ingestion.md` (intent triage), then `references/cloudwatch-omni/gcp-ingestion/custom-telemetry.md` (the CloudWatch-agent-on-GCE/GKE procedure)
Connect Slack to a Space for the first time	`references/cloudwatch-omni/slack-integration.md`
Connect the AWS DevOps Agent to a Space (Settings → Integrations → AWS DevOps Agent), an empty agent-space picker, or Investigations reporting the AWS DevOps Agent is not connected	`references/cloudwatch-omni/devops-agent-integration.md`. Creating the DevOps Agent space itself and running or chatting with investigations belong to the `aws-agents-for-devsecops` plugin
Whether a custom MCP tool server can be registered (Omni has none)	`references/cloudwatch-omni/custom-mcp-integration.md`
Enable or configure CloudWatch Omni Intelligence for a Space — attach an optional customer-managed KMS key (`put-intelligence-configuration`), connect an intelligence integration (`AWS_INTEGRATION`, `AWS_CONFIG_SLREC`, `EXTERNAL_AGENT`), check current config, or troubleshoot enablement (404, AccessDenied, empty graph)	`references/cloudwatch-omni/intelligence-setup.md`

Routing Rules

If the user is asking what Omni is, what a Domain or Space or grant means, where to start, or the end-to-end order of steps from nothing — rather than asking to perform one setup step — route to references/cloudwatch-omni/app-basics.md and answer the sequence from its "Setup order" section (not from a single procedure file such as collector.md, which covers one step). It also carries the boundary against aws-observability. A question about one concept's rules — a Space's Region relative to its Domain, the roles create-space needs — belongs to the procedure file for that concept (spaces-and-domains.md), which the routing table names.
If the request is about instrumenting an AI agent project (adding OTel to a framework such as LangChain, LangGraph, Strands, CrewAI, OpenAI Agents, or Vercel AI), it is Omni — route to references/cloudwatch-omni/omni-agents-instrumentation/omni-agents-instrumentation.md. There is no Application Signals path for an agent framework.
If the request is about instrumenting a general application (a service on EC2, ECS, EKS, or Lambda in Python, Node.js, Java, or .NET — not an AI-agent framework), decide the product before opening any procedure file — the two paths are mutually exclusive on a workload:
- Names Application Signals, ServiceEvents, a "monitored service", the service map, the amazon-cloudwatch-observability add-on, the CloudWatch Agent's application_signals configuration, or port 4316 → the CloudWatch path: references/cloudwatch/application-signals-onboarding.md, then the matching references/cloudwatch/appsignals-guides/<platform>-<language>.md.
- Names Omni, Application Observability, a Space, a Domain, or a Dataset — or explicitly rules Application Signals out ("plain OTel", "no add-on", "no agent sidecar") → the Omni path: references/cloudwatch-omni/instrumentation/instrumentation.md.
- Naming the CloudWatch Agent alone decides nothing: it is also the Omni path's default collector. An agent with an opentelemetry section or OTLP receivers on 4317/4318 is the Omni collector, and its delivery problems belong to rule 6.
- Names neither → rule 4.
If an instrumentation, ADOT, or collector request names neither product (neither Omni or a Space, nor Application Signals, ServiceEvents, or the amazon-cloudwatch-observability add-on), probe for a Space in the target Region before choosing. list-spaces is account-global: the --region flag only selects the endpoint, and the response lists every Space in the account, each with its own region, so filter to the target Region rather than trusting a non-empty list: aws cloudwatchomni list-spaces --region <region> --query "items[?region=='<region>']". A Space in that Region → the Omni path, references/cloudwatch-omni/instrumentation/instrumentation.md. An empty filtered list → the customer has not adopted Omni in that Region, so Application Signals is the live product for them: references/cloudwatch/application-signals-onboarding.md. Either way the request stays in this skill — the probe picks the folder, not the skill. If the probe errors with an unknown service, that is the CLI model, not evidence Omni is absent — fall back to the customer's wording and ask only if still inconclusive.
If the request is about Space/Domain creation or configuration, route to the matching setup reference. An authorization error that appears the moment a just-created Space is used is part of this — it is a space access role trust-policy problem (create-space never verifies the role can be assumed), so route to references/cloudwatch-omni/spaces-and-domains.md → "If the Space was created but cannot be used", not to the access-grants reference.
If the user asks about getting telemetry to a destination — telemetry not yet reaching CloudWatch — deploy an OTel Collector on EC2/ECS/EKS so an instrumented workload has somewhere to send OTLP, and wire the app's OTEL_EXPORTER_OTLP_ENDPOINT to it: references/cloudwatch-omni/instrumentation/collector.md. The same file covers an OTLP collector that is already deployed but whose telemetry does not arrive, in whole or for one signal (its Verify section and the platform guide's troubleshooting table). The collector exports straight to CloudWatch's own per-signal OTLP endpoints. If instead the telemetry is already in CloudWatch log groups and needs forwarding into the Dataset, route to references/cloudwatch-omni/data-forwarding-and-centralization.md. Both the collector's OTLP export and CloudWatch's OTLP endpoints authenticate with SigV4 (AWS credentials), including OIDC-federated credentials for a workload outside AWS (see references/cloudwatch-omni/azure-ingestion/custom-telemetry.md or references/cloudwatch-omni/gcp-ingestion/custom-telemetry.md). A purely token-based ingestion path for a sender that cannot obtain AWS credentials at all is not covered by these skills; do not improvise one, and say so plainly.
If the request is about getting Azure or Google Cloud telemetry into CloudWatch, route by cloud — references/cloudwatch-omni/azure-ingestion/azure-ingestion.md for Azure, references/cloudwatch-omni/gcp-ingestion/gcp-ingestion.md for Google Cloud — and decide by intent. If the customer wants the telemetry their own application produces (the logs/metrics/traces from their code), that is supported via the CloudWatch agent on an Azure VM or AKS cluster, or on a GCE instance or GKE cluster — follow the reference. If they want telemetry their cloud's resources emit on their own (the equivalent of AWS VPC flow logs / Route 53 logs: Azure resource and platform logs, or Google Cloud Audit Logs), that is not supported — say so and do not attempt a setup. If the request names no cloud, ask which one before giving any steps.
If the user asks to connect, enable, or authorize Slack for a Space for the first time (including granting the operator permission to use it), route to references/cloudwatch-omni/slack-integration.md. Using Slack after it is connected (posting findings to a channel, mentioning the assistant, or searching Slack) happens in Slack and the console and is not covered by these skills; Slack as an alert notification target is covered by aws-observability's Omni alerts reference.
Application Signals, ServiceEvents, and Dynamic Instrumentation are CloudWatch features, and they split by setup versus use — not by skill. Enabling them on a service that does not have them yet is setup and belongs here, under references/cloudwatch/: Application Signals onboarding and ServiceEvents in references/cloudwatch/application-signals-onboarding.md (with the CI/CD metadata chain in references/cloudwatch/application-signals-cicd-metadata.md), and switching Dynamic Instrumentation on at instrumentation time in that same file's Step 5d. They must never appear in an Omni instrumentation change — for that they remain out of scope, and rule 3 is how you tell the two paths apart. Using them once the service reports — reading the service map, alarming on Application Signals metrics, or placing breakpoints and reading snapshots with Dynamic Instrumentation — is day-to-day work: STOP and route to aws-observability.
If the user already has a working Space, or a service already reporting to Application Signals, and is asking about queries/dashboards/alarms/alerts/investigation, STOP and route to aws-observability.
If the user asks to register their own MCP tool server, connect a custom MCP server, or set up MCP authentication: CloudWatch Omni does not let customers add a custom MCP server. There is no console screen for it and the integration type is not enabled for customer use, so the attempt is refused. Say so plainly and stop; do not describe a setup flow, and do not offer a substitute. See references/cloudwatch-omni/custom-mcp-integration.md.
If the user asks to connect GitHub, link a GitHub organization or repositories, or enrich the application map from source code: CloudWatch Omni has no GitHub integration. Say so plainly and stop — do not walk through a setup. Do not offer a substitute: the Application Signals GitHub Action and the CloudWatch Logs GitHub audit-log source are separate CloudWatch features, not a way to connect GitHub to Omni, and a custom MCP server is not a GitHub integration. A GitHub-related feature found in public documentation is not evidence of an Omni GitHub integration.
If the user asks to enable or configure CloudWatch Omni Intelligence — attach or remove a customer-managed KMS key for intelligence data, add an intelligence integration (AWS_INTEGRATION, AWS_CONFIG_SLREC, or EXTERNAL_AGENT), read the current intelligence configuration, or troubleshoot why it will not turn on (a 404 UnknownOperationException, an AccessDeniedException, or an empty context graph) — route to references/cloudwatch-omni/intelligence-setup.md. That reference is the source of truth for the KMS ARN rules (full key ARN only, kms:Decrypt/kms:GenerateDataKey, exactly one of --kms-key-arn/--remove-kms-key, and that revoking the key makes intelligence data unrecoverable), the per-integration extra IAM, the two AccessDenied causes, and why "not enabled" is a 404 rather than an AccessDenied. Reading or interpreting the intelligence data once it is on is day-to-day use: STOP and route to the aws-observability skill — triaging insights, observations, or anomalies goes to its intelligence.md reference, which covers the ListInsights / ListObservations / GetContextGraph operations and the 404-vs-403-vs-local error rule; a pure topology or blast-radius question goes to its context-graph.md reference. For Slack specifically, route to references/cloudwatch-omni/slack-integration.md; for a custom MCP server, route to references/cloudwatch-omni/custom-mcp-integration.md (not supported). Do not offer GitHub, PagerDuty, Teams, MODEL_PROVIDER, or CUSTOM_MCP as intelligence integration types.
If the request names neither product — "add observability to my API", "how should I begin monitoring my workload", with no mention of Omni, a Space, a Domain, or of a CloudWatch feature — do not assume either one. Which product a customer wants to set up is their intent, not something the account's current state tells you, so ask, framed product versus product: CloudWatch Omni (Application/Agent Observability — Domains, Spaces, grants, a Dataset) or CloudWatch (log groups, alarms, Log Insights, Application Signals). Both answers are served here, from different reference folders, so this is a question about which product to set up — not about which skill to use, and not a reason to hand the request to aws-observability. In the same message, explain the boundary so they can choose — it is setup versus use: creating or configuring a Domain, Space, grant, or Access Profile, getting telemetry flowing for the first time, instrumenting an application or agent for Omni, and onboarding a service to Application Signals are all first-time setup and belong here; querying telemetry, building dashboards, configuring alarms or Omni alerts, investigating a live problem, and debugging with Dynamic Instrumentation are day-to-day use and belong to aws-observability. Do not start walking through Domain/Space creation or Application Signals onboarding until they confirm which product; having asked, stop and wait.
If the user asks to connect the AWS DevOps Agent to a Space, or the connect card or Investigations shows it is not connected, route to references/cloudwatch-omni/devops-agent-integration.md. These skills cover only that Omni-side connection. Creating or configuring the DevOps Agent space, and running, following, or chatting with its investigations from a coding agent, belong to the agent's dedicated aws-agents-for-devsecops plugin; point the customer there rather than reconstructing that workflow.

Files

references/cloudwatch/application-signals-onboarding.md

Application Signals Onboarding (Enable Auto-Instrumentation via ADOT)

Enable AWS Application Signals for a service that is not yet instrumented, by using ADOT (AWS Distro for OpenTelemetry) auto-instrumentation SDKs and making minimal, reviewable changes to the customer's infrastructure-as-code, Dockerfiles, CI/CD workflows, and deployment manifests. This is the enablement side of observability (turning an un-instrumented service into one that reports to Application Signals via ADOT).

This is the CloudWatch path, not the Omni one. Application Signals is a CloudWatch feature: the amazon-cloudwatch-observability add-on, the CloudWatch Agent, ServiceEvents, port 4316, and a monitored service in the service map. The Omni path — plain ADOT SDK exporting toward a Space, with none of that wiring — is references/cloudwatch-omni/instrumentation/instrumentation.md. Pick by what the customer named; if they named neither, probe for a Space in the target Region per SKILL.md's routing rules.

Once the service reports, querying it, alarming on it, reading its service map, and debugging it live with Dynamic Instrumentation are day-to-day use and belong to the aws-observability skill.

Never modify application source code (.py, .js, .ts, .java, .cs). Only edit IaC, Dockerfiles, CI/CD workflows, dependency files, and deployment manifests. Make the minimum changes needed and preserve existing configuration. Present changes for the user to review; do not run terraform apply, cdk deploy, or kubectl apply automatically.

Scope: two tiers

Onboarding has two tiers. Apply the second only when it is supported for the platform + language.

Tier	What it adds	Supported on
1. Application Signals enablement (always)	ADOT auto-instrumentation: CloudWatch Observability add-on (EKS), CloudWatch Agent, IAM, the inject annotation / init container / SDK install	All platforms (EC2, ECS, EKS, Lambda) and all languages (Python, Node.js, Java, .NET)
2. ServiceEvents extras (when supported)	Git & deployment metadata env vars (CI/CD propagation) + OTLP endpoints + Dynamic Instrumentation	EC2, ECS, EKS with Python, Node.js, Java only

Minimum component versions for ServiceEvents (Tier 2). The base Application Signals (Tier 1) works on any recent version. ServiceEvents requires:

Component	Minimum for ServiceEvents	Notes	Latest version links
CloudWatch Agent	`1.300070.0` (recommended — includes on-prem credential bugfix) or `1.300069.0`	Use latest by default; flag to the user if they are on an older version	—
CloudWatch Observability EKS add-on	`v6.3.0`	Use latest by default; flag if the customer's IaC pins an older version	—
ADOT Python SDK / ECS init container	`0.18.0`	pip: `aws-opentelemetry-distro==0.18.0`; ECR: `adot-autoinstrumentation-python:v0.18.0`	releases · ECR
ADOT Node.js SDK / ECS init container	`0.12.0`	npm: `@aws/[email protected]`; ECR: `adot-autoinstrumentation-node:v0.12.0`	releases · ECR
ADOT Java agent / ECS init container	`2.28.2`	jar: `aws-opentelemetry-agent-2.28.2.jar`; ECR: `adot-autoinstrumentation-java:v2.28.2`	releases · ECR
ADOT .NET / ECS init container	ServiceEvents not supported on .NET		releases · ECR

Tier 2 is NOT supported on Lambda or .NET. For a Lambda service, or a .NET service on any platform, do Tier 1 only — the service still gets Application Signals, just without the ServiceEvents metadata/OTLP/DI env vars. Do not add OTEL_AWS_SERVICE_EVENTS_*, OTEL_AWS_OTLP_*, or OTEL_AWS_DYNAMIC_INSTRUMENTATION_* env vars for Lambda or .NET.

Step 1: Determine platform and language

Detect from the IaC and app code, and confirm with the user if ambiguous:

EKS: k8s Deployment manifests (kind: Deployment), Helm charts, kubectl in scripts, Terraform aws_eks_*, the amazon-cloudwatch-observability add-on.
ECS: ECS task definitions, containerDefinitions, Terraform aws_ecs_*.
Lambda: Lambda function definitions, SAM templates, Terraform aws_lambda_function.
EC2: EC2 instances, userdata scripts, launch templates, Terraform aws_instance.
Language: requirements.txt/pyproject.toml/*.py → Python; package.json/*.ts/*.js → Node.js (nodejs); pom.xml/build.gradle/*.java → Java; *.csproj/*.sln/*.cs → .NET (dotnet).

Step 2 (EKS only): Install or import the CloudWatch Observability add-on

The amazon-cloudwatch-observability add-on injects ADOT auto-instrumentation via init containers and runs the CloudWatch Agent.

Prefer the EKS add-on (aws_eks_addon / CfnAddon) — do NOT introduce helm_release to replace an existing add-on (the add-on provides functionality the Helm chart alone does not, e.g. automatic OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT injection). If the user's IaC already uses helm_release for this chart, work with their existing setup.

Check whether the add-on is already enabled. Present the user with these options and proceed based on their response:

You run it — offer to run the AWS CLI command yourself (requires CLI/credentials access and the cluster name + region from the IaC):
```
aws eks describe-addon --cluster-name <cluster-name> --addon-name amazon-cloudwatch-observability --region <region>
```
A successful response means it exists; ResourceNotFoundException means it does not.
User runs it — ask the user to run the command above themselves or check the EKS console → Add-ons tab, and share the result.
User says it's not enabled — proceed to add the add-on (see below).
User says it's already enabled — proceed to the import step (see "Add-on already exists" below).

Add-on does NOT exist: add the aws_eks_addon / CfnAddon resource:

resource "aws_eks_addon" "cloudwatch_observability" {
  cluster_name = ...
  addon_name   = "amazon-cloudwatch-observability"
  # addon_version omitted = uses the latest default version (recommended).
  # ServiceEvents requires v6.3.0+. If the customer's IaC pins an older version, flag it.
}

Add-on already exists (Terraform): still add the resource above, and add a terraform import step to the CI/CD workflow before terraform apply so apply uses UpdateAddon instead of CreateAddon. Use || true so reruns don't fail:

# Import existing CW Observability add-on into Terraform state (first run only; can be removed after).
# Add only this import line, BEFORE the workflow's existing `terraform apply` step, and mention that it can be removed after the first run as a comment.
terraform import -var="region=..." -var="cluster_name=..." \
  aws_eks_addon.cloudwatch_observability <cluster-name>:amazon-cloudwatch-observability || true

Add-on already exists (CDK): do NOT add it to CDK; no change needed.

Do NOT introduce helm_release, kubernetes, or helm provider resources for this purpose.

Step 3: IAM permissions for the CloudWatch Agent

The CloudWatch Agent needs CloudWatchAgentServerPolicy and AWSXRayDaemonWriteAccess to send metrics, logs, and traces. When ServiceEvents Dynamic Instrumentation applies (Tier 2), also add a custom policy with application-signals:ListInstrumentationConfigurations and application-signals:ReportInstrumentationConfigurationStatus on Resource: "*".

Attach to the role the CloudWatch Agent uses, per platform:

EKS: the node group's IAM role (used by the CloudWatch Agent pods).
ECS: the role used by the CloudWatch Agent container (task role or execution role, depending on deployment).
EC2: the instance profile / role used by the CloudWatch Agent process.

EKS — terraform-aws-modules/eks/aws module (most common): add to iam_role_additional_policies:

resource "aws_iam_policy" "application_signals_di" {
  name = "${var.cluster_name}-${var.region}-application-signals-di"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = [
        "application-signals:ListInstrumentationConfigurations",
        "application-signals:ReportInstrumentationConfigurationStatus"
      ]
      # Resource = "*" is the recommended scope for Dynamic Instrumentation: these
      # application-signals actions do not support resource-level permissions.
      Resource = "*"
    }]
  })
}

eks_managed_node_groups = {
  main = {
    iam_role_additional_policies = {
      CloudWatchAgentServerPolicy = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
      AWSXRayDaemonWriteAccess    = "arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess"
      ApplicationSignalsDI        = aws_iam_policy.application_signals_di.arn
    }
  }
}

For raw aws_iam_role / ECS / EC2, attach the same three policies via aws_iam_role_policy_attachment. Use the exact managed-policy name AWSXRayDaemonWriteAccess (not AWSXRayWriteOnlyAccess). Omit the application_signals_di policy entirely for Lambda/.NET (Tier 1 only).

Note: the per-language guide (Step 4) may mention CloudWatchAgentServerPolicy but omit AWSXRayDaemonWriteAccess, or use a raw attachment pattern that doesn't match the module's iam_role_additional_policies syntax. Match the actual IaC pattern; prefer this step's guidance if they conflict.

Step 4: Apply the per-platform, per-language enablement guide

Read the guide for the detected combination and apply its instrumentation changes (the inject annotation on EKS, the ADOT init container on ECS, the SDK/agent install on EC2, the Lambda layer on Lambda):

references/cloudwatch/appsignals-guides/<platform>-<language>.md

Valid platforms: ec2, ecs, eks, lambda. Valid languages: python, nodejs, java, dotnet. Example: Python on EKS → references/cloudwatch/appsignals-guides/eks-python.md.

If the running agent has the AWS MCP server / Application Signals tooling available, get_enablement_guide can be used as a fallback for the same parameters.

Step 5 (Tier 2 only — skip for Lambda and .NET): ServiceEvents env vars

For EC2/ECS/EKS with Python/Node.js/Java, add the ServiceEvents environment variables.

5a and 5b are the core of Tier 2 — apply them by default, do NOT ask the user whether to include them. They are part of what "enable ServiceEvents / onboard to Application Signals" means. They differ in how strict they are:

5b (OTLP endpoints for transport) is required where it applies — without it ServiceEvents telemetry has nowhere to go. Wire it per the platform (see 5b for the ECS/EC2-vs-EKS specifics).
5a (git/deployment metadata) is best-effort — it's correlation data that degrades gracefully. Wire in whatever the IaC supports; if a value can't be sourced (e.g. no CI/CD provider for a git URL / commit SHA, or no deploy-time hook for the deployment vars), set what you can and skip the rest, noting it in the Step 6 review rather than blocking onboarding or interrogating the user.

Only 5c and 5d are optional opt-in questions.

5a. Git & deployment metadata (CI/CD propagation) — should be applied by default, best-effort

Add these 5 env vars, using the exact names below. See application-signals-cicd-metadata.md for the full propagation-chain procedure and per-provider patterns.

Variable (exact name)	When set
`OTEL_AWS_SERVICE_EVENTS_GIT_REPO_URL`	build-time (bake into image)
`OTEL_AWS_SERVICE_EVENTS_GIT_COMMIT_SHA`	build-time (bake into image)
`OTEL_AWS_SERVICE_EVENTS_DEPLOYMENT_URL`	deploy-time (runtime env var)
`OTEL_AWS_SERVICE_EVENTS_DEPLOYMENT_ID`	deploy-time (runtime env var)
`OTEL_AWS_SERVICE_EVENTS_DEPLOYMENT_TIMESTAMP`	deploy-time (runtime env var)

5b. OTLP endpoints — required in specific environments

ServiceEvents adds two OTLP endpoint env vars — OTEL_AWS_OTLP_LOGS_ENDPOINT and OTEL_AWS_OTLP_METRICS_ENDPOINT. These are in addition to (not replacements for) the base Application Signals exporter env vars the per-platform guide already sets (OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT). On ECS/EC2 a fully onboarded service ends up with all of them. All point at the CloudWatch Agent's OTLP receiver on port 4316 (NOT the OpenTelemetry SDK default 4318). Where the two ServiceEvents vars are set depends on the platform:

Variable	EKS	ECS / EC2
`OTEL_AWS_OTLP_LOGS_ENDPOINT`	Auto-injected by the CloudWatch Observability add-on — do NOT set as a pod env var	Set manually: `http://localhost:4316/v1/logs` (ECS sidecar / EC2), or the CloudWatch Agent host/IP on port 4316 (ECS daemon)
`OTEL_AWS_OTLP_METRICS_ENDPOINT`	Auto-injected — do NOT set	Set manually: `http://localhost:4316/v1/metrics` (ECS sidecar / EC2), or the CloudWatch Agent host/IP on port 4316 (ECS daemon)

EKS: do NOT manually set the OTLP endpoint env vars on the pod — the amazon-cloudwatch-observability add-on injects them into instrumented pods with the correct values. On EKS, Step 5b typically adds nothing to the Deployment manifest; the Step 5a metadata env vars are still set as usual.

Steps 5c and 5d are the only parts of onboarding to ask the user about — two separate, optional ServiceEvents features, both off by default and both Tier 2 (EC2/ECS/EKS × Python/Node.js/Java). (5a and 5b above are not opt-in questions — they are applied by default; see the Step 5 intro.) Ask the user about 5c and 5d each as its own distinct question before moving to Review — they are independent (the user may want neither, either, or both). Fold whatever the user opts into the same place as the other Step 5 env vars (k8s Deployment env, ECS container env, or EC2 process/userdata env), so Step 6 reviews the complete set.

5c (optional): Per-function instrumentation

Ask the user whether they want per-function (FunctionCall) telemetry for their own application code. It emits nothing by default — but not because a toggle is off: OTEL_AWS_SERVICE_EVENTS_FUNCTION_INSTRUMENT_ENABLED is already true by default. What suppresses output is the empty OTEL_AWS_SERVICE_EVENTS_PACKAGES_INCLUDE allowlist. The two work as a pair — with the flag on but no allowlist, the SDK installs the hooks and instruments nothing. So opting in means setting one env var (do NOT set the enable flag — it is already on):

Variable	Value
`OTEL_AWS_SERVICE_EVENTS_PACKAGES_INCLUDE`	The only way to opt code in. Empty = nothing instrumented (there is no implicit default scope). On Node.js, a list entry of exactly `` or `` is dropped (with a warning) as too broad — but partial wildcards (`/src/`, `.js`) are fine.

The match syntax differs per SDK — set it to the customer's own application code, not third-party libraries:

SDK	Form	Example
Java	Java package prefix (dot-separated; no wildcard needed)	`com.example.simplesample`, `com.amazon.indico`
Python	dotted module path + `.*`	`indico.`, `myapp.`
Node.js	path glob (minimatch) matched against the file's absolute resolved path (NOT a module name)	`/indico/src/` — i.e. `/<app-dir>/src/` for code under `<app-dir>/src/`

Determining the value — inspect the customer's source layout. PACKAGES_INCLUDE is the one onboarding value that depends on how the customer's code is organized, so read the repo to derive it (reading source to determine config is allowed; the never-modify rule is about editing source, not looking at it). Per SDK:

Java — find the application's root package from the source tree (src/main/java/<group>/<artifact>/…) or the package/namespace declarations and groupId in pom.xml/build.gradle. Use the top-level package that covers the customer's own classes, e.g. com.amazon.indico.
Python — find the top-level package directory (the one with __init__.py, or the name/packages in pyproject.toml/setup.py) and append .*, e.g. myapp.*.
Node.js — find the directory holding the customer's own source (commonly src/, or main/exports in package.json) and build a path glob **/<app-dir>/src/**. Remember it matches the absolute runtime path, so anchor on a suffix that survives the build/deploy (the **/ prefix), not the repo-relative path.

If the layout is ambiguous or spans multiple top-level packages, confirm the intended scope with the user rather than guessing — too broad an allowlist adds overhead and noise; too narrow misses functions. Prefer the customer's own application packages over dependencies unless the user explicitly wants a dependency instrumented.

Node.js — the leading **/ is required, not optional. The SDK matches the pattern against the fully-resolved absolute path (e.g. /app/indico/src/handlers/order.js), which begins with deploy-specific prefixes the customer doesn't control (/app, the WORKDIR, etc.). minimatch's matchBase only helps for slash-free patterns; any pattern containing a / (like …/src/**) is anchored to the whole absolute path, so indico/src/** matches nothing. Lead with **/ to absorb the prefix (**/indico/src/**), or — less portably — hardcode the absolute path (/app/indico/src/**). Usually you want the customer's own application code.

5d (optional): Dynamic Instrumentation

Ask the user — as a separate question from 5c — whether they want Dynamic Instrumentation. It shares OTEL_AWS_OTLP_LOGS_ENDPOINT with ServiceEvents. To enable, set OTEL_AWS_DYNAMIC_INSTRUMENTATION_ENABLED=true — on EKS either as a pod env var on the Deployment OR via the add-on's autoInstrumentationConfiguration (configuration_values); on ECS/EC2 as a container/process env var. Leave it off (omit, or set false) unless the user wants it.

Variable	EKS	ECS / EC2
`OTEL_AWS_DYNAMIC_INSTRUMENTATION_ENABLED`	Opt in by EITHER setting it `true` as a pod env var OR via the add-on's `autoInstrumentationConfiguration`	Set `true` to opt in
`OTEL_AWS_DYNAMIC_INSTRUMENTATION_API_URL`	Auto-injected — do NOT set	Only needed on ECS daemon (CloudWatch Agent host/IP on port 2000); default `localhost:2000` works on ECS sidecar / EC2

Step 6: Review

Summarize all changes grouped by file, state the platform + language, list the env vars that will reach the app at runtime (including any optional 5c / 5d features the user opted into), and note build-time vs deploy-time. Explicitly call out anything that was NOT set — in particular any 5a git/deployment metadata vars skipped because their value couldn't be sourced (which ones, and why, e.g. "no CI/CD provider detected to supply GIT_COMMIT_SHA"), so the user knows the metadata is partial and can wire it manually if they want full deployment correlation. Present for the user to review and commit. Do not deploy automatically.

Constraints

Minimum changes; preserve existing content and formatting; never duplicate an env var, policy, or resource that already exists.
Only IaC, Dockerfiles, CI/CD workflows, dependency files, and deployment manifests — never application source code.
OTLP endpoints must target the CloudWatch Agent's OTLP receiver on port 4316.
Use exact env var names and the exact managed-policy name AWSXRayDaemonWriteAccess.
Lambda and .NET get Tier 1 (Application Signals enablement) only — no ServiceEvents env vars.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an MCP server, Needs Python

Security

What our scanner noticed· Medium risk

References to credential files
Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

AWS

Publisher

AWS (Official)

Source

aws/agent-toolkit-for-aws · skills/specialized-skills/operations-skills/setting-up-cloudwatch-observability · commit 0a218f8

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
NOTICE (added by AutoGineer from NOTICE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by AWS or AWS.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download setting-up-cloudwatch-observability.zip