AutoGineer
Generated banner for Setting Up Cloudwatch Observability
Generated logo for Setting Up Cloudwatch Observability

Setting Up Cloudwatch Observability

AWSOfficialPublished by AWS

  • Skill
  • Tool integration
  • Cloud and DevOps
  • Apache-2.0
  • Medium risk
Download ZIP

From the author

Sets up CloudWatch observability for the first time - Omni (CloudWatch Application Observability) and classic CloudWatch. Omni: creating a Space or Domain; access grants (who has access, at what level), access profiles bounding alerts, integrations, agents; instrumenting an app or AI agent with the plain ADOT SDK so traces reach Omni (Python/Node/Java/.NET on EC2/ECS/EKS/Lambda), incl. no-image-rebuild, CoreCLR vars; ingesting Azure or Google Cloud telemetry via the CloudWatch agent (VM/AKS, GCE/GKE); connecting Slack or the AWS DevOps Agent to a Space; whether GitHub or a custom MCP tool server (HTTP/stdio) can connect. CloudWatch: onboarding a service to Application Signals: ADOT auto-instrumentation, amazon-cloudwatch-observability add-on, monitored service, ServiceEvents, CI/CD metadata, Terraform. For using what is set up - queries, dashboards, alarms, Omni alerts, X-Ray, synthetics, Dynamic Instrumentation, AI agent evaluation, detecting Omni vs classic - use aws-observability.
— the description by AWS

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides the end-to-end setup procedures for two distinct CloudWatch observability products: CloudWatch Application Observability (Omni) and classic CloudWatch Application Signals. It acts as a central routing hub, helping users identify the correct product based on their requirements and guiding them through the specific onboarding, instrumentation, and configuration steps for each. It covers resource creation, access management, and telemetry ingestion, while clearly delineating the boundaries between the two products to ensure correct implementation.

What it does

  • Routes setup requests to the correct product path based on user intent and environment.
  • Guides users through CloudWatch Omni Domain and Space creation, access grants, and profile configuration.
  • Provides instructions for instrumenting applications and AI agents with the ADOT SDK for Omni.
  • Details the onboarding process for classic CloudWatch Application Signals, including auto-instrumentation and add-ons.
  • Supports telemetry ingestion configuration for Azure and Google Cloud environments.
  • Manages integration setups for Slack and the AWS DevOps Agent.

When to use it

  • When you need to set up CloudWatch Application Observability (Omni) for the first time.
  • When you need to onboard a service to classic CloudWatch Application Signals.
  • When you are unsure which CloudWatch observability product is appropriate for your workload.
  • When you need to configure access grants, profiles, or integrations for an existing Space.
  • When you need to instrument an application or AI agent to export telemetry to CloudWatch.

Advantages

  • Prevents configuration conflicts by enforcing mutually exclusive setup paths for Omni and Application Signals.
  • Provides a structured, step-by-step sequence for complex resource hierarchies like Domains and Spaces.
  • Clarifies the distinction between setup tasks and day-to-day operational tasks.
  • Ensures correct principal resolution and permission scoping for access grants and profiles.

Usage examples

“How do I start setting up CloudWatch Application Observability?”
The agent will explain the setup order (Domain, Space, grants, telemetry) and direct you to the app-basics reference to begin.
“I need to onboard my EKS service to Application Signals.”
The agent will route you to the Application Signals onboarding guide and the specific EKS-language instrumentation steps.
“How do I give a team access to my CloudWatch Omni Space?”
The agent will guide you through resolving the principal and using the access-grants reference to configure the appropriate permission level.
“Can I connect my GitHub repository to CloudWatch Omni?”
The agent will clarify that Omni does not support GitHub integration and stop the setup process.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/aws/setting-up-cloudwatch-observability/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip setting-up-cloudwatch-observability.zip -d .agents/skills

Documentation by AWS

Setting Up CloudWatch Observability

Scope: First-time CloudWatch observability setup, for both products — a CloudWatch Application Observability (Omni) Space from creation through first traces flowing, and onboarding a service to classic CloudWatch Application Signals. For using what is already set up (queries, dashboards, alarms, Omni alerts, evaluations, live debugging), route to aws-observability.

Two products, two reference folders

This skill owns setup for two products that share the CloudWatch name but are separate services. The folder a reference lives in is the signal for which product it belongs to — the same convention aws-observability uses:

FolderProductWhat setup means there
references/cloudwatch-omni/CloudWatch Omni (Application / Agent Observability)Domain → Space → grants → telemetry in → plain ADOT SDK instrumentation. No add-on, no CloudWatch Agent application_signals config, no port 4316.
references/cloudwatch/Classic CloudWatchOnboarding a service to Application Signals: ADOT auto-instrumentation, the amazon-cloudwatch-observability EKS add-on, the CloudWatch Agent, monitored service, ServiceEvents, CI/CD git/deployment metadata, port 4316.

The two instrumentation paths are mutually exclusive on a given workload — the Omni path explicitly forbids the Application Signals env vars and the add-on, and vice versa. Decide the product first (Routing Rules 1–4 below), then stay inside that folder. Enabling one does not replace the other; an account may run both, on different services.

Works best with the AWS MCP server — enables running AWS CLI commands directly. All guidance also works with standard AWS CLI access (aws cloudwatchomni ... on the Omni path; aws eks, aws iam, and aws application-signals on the Application Signals path).

Concepts — Omni (references/cloudwatch-omni/)

These are Omni's resources. None of them exist on the Application Signals path — that one has no Domain, Space, grant, or Dataset; its unit is a monitored service, and access is plain IAM. Do not ask an Application Signals onboarding customer for a spaceId.

TermWhat it is
DomainThe identity boundary. Carries the authorization provider (IAM or Identity Center) and owns the endpoint URL customers reach Omni through. One per account, or one shared across an AWS Organization.
SpaceA workspace holding telemetry, in exactly one account and one Region. Created under a Domain. At most one per account per Region. Region rule: under an IAM-only Domain a Space may sit in a Region other than the Domain's; under an account-scoped Identity Center Domain the Space must be in the Domain's own Region; under an org-scoped Identity Center Domain it must be in a Region the instance is replicated to.
Access grantAttaches a principal — person, group, IAM identity, or async workload — to one Space at a permission level. The only way anyone reaches data through a Space — it does not restrict the source CloudWatch log groups, which stay readable under their own IAM.
Access ProfileA named boundary for async workloads (alerts, integrations, agents) that act without a person in the loop. It is only a named container: create-access-profile takes a Space, a name, and a description and nothing else — no permission, action, or scope input. It does something only once two separate sets of grants exist (what the profile may do; which workloads may assume it) and a workload names it.
DatasetWhat queries run against. Telemetry arrives through the CloudWatch OTLP endpoints, or by forwarding what is already in CloudWatch log groups.

Omni setup order from nothing: Domain → Space → grants → telemetry in → instrumentation. (The Application Signals equivalent is much shorter and has no prerequisite resources — add-on/agent, IAM, then the per-platform enablement change; see references/cloudwatch/application-signals-onboarding.md.) "Telemetry in" (a collector exporting to CloudWatch's OTLP endpoints, plus dataset forwarding for what is already in CloudWatch) is its own step, separate from instrumenting the workloads. Access Profiles are conditional, not a step in the sequence — only when async workloads (alerts, integrations, agents) are involved. Whenever you give this sequence, also say that instrumentation or forwarding started before a Space exists appears to succeed while delivering telemetry nowhere the customer can see — a customer who checks Omni first reads a working, empty Space as a failure. For the concept relationships and the full arc, see references/cloudwatch-omni/app-basics.md.

This is a routing skill. Classify the user's setup request and delegate to the correct reference. Omni references live under references/cloudwatch-omni/; classic-CloudWatch (Application Signals) references live under references/cloudwatch/.

User intentReference
Onboard a service to Application Signals (auto-instrumentation, the amazon-cloudwatch-observability EKS add-on, CloudWatch Agent IAM for Application Signals, monitored service, reporting telemetry to Application Signals, ServiceEvents, the two onboarding tiers)references/cloudwatch/application-signals-onboarding.md
Propagate ServiceEvents git/deployment metadata through CI/CD (the 5 OTEL_AWS_SERVICE_EVENTS_* vars, per-provider patterns)references/cloudwatch/application-signals-cicd-metadata.md
Per-platform × per-language Application Signals enablement steps once platform and language are knownThe matching references/cloudwatch/appsignals-guides/<platform>-<language>.md (e.g. references/cloudwatch/appsignals-guides/eks-python.md)
Turn on Dynamic Instrumentation for a service at onboarding time (the OTEL_AWS_DYNAMIC_INSTRUMENTATION_* vars and their IAM)references/cloudwatch/application-signals-onboarding.md (Step 5d). Using it to debug is aws-observability
Understand what Omni is, its concepts, or where to startreferences/cloudwatch-omni/app-basics.md
Instrument an AI agent (ADOT, OpenInference, framework detection, trace verification), or deploy an agent to production and get traces flowing to CloudWatch — env vars per platform (AgentCore, Lambda, or other platforms such as ECS/EC2/EKS), routing spans to a custom trace log group, IAM permissions needed, how the export authenticates (AWS credentials, no API token), ADOT version requirementsreferences/cloudwatch-omni/omni-agents-instrumentation/omni-agents-instrumentation.md (§ Production deployment for the deploy case)
Per-framework OpenInference guide (LangChain, LangGraph, Strands, CrewAI, OpenAI Agents, Vercel AI) once the agent framework is knownreferences/cloudwatch-omni/omni-agents-instrumentation/openinference-framework-guide.md, then the matching references/cloudwatch-omni/omni-agents-instrumentation/instrument-<framework>.md
Instrument an application (ADOT SDK on EC2/ECS/EKS/Lambda — Python, Node.js, Java, .NET)references/cloudwatch-omni/instrumentation/instrumentation.md
Emit a custom application or agent metric so it is queryable in Omni (why OTLP and not PutMetricData/EMF)references/cloudwatch-omni/instrumentation/instrumentation.md (§ Custom metrics)
Create an account-scoped Space or Domainreferences/cloudwatch-omni/spaces-and-domains.md
Whether a Space can be in a different Region from its Domainreferences/cloudwatch-omni/spaces-and-domains.md (Prerequisites → Region rules)
The AgentCore evaluation role that create-space asks for — what it is, whether to create onereferences/cloudwatch-omni/spaces-and-domains.md (Step 3 → Also resolve the AgentCore evaluation role)
Let Omni run Bedrock models for a Space (Agent Playground, evaluations run from Omni), or a Bedrock AccessDeniedException from those featuresreferences/cloudwatch-omni/spaces-and-domains.md (Step 3 → Optional: let Omni invoke Bedrock models)
A Space that was created successfully but returns an authorization error when usedreferences/cloudwatch-omni/spaces-and-domains.md (Troubleshooting → If the Space was created but cannot be used) — a space access role trust-policy problem, not a grant problem
Create a Domain shared across an AWS Organizationreferences/cloudwatch-omni/org-domains.md
Configure access grants for people or IAM identities or alertsreferences/cloudwatch-omni/access-grants.md
An aws cloudwatchomni command reporting that the service is not supported, or a request to repair the customer's CLI or SDK so Omni commands workNot a setup task — a local client-version problem. STOP and route to aws-observability
A programmatic caller denied on a Space it already had working — a CI role, SDK client, CloudFormation role, coding agent, or a workload under an Access Profilereferences/cloudwatch-omni/access-grants.md (Grants, IAM, and Access Profiles), and aws-observability for the API-side view
Bound an alert, integration, or agent with an Access Profilereferences/cloudwatch-omni/access-profiles.md
Deploy an OTel Collector so an instrumented app has somewhere to export to (EC2/ECS/EKS) — it exports to CloudWatch's own per-signal OTLP endpointsreferences/cloudwatch-omni/instrumentation/collector.md
An OTLP collector already running (a CloudWatch Agent with an opentelemetry section or OTLP on 4317/4318, or otelcol-contrib) whose telemetry does not arrive, in whole or for one signal such as tracesreferences/cloudwatch-omni/instrumentation/collector.md (Verify), then the platform collector guide it links for the troubleshooting table
Enable Transaction Search so traces reach a Space (spans land in aws/spans only once it is on — per account, per Region)references/cloudwatch-omni/instrumentation/collector.md (Step 1)
Forward telemetry already in CloudWatch into the Datasetreferences/cloudwatch-omni/data-forwarding-and-centralization.md
Send your application's own telemetry from Azure (the logs/metrics/traces your service emits, via the CloudWatch agent on an Azure VM/AKS)references/cloudwatch-omni/azure-ingestion/azure-ingestion.md (intent triage), then references/cloudwatch-omni/azure-ingestion/custom-telemetry.md (the CloudWatch-agent-on-VM/AKS procedure)
Send your application's own telemetry from Google Cloud (the logs/metrics/traces your service emits, via the CloudWatch agent on a GCE instance/GKE cluster)references/cloudwatch-omni/gcp-ingestion/gcp-ingestion.md (intent triage), then references/cloudwatch-omni/gcp-ingestion/custom-telemetry.md (the CloudWatch-agent-on-GCE/GKE procedure)
Connect Slack to a Space for the first timereferences/cloudwatch-omni/slack-integration.md
Connect the AWS DevOps Agent to a Space (Settings → Integrations → AWS DevOps Agent), an empty agent-space picker, or Investigations reporting the AWS DevOps Agent is not connectedreferences/cloudwatch-omni/devops-agent-integration.md. Creating the DevOps Agent space itself and running or chatting with investigations belong to the aws-agents-for-devsecops plugin
Whether a custom MCP tool server can be registered (Omni has none)references/cloudwatch-omni/custom-mcp-integration.md
Enable or configure CloudWatch Omni Intelligence for a Space — attach an optional customer-managed KMS key (put-intelligence-configuration), connect an intelligence integration (AWS_INTEGRATION, AWS_CONFIG_SLREC, EXTERNAL_AGENT), check current config, or troubleshoot enablement (404, AccessDenied, empty graph)references/cloudwatch-omni/intelligence-setup.md

Routing Rules

  1. If the user is asking what Omni is, what a Domain or Space or grant means, where to start, or the end-to-end order of steps from nothing — rather than asking to perform one setup step — route to references/cloudwatch-omni/app-basics.md and answer the sequence from its "Setup order" section (not from a single procedure file such as collector.md, which covers one step). It also carries the boundary against aws-observability. A question about one concept's rules — a Space's Region relative to its Domain, the roles create-space needs — belongs to the procedure file for that concept (spaces-and-domains.md), which the routing table names.

  2. If the request is about instrumenting an AI agent project (adding OTel to a framework such as LangChain, LangGraph, Strands, CrewAI, OpenAI Agents, or Vercel AI), it is Omni — route to references/cloudwatch-omni/omni-agents-instrumentation/omni-agents-instrumentation.md. There is no Application Signals path for an agent framework.

  3. If the request is about instrumenting a general application (a service on EC2, ECS, EKS, or Lambda in Python, Node.js, Java, or .NET — not an AI-agent framework), decide the product before opening any procedure file — the two paths are mutually exclusive on a workload:

    • Names Application Signals, ServiceEvents, a "monitored service", the service map, the amazon-cloudwatch-observability add-on, the CloudWatch Agent's application_signals configuration, or port 4316 → the CloudWatch path: references/cloudwatch/application-signals-onboarding.md, then the matching references/cloudwatch/appsignals-guides/<platform>-<language>.md.

    • Names Omni, Application Observability, a Space, a Domain, or a Dataset — or explicitly rules Application Signals out ("plain OTel", "no add-on", "no agent sidecar") → the Omni path: references/cloudwatch-omni/instrumentation/instrumentation.md.

    • Naming the CloudWatch Agent alone decides nothing: it is also the Omni path's default collector. An agent with an opentelemetry section or OTLP receivers on 4317/4318 is the Omni collector, and its delivery problems belong to rule 6.

    • Names neither → rule 4.

  4. If an instrumentation, ADOT, or collector request names neither product (neither Omni or a Space, nor Application Signals, ServiceEvents, or the amazon-cloudwatch-observability add-on), probe for a Space in the target Region before choosing. list-spaces is account-global: the --region flag only selects the endpoint, and the response lists every Space in the account, each with its own region, so filter to the target Region rather than trusting a non-empty list: aws cloudwatchomni list-spaces --region <region> --query "items[?region=='<region>']". A Space in that Region → the Omni path, references/cloudwatch-omni/instrumentation/instrumentation.md. An empty filtered list → the customer has not adopted Omni in that Region, so Application Signals is the live product for them: references/cloudwatch/application-signals-onboarding.md. Either way the request stays in this skill — the probe picks the folder, not the skill. If the probe errors with an unknown service, that is the CLI model, not evidence Omni is absent — fall back to the customer's wording and ask only if still inconclusive.

  5. If the request is about Space/Domain creation or configuration, route to the matching setup reference. An authorization error that appears the moment a just-created Space is used is part of this — it is a space access role trust-policy problem (create-space never verifies the role can be assumed), so route to references/cloudwatch-omni/spaces-and-domains.md → "If the Space was created but cannot be used", not to the access-grants reference.

  6. If the user asks about getting telemetry to a destination — telemetry not yet reaching CloudWatch — deploy an OTel Collector on EC2/ECS/EKS so an instrumented workload has somewhere to send OTLP, and wire the app's OTEL_EXPORTER_OTLP_ENDPOINT to it: references/cloudwatch-omni/instrumentation/collector.md. The same file covers an OTLP collector that is already deployed but whose telemetry does not arrive, in whole or for one signal (its Verify section and the platform guide's troubleshooting table). The collector exports straight to CloudWatch's own per-signal OTLP endpoints. If instead the telemetry is already in CloudWatch log groups and needs forwarding into the Dataset, route to references/cloudwatch-omni/data-forwarding-and-centralization.md. Both the collector's OTLP export and CloudWatch's OTLP endpoints authenticate with SigV4 (AWS credentials), including OIDC-federated credentials for a workload outside AWS (see references/cloudwatch-omni/azure-ingestion/custom-telemetry.md or references/cloudwatch-omni/gcp-ingestion/custom-telemetry.md). A purely token-based ingestion path for a sender that cannot obtain AWS credentials at all is not covered by these skills; do not improvise one, and say so plainly.

  7. If the request is about getting Azure or Google Cloud telemetry into CloudWatch, route by cloud — references/cloudwatch-omni/azure-ingestion/azure-ingestion.md for Azure, references/cloudwatch-omni/gcp-ingestion/gcp-ingestion.md for Google Cloud — and decide by intent. If the customer wants the telemetry their own application produces (the logs/metrics/traces from their code), that is supported via the CloudWatch agent on an Azure VM or AKS cluster, or on a GCE instance or GKE cluster — follow the reference. If they want telemetry their cloud's resources emit on their own (the equivalent of AWS VPC flow logs / Route 53 logs: Azure resource and platform logs, or Google Cloud Audit Logs), that is not supported — say so and do not attempt a setup. If the request names no cloud, ask which one before giving any steps.

  8. If the user asks to connect, enable, or authorize Slack for a Space for the first time (including granting the operator permission to use it), route to references/cloudwatch-omni/slack-integration.md. Using Slack after it is connected (posting findings to a channel, mentioning the assistant, or searching Slack) happens in Slack and the console and is not covered by these skills; Slack as an alert notification target is covered by aws-observability's Omni alerts reference.

  9. Application Signals, ServiceEvents, and Dynamic Instrumentation are CloudWatch features, and they split by setup versus use — not by skill. Enabling them on a service that does not have them yet is setup and belongs here, under references/cloudwatch/: Application Signals onboarding and ServiceEvents in references/cloudwatch/application-signals-onboarding.md (with the CI/CD metadata chain in references/cloudwatch/application-signals-cicd-metadata.md), and switching Dynamic Instrumentation on at instrumentation time in that same file's Step 5d. They must never appear in an Omni instrumentation change — for that they remain out of scope, and rule 3 is how you tell the two paths apart. Using them once the service reports — reading the service map, alarming on Application Signals metrics, or placing breakpoints and reading snapshots with Dynamic Instrumentation — is day-to-day work: STOP and route to aws-observability.

  10. If the user already has a working Space, or a service already reporting to Application Signals, and is asking about queries/dashboards/alarms/alerts/investigation, STOP and route to aws-observability.

  11. If the user asks to register their own MCP tool server, connect a custom MCP server, or set up MCP authentication: CloudWatch Omni does not let customers add a custom MCP server. There is no console screen for it and the integration type is not enabled for customer use, so the attempt is refused. Say so plainly and stop; do not describe a setup flow, and do not offer a substitute. See references/cloudwatch-omni/custom-mcp-integration.md.

  12. If the user asks to connect GitHub, link a GitHub organization or repositories, or enrich the application map from source code: CloudWatch Omni has no GitHub integration. Say so plainly and stop — do not walk through a setup. Do not offer a substitute: the Application Signals GitHub Action and the CloudWatch Logs GitHub audit-log source are separate CloudWatch features, not a way to connect GitHub to Omni, and a custom MCP server is not a GitHub integration. A GitHub-related feature found in public documentation is not evidence of an Omni GitHub integration.

  13. If the user asks to enable or configure CloudWatch Omni Intelligence — attach or remove a customer-managed KMS key for intelligence data, add an intelligence integration (AWS_INTEGRATION, AWS_CONFIG_SLREC, or EXTERNAL_AGENT), read the current intelligence configuration, or troubleshoot why it will not turn on (a 404 UnknownOperationException, an AccessDeniedException, or an empty context graph) — route to references/cloudwatch-omni/intelligence-setup.md. That reference is the source of truth for the KMS ARN rules (full key ARN only, kms:Decrypt/kms:GenerateDataKey, exactly one of --kms-key-arn/--remove-kms-key, and that revoking the key makes intelligence data unrecoverable), the per-integration extra IAM, the two AccessDenied causes, and why "not enabled" is a 404 rather than an AccessDenied. Reading or interpreting the intelligence data once it is on is day-to-day use: STOP and route to the aws-observability skill — triaging insights, observations, or anomalies goes to its intelligence.md reference, which covers the ListInsights / ListObservations / GetContextGraph operations and the 404-vs-403-vs-local error rule; a pure topology or blast-radius question goes to its context-graph.md reference. For Slack specifically, route to references/cloudwatch-omni/slack-integration.md; for a custom MCP server, route to references/cloudwatch-omni/custom-mcp-integration.md (not supported). Do not offer GitHub, PagerDuty, Teams, MODEL_PROVIDER, or CUSTOM_MCP as intelligence integration types.

  14. If the request names neither product — "add observability to my API", "how should I begin monitoring my workload", with no mention of Omni, a Space, a Domain, or of a CloudWatch feature — do not assume either one. Which product a customer wants to set up is their intent, not something the account's current state tells you, so ask, framed product versus product: CloudWatch Omni (Application/Agent Observability — Domains, Spaces, grants, a Dataset) or CloudWatch (log groups, alarms, Log Insights, Application Signals). Both answers are served here, from different reference folders, so this is a question about which product to set up — not about which skill to use, and not a reason to hand the request to aws-observability. In the same message, explain the boundary so they can choose — it is setup versus use: creating or configuring a Domain, Space, grant, or Access Profile, getting telemetry flowing for the first time, instrumenting an application or agent for Omni, and onboarding a service to Application Signals are all first-time setup and belong here; querying telemetry, building dashboards, configuring alarms or Omni alerts, investigating a live problem, and debugging with Dynamic Instrumentation are day-to-day use and belong to aws-observability. Do not start walking through Domain/Space creation or Application Signals onboarding until they confirm which product; having asked, stop and wait.

  15. If the user asks to connect the AWS DevOps Agent to a Space, or the connect card or Investigations shows it is not connected, route to references/cloudwatch-omni/devops-agent-integration.md. These skills cover only that Omni-side connection. Creating or configuring the DevOps Agent space, and running, following, or chatting with its investigations from a coding agent, belong to the agent's dedicated aws-agents-for-devsecops plugin; point the customer there rather than reconstructing that workflow.

Files

references/cloudwatch/appsignals-guides/ec2-nodejs.md

Enable AWS Application Signals for Node.js on EC2

Your task is to modify Infrastructure as Code (IaC) files to enable AWS Application Signals for a Node.js application running on EC2 instances. You will update IAM permissions, install monitoring agents, and configure OpenTelemetry instrumentation through UserData scripts.

What You Will Accomplish

After completing this task:

  • The EC2 instance will have permissions to send telemetry data to CloudWatch

  • The CloudWatch Agent will be installed and configured for Application Signals

  • The Node.js application will be automatically instrumented with AWS Distro for OpenTelemetry (ADOT)

  • Traces, metrics, and performance data will appear in the CloudWatch Application Signals console

  • The user will be able to see service maps, SLOs, and application performance metrics without manual code instrumentation

Critical Requirements

Error Handling:

  • If you cannot determine required values from the IaC, STOP and ask the user

  • For multiple EC2 instances, ask which one(s) to modify

  • Preserve all existing UserData commands; add new ones in sequence

Do NOT:

  • Run deployment commands automatically (cdk deploy, terraform apply, etc.)

  • Remove existing application startup logic

  • Skip the user approval step before deployment

IaC Tool Support

Code examples use CDK TypeScript syntax. If you are working with Terraform or CloudFormation, translate the CDK syntax to the appropriate format while keeping all bash commands identical. The UserData bash commands (CloudWatch Agent installation, ADOT installation, environment variables) are universal across all IaC tools - only the wrapper syntax differs.

Before You Start: Gather Required Information

Execute these steps to collect the information needed for configuration:

Step 1: Determine Deployment Type

Read the UserData script and look for the application startup command. This is typically one of the last commands in UserData.

If you see:

  • docker run or docker start → Docker deployment

  • node, npm start, yarn start, or similar → Non-Docker deployment

If unclear:

  • Ask the user: "Is your Node.js application running in a Docker container or directly on the EC2 instance?" DO NOT GUESS

Critical distinction: Where does the Node.js process run?

  • Docker: Node.js runs inside a container → Modify Dockerfile

  • Non-Docker: Node.js runs directly on EC2 → Modify UserData

Step 2: Extract Placeholder Values

Analyze the existing IaC to determine these values for Application Signals enablement:

  • {{SERVICE_NAME}}

    • Why It Matters: Sets the service name displayed in Application Signals console via OTEL_RESOURCE_ATTRIBUTES=service.name={{SERVICE_NAME}}

    • How to Find It: Use the application name, stack name, or construct ID. Look for service/app names in the IaC.

    • Example Value: my-nodejs-app

    • Required For: Both Docker and non-Docker

  • {{ENTRY_POINT}}

    • Why It Matters: Used to start the application with OpenTelemetry instrumentation: node --require ... {{ENTRY_POINT}}

    • How to Find It: Find the JavaScript file that starts the application (look for node commands in UserData)

    • Example Value: server.js, index.js, or app.js

    • Required For: Non-Docker

  • {{APP_DIR}}

    • Why It Matters: Node.js needs to run from the correct directory to find application files and dependencies

    • How to Find It: Find where the application code is deployed (look for cd, git clone, or file copy commands in UserData)

    • Example Value: /opt/myapp

    • Required For: Non-Docker

For Docker-based deployments you will also need to find these additional values:

  • {{APP_NAME}}

    • Why It Matters: Used to reference the container for operations like docker logs {{APP_NAME}}, docker exec, health checks, etc.

    • How to Find It: Find container name in docker run --name or use {{SERVICE_NAME}}-container

    • Example Value: nodejs-express-app

    • Required For: Docker

  • {{IMAGE_URI}}

    • Why It Matters: This is the identifier for the application that Docker will run

    • How to Find It: Find the Docker image in docker run or docker pull commands

    • Example Value: 123456789012.dkr.ecr.us-east-1.amazonaws.com/my-app:latest

    • Required For: Docker

If you cannot determine a value: Ask the user for clarification before proceeding. Do not guess or make up values.

Step 3: Identify Instance OS

Determine the operating system to use the correct package manager and installation commands.

Amazon Linux:

  • Amazon Linux 2: Use yum package manager

  • Amazon Linux 2023: Use dnf package manager

  • How to detect: Look for existing package install commands in UserData (check for yum or dnf), or look for AMI references containing al2 or al2023

Other Linux distributions:

  • Ubuntu/Debian: Use apt package manager

  • Fedora/RHEL/CentOS: Use dnf or yum package manager

If unclear: Look for AMI name/ID in the IaC or ask the user which OS the EC2 instance is running. Do not guess or make up values.

Step 4: Determine Module Format

Determine if the Node.js application uses CommonJS or ESM module format. This affects which ADOT dependencies to install and which node flags to use.

Check the application's package.json file:

  • Look for "type": "module" → ESM format

  • Look for "type": "commonjs" or no type field → CommonJS format (default)

Alternative checks:

  • If the main application file has .mjs extension → ESM format

  • If the main application file has .cjs extension → CommonJS format

  • If .js extension → Depends on package.json type field

If unclear:

  • Ask the user: "Does your Node.js application use ESM module format (type: module in package.json)?" DO NOT GUESS

  • Default to CommonJS if package.json doesn't specify type

Instructions

Follow these steps in sequence:

Step 1: Locate the IaC Files

Search for EC2 instance definitions using these patterns:

CDK:

new ec2.Instance(
ec2.Instance(
CfnInstance(

Terraform:

resource "aws_instance"

CloudFormation:

AWS::EC2::Instance

Read the file(s) containing the EC2 instance definition. You need to identify:

  1. The instance resource/construct

  2. The IAM role attached to the instance

  3. The UserData script or property

Step 2: Locate the IAM Role

Find the IAM role attached to the EC2 instance

CDK:

role: someRole
new iam.Role(this, 'RoleName'
Step 3: Update the IAM Role

What this policy is for, and its scope. CloudWatchAgentServerPolicy goes on the instance role for the CloudWatch agent, which receives telemetry locally and forwards it to CloudWatch and X-Ray — so these permissions are what let the agent reach those destinations, not something the instrumentation itself needs. On EC2, ECS, and EKS, an ADOT-SDK-only setup adds no IAM to the workload at all (see this skill's references/cloudwatch-omni/instrumentation/instrumentation.md, which forbids attaching this policy on that path). Lambda is the exception — that path does grant its execution role X-Ray write permissions.

The policy is broader than this configuration needs: it grants 14 actions, all on Resource: "*", including ec2:DescribeVolumes and logs:PutRetentionPolicy, which an Application-Signals-only agent config does not use. If the customer wants to trim it, resource scoping is the more valuable axis than pruning actions — dropping actions still leaves logs:PutLogEvents and logs:CreateLogGroup on every log group in the account and cloudwatch:PutMetricData on every namespace, so the instance role can still write over unrelated services' logs. Scope the resources and add an aws:ResourceAccount condition, the way CloudWatchLambdaApplicationSignalsExecutionRolePolicy does for the Lambda path.

Read the live document before changing anything — it takes two calls, since the version id is required and is not knowable up front: aws iam get-policy --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy --query Policy.DefaultVersionId then aws iam get-policy-version --policy-arn <same> --version-id <that>. Do not hand-roll an action list from this page: the agent still creates the Application Signals log group (/aws/application-signals/data), so logs:CreateLogGroup and logs:CreateLogStream must survive any trim. A denial does not surface in the console — the agent records AccessDenied in its own log (/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log on Linux), and the symptom is that telemetry never starts arriving. Check that file first.

Add the CloudWatch Agent Server Policy to the IAM role's managed policies.

CDK:

const role = new iam.Role(this, 'AppRole', {
  assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'),
  managedPolicies: [
    iam.ManagedPolicy.fromAwsManagedPolicyName('CloudWatchAgentServerPolicy'),
    // ... keep existing policies
  ],
});
Step 4: Modify UserData - Add Prerequisites

Add a CloudWatch Agent installation command to the UserData script.

CRITICAL for Terraform Users: When modifying Terraform user_data heredocs, you MUST preserve the EXACT indentation of existing lines. Terraform's <<-EOF syntax strips leading whitespace, but only if indentation is consistent. When adding new bash commands:

  • Count the leading spaces/tabs on existing lines in the heredoc

  • Apply the SAME amount of leading whitespace to all new lines you add

  • Do NOT modify the indentation of any existing lines

If indentation is inconsistent, Terraform will NOT strip the whitespace, causing the deployed script to have leading spaces before #!/bin/bash, which will cause cloud-init to fail.

CDK TypeScript example:

instance.userData.addCommands(
  'dnf install -y amazon-cloudwatch-agent',  // Use dnf for AL2023, yum for AL2
  // ... rest of UserData follows
);

Placement: Add this command early in the UserData script:

  • If system update commands exist (like dnf update -y, apt-get update), add it immediately after those

  • If no system update commands exist, add it at the very beginning of UserData

  • This should come before any application dependency installations or application setup commands

For other Linux distributions: CloudWatch Agent may not be available via the OS package manager. Refer to AWS CloudWatch Agent installation docs for distribution-specific instructions.

Step 5: Modify UserData - Configure CloudWatch Agent

The CloudWatch Agent was installed in Step 4. Now configure it for Application Signals:

CDK TypeScript example:

instance.userData.addCommands(
  '# Create CloudWatch Agent configuration for Application Signals',
  "cat > /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json << 'EOF'",
  '{',
  '  "traces": {',
  '    "traces_collected": {',
  '      "application_signals": {}',
  '    }',
  '  },',
  '  "logs": {',
  '    "metrics_collected": {',
  '      "application_signals": {}',
  '    }',
  '  }',
  '}',
  'EOF',
  '',
  '# Start CloudWatch Agent with Application Signals configuration',
  '/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \\',
  '  -a fetch-config \\',
  '  -m ec2 \\',
  '  -s \\',
  '  -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json',
);
Step 6: Install ADOT Node.js Auto-Instrumentation SDK

Choose based on deployment type AND module format identified in "Before You Start".

Option A: Docker Deployment - Modify Dockerfile

For Docker deployments, modify the Dockerfile in the application directory.

Add the ADOT Node.js SDK installation AFTER any existing npm install or dependency installation commands:

For CommonJS applications:

# Install ADOT Node.js auto-instrumentation (use latest; ServiceEvents requires >=0.12.0)
RUN npm install @aws/aws-distro-opentelemetry-node-autoinstrumentation

For ESM applications:

# Install ADOT Node.js auto-instrumentation with ESM support (use latest; ServiceEvents requires >=0.12.0)
RUN npm install @aws/aws-distro-opentelemetry-node-autoinstrumentation @opentelemetry/instrumentation

Why modify Dockerfile, not UserData: The ADOT package must be installed inside the container image, not on the EC2 host. UserData commands run on the host and won't affect the containerized application.

Option B: Non-Docker Deployment - Modify UserData

For non-Docker deployments, add to UserData AFTER CloudWatch Agent configuration:

For CommonJS applications:

instance.userData.addCommands(
  '# Install ADOT Node.js auto-instrumentation (must run in the app directory so the',
  '# package lands in {{APP_DIR}}/node_modules where Node module resolution finds it)',
  'cd {{APP_DIR}} && npm install @aws/aws-distro-opentelemetry-node-autoinstrumentation',
);

For ESM applications:

instance.userData.addCommands(
  '# Install ADOT Node.js auto-instrumentation with ESM support (run in the app directory)',
  'cd {{APP_DIR}} && npm install @aws/aws-distro-opentelemetry-node-autoinstrumentation @opentelemetry/instrumentation',
);
Step 7: Modify Application Startup to Load ADOT Agent

Choose based on deployment type AND module format identified in "Before You Start".

Option A: Docker Deployment

For Docker deployments, you need to modify both the Dockerfile CMD and the UserData docker run command.

1. Modify Dockerfile CMD to load ADOT agent:

Find the CMD line in your Dockerfile and modify it based on module format:

For CommonJS applications:

# Before:
CMD ["node", "app.js"]

# After:
CMD ["node", "--require", "@aws/aws-distro-opentelemetry-node-autoinstrumentation/register", "app.js"]

For ESM applications:

# Before:
CMD ["node", "app.js"]

# After:
CMD ["node", "--import", "@aws/aws-distro-opentelemetry-node-autoinstrumentation/register", "--experimental-loader=@opentelemetry/instrumentation/hook.mjs", "app.js"]

2. Add environment variables to docker run command in UserData:

Container networking — match the customer's existing setup (minimal change). The example below uses --network host with localhost:4316 endpoints. That pairing is one option, not a hard requirement — the right choice depends on how the container already reaches the host-installed CloudWatch Agent. Don't change the customer's networking model just to instrument; instead pick the variant that fits theirs:

  • Already using --network host (or willing to): keep it, and the localhost:4316 / localhost:2000 endpoints in the example work as-is. Trade-off: host networking shares the host's network namespace (no container isolation), though the agent's ports can stay bound to loopback, unreachable off-host. For production, it is recommended to restrict the OTLP 4316 / proxy 2000 ports via EC2 security groups / host firewall and to avoid co-locating untrusted containers; this guide does not apply those controls, so assess and configure them for your environment.

  • Using a bridge/default network: don't add --network host. Point the endpoints at the host instead — host.docker.internal:4316/:2000 (add --add-host=host.docker.internal:host-gateway on Linux) or the bridge gateway IP. This requires the CloudWatch Agent to listen on a non-loopback address, so it is recommended to restrict those ports with security groups / host firewall.

  • Option 2 — CloudWatch Agent as a sidecar container (most isolated): run the agent as another container on the same user-defined Docker network and target it by name (e.g. cwagent:4316). Nothing binds to host interfaces. This is the same model the ECS guides use; choose it if the customer prefers full container isolation over a host-installed agent.

Find the existing docker run command in UserData. Replace it with (this shows the --network host example — adapt per the networking variant you chose above):

instance.userData.addCommands(
  '# Run container with Application Signals environment variables',
  `docker run -d --name {{APP_NAME}} \\`,
  `  -e OTEL_METRICS_EXPORTER=none \\`,
  `  -e OTEL_LOGS_EXPORTER=none \\`,
  `  -e OTEL_AWS_APPLICATION_SIGNALS_ENABLED=true \\`,
  `  -e OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \\`,
  `  -e OTEL_TRACES_SAMPLER=xray \\`,
  `  -e OTEL_TRACES_SAMPLER_ARG=endpoint=http://localhost:2000 \\`,
  `  -e OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT=http://localhost:4316/v1/metrics \\`,
  `  -e OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4316/v1/traces \\`,
  `  -e OTEL_RESOURCE_ATTRIBUTES=service.name={{SERVICE_NAME}} \\`,
  `  --network host \\`,
  `  {{IMAGE_URI}}`,
);
Option B: Non-Docker Deployment

For non-Docker deployments, set environment variables and modify the node startup command based on module format.

Find the existing command that starts the Node.js application. Add the environment variables BEFORE it and modify the startup command:

For CommonJS applications:

instance.userData.addCommands(
  '# Set OpenTelemetry environment variables',
  'export OTEL_METRICS_EXPORTER=none',
  'export OTEL_LOGS_EXPORTER=none',
  'export OTEL_AWS_APPLICATION_SIGNALS_ENABLED=true',
  'export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf',
  'export OTEL_TRACES_SAMPLER=xray',
  'export OTEL_TRACES_SAMPLER_ARG=endpoint=http://localhost:2000',
  'export OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT=http://localhost:4316/v1/metrics',
  'export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4316/v1/traces',
  'export OTEL_RESOURCE_ATTRIBUTES=service.name={{SERVICE_NAME}}',
  '',
  '# Start application with ADOT agent',
  'cd {{APP_DIR}}',
  'node --require "@aws/aws-distro-opentelemetry-node-autoinstrumentation/register" {{ENTRY_POINT}}',
);

For ESM applications:

instance.userData.addCommands(
  '# Set OpenTelemetry environment variables',
  'export OTEL_METRICS_EXPORTER=none',
  'export OTEL_LOGS_EXPORTER=none',
  'export OTEL_AWS_APPLICATION_SIGNALS_ENABLED=true',
  'export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf',
  'export OTEL_TRACES_SAMPLER=xray',
  'export OTEL_TRACES_SAMPLER_ARG=endpoint=http://localhost:2000',
  'export OTEL_AWS_APPLICATION_SIGNALS_EXPORTER_ENDPOINT=http://localhost:4316/v1/metrics',
  'export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4316/v1/traces',
  'export OTEL_RESOURCE_ATTRIBUTES=service.name={{SERVICE_NAME}}',
  '',
  '# Start application with ADOT agent (ESM)',
  'cd {{APP_DIR}}',
  'node --import "@aws/aws-distro-opentelemetry-node-autoinstrumentation/register" \\',
  '  --experimental-loader=@opentelemetry/instrumentation/hook.mjs \\',
  '  {{ENTRY_POINT}}',
);

Note for systemd services: If the application uses systemd (look for .service files or systemctl commands in UserData), translate the export statements to Environment= directives in the service file, set WorkingDirectory={{APP_DIR}}, and update ExecStart= to use the appropriate node flags. After modifying the service file, add systemctl daemon-reload and systemctl restart <service> to UserData

Completion

Tell the user:

"I've completed the Application Signals enablement for your Node.js application. Here's what I modified:

Files Changed:

  • IAM role: Added CloudWatchAgentServerPolicy

  • UserData: Installed and configured CloudWatch Agent

  • UserData: Installed ADOT Node.js SDK

  • UserData/Service file: Added OpenTelemetry environment variables and node startup flags

  • Dockerfile: Installed ADOT Node.js SDK and modified CMD with node flags (if using Docker)

Next Steps:

  1. Ensure that Application Signals is enabled in AWS account.

  2. Review the changes I made using git diff

  3. Deploy your infrastructure:

    • For CDK: cdk deploy

    • For Terraform: terraform apply

    • For CloudFormation: Deploy your stack

  4. After deployment, wait 5-10 minutes for telemetry data to start flowing

Verification: Once deployed, you can verify Application Signals is working by:

  • Opening the AWS CloudWatch Console

  • Navigating to Application Signals → Services

  • Looking for your service (named: {{SERVICE_NAME}})

  • Checking that traces and metrics are being collected

Monitor Application Health: After enablement, you can monitor your application's operational health using Application Signals dashboards. For more information, see Monitor the operational health of your applications with Application Signals.

Troubleshooting If you encounter any other issues, refer to the CloudWatch APM troubleshooting guide.

Let me know if you'd like me to make any adjustments before you deploy!"

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an MCP server, Needs Python

Security

What our scanner noticed· Medium risk

  • References to credential files
  • Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
AWS
Publisher
AWS (Official)
Source
aws/agent-toolkit-for-aws · skills/specialized-skills/operations-skills/setting-up-cloudwatch-observability · commit 0a218f8
Licence
Apache License 2.0 (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • NOTICE (added by AutoGineer from NOTICE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by AWS or AWS.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download setting-up-cloudwatch-observability.zip