Managing Amazon Msk

Monitor and Alarm MSK Clusters

Monitoring Levels

MSK Provisioned clusters support 4 monitoring levels. Each level includes all metrics from the previous level.

Recommendation: Use PER_BROKER as the minimum for production clusters. Use PER_TOPIC_PER_BROKER if you need per-topic throughput visibility. Use PER_TOPIC_PER_PARTITION only if you need partition-level consumer lag granularity.

Key metrics NOT available on Express (do not alarm on these for Express clusters): UnderReplicatedPartitions, OfflinePartitionsCount, KafkaDataLogsDiskUsed, HeapMemoryAfterGC, BurstBalance, VolumeQueueLength, BwInAllowanceExceeded, BwOutAllowanceExceeded. Express uses ProduceThrottleTime, FetchThrottleTime, and StorageUsed instead.

For the full metric list per broker type and monitoring level, search AWS docs for "MSK CloudWatch metrics Standard brokers" or "MSK CloudWatch metrics Express brokers".

Update monitoring level:

# Get the cluster's current revision version first (this is an opaque revision string, NOT a Kafka version number):
# aws kafka describe-cluster-v2 --cluster-arn <cluster-arn> --query 'ClusterInfo.CurrentVersion' --output text

aws kafka update-monitoring --cluster-arn <cluster-arn> --current-version <cluster-current-version> \
  --enhanced-monitoring PER_BROKER

Follow SNS security best practices when wiring alarm actions to an SNS topic (e.g. --alarm-actions for aws cloudwatch put-metric-alarm calls).

CloudWatch Controller Metric Reporting Issues

Metrics like GlobalPartitionCount count the partitions across all topics in the cluster, excluding replicas, and require the Maximum statistic to interpret correctly. CloudWatch's default Average therefore spreads the true value across all brokers and returns roughly true_count / broker_count — so on an N-broker cluster it reads about 1/N of reality. Use the Maximum to see the true cluster-wide metrics.

MSK cluster-level metrics (DEFAULT level, dimension: Cluster Name) are emitted by the controller broker. Every broker emits them, but only the active controller reports the real value while the other brokers report 0. The same controller-metric behavior applies to GlobalTopicCount, OfflinePartitionsCount, and PreferredReplicaImbalanceCount: never use Average. Use Maximum to read the true value; Sum also works for a simple "> 0" breach check (the non-controller zeros contribute nothing), which is why the OfflinePartitionsCount critical alarm below is expressed as Sum > 0.

Recommended Alarms

Critical alarms (set these for every cluster)

Performance alarms (most require PER_BROKER level)

Consumer lag alarms (DEFAULT level)

Dimensions for all four: Cluster Name, Consumer Group, Topic. Do NOT add Broker ID — consumer lag is not a per-broker metric.

Do NOT invent metric names like ConsumerLag, RecordLag, or EstimatedMaximumLag — those are not real MSK metrics.

Per-broker capacity alarms

Standard brokers do not have MSK-enforced per-broker throughput quotas — throughput is bounded by EBS volume type and EC2 network bandwidth instead. For Standard throughput alarming, use BwInAllowanceExceeded and BwOutAllowanceExceeded in the performance alarms section above.

Express-specific alarms

StorageUsed alarm — choose ONE of the two patterns:

1. Default (retention unknown, or you want to catch unusual growth): anomaly-detection alarm on StorageUsed. CloudWatch builds a per-cluster baseline and alerts when usage falls outside the predicted band — no static threshold needed and it adapts as the workload changes. Use LessThanLowerOrGreaterThanUpperThreshold with ANOMALY_DETECTION_BAND(m1, 2). This is the recommended default for new clusters where you don't yet know the steady-state size.

   Copy

   aws cloudwatch put-metric-alarm \
     --alarm-name MSK-<cluster>-StorageUsed-Anomaly \
     --metrics '[
       {"Id":"m1","MetricStat":{"Metric":{"Namespace":"AWS/Kafka","MetricName":"StorageUsed","Dimensions":[{"Name":"Cluster Name","Value":"<cluster>"}]},"Period":300,"Stat":"Maximum"},"ReturnData":true},
       {"Id":"ad1","Expression":"ANOMALY_DETECTION_BAND(m1, 2)","Label":"StorageUsed (expected)"}
     ]' \
     --threshold-metric-id ad1 \
     --comparison-operator LessThanLowerOrGreaterThanUpperThreshold \
     --evaluation-periods 3 --datapoints-to-alarm 3 \
     --treat-missing-data notBreaching

2. Static threshold (retention is known): Maximum > SUM(BytesInPerSec) × retention_seconds × 3 for 15 min (RF=3 on Express). Use this once the cluster has been running long enough to know steady-state ingress and retention is locked.

CloudWatch Dimension Reference

All MSK metrics use namespace AWS/Kafka.

Important: Consumer group names MUST use ASCII characters only. Non-ASCII characters cause consumer lag metrics to be silently dropped from CloudWatch.

Dashboard Construction

Create a CloudWatch dashboard with these sections:

1. Cluster Health: ActiveControllerCount, OfflinePartitionsCount (Standard), UnderReplicatedPartitions (Standard); for Express add ProduceThrottleTime and FetchThrottleTime

2. CPU: CpuUser per broker (line chart, one series per broker to spot AZ skew)

3. Throughput: BytesInPerSec and BytesOutPerSec per broker

4. Storage: KafkaDataLogsDiskUsed per broker (Standard); StorageUsed cluster-level (Express)

5. Consumer Lag: MaxOffsetLag per consumer group/topic

6. Request Performance: RequestHandlerAvgIdlePercent, ProduceTotalTimeMsMean per broker

Use metric math for composite CPU alarm: m1 + m2 where m1 is defined as CpuUser and m2 is defined as CpuSystem for the target broker.