Service Mesh Observability

From the author

Implement comprehensive observability for service meshes including distributed tracing, metrics, and visualization. Use when setting up mesh monitoring, debugging latency issues, or implementing SLOs for service communication.

— the description by Seth Hobson

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured approach to monitoring service meshes like Istio and Linkerd. It covers the three pillars of observability—metrics, traces, and logs—and defines golden signals for effective mesh health monitoring. The guide includes practical templates for Prometheus, Grafana, and Jaeger, alongside best practices for sampling, alerting, and cost management in production environments.

What it does

Defines core observability pillars for service mesh environments
Establishes golden signals for monitoring latency, traffic, errors, and saturation
Provides configuration templates for Istio and Prometheus integration
Offers PromQL queries for tracking request rates, error rates, and P99 latency
Details deployment patterns for Jaeger distributed tracing
Outlines CLI commands for Linkerd visualization and traffic inspection

When to use it

When setting up monitoring for a new service mesh deployment
When debugging latency bottlenecks or intermittent service errors
When defining and implementing Service Level Objectives (SLOs)
When visualizing complex service dependencies within a cluster
When troubleshooting connectivity issues between microservices

Advantages

Standardizes observability practices across different mesh technologies
Includes ready-to-use templates for common monitoring stacks
Provides actionable PromQL queries for immediate implementation
Balances performance monitoring with cost-conscious sampling strategies

Usage examples

“How do I calculate the P99 latency for my services in Istio?”: The skill provides a specific PromQL query using histogram_quantile to calculate P99 latency based on istio_request_duration_milliseconds_bucket metrics.
“What are the best practices for sampling traces in production?”: The guide recommends 100% sampling in development environments and 1-10% in production to balance visibility with storage costs.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/wshobson/service-mesh-observability/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip service-mesh-observability.zip -d .agents/skills

Documentation by Seth Hobson

Complete guide to observability patterns for Istio, Linkerd, and service mesh deployments.

When to Use This Skill

Setting up distributed tracing across services
Implementing service mesh metrics and dashboards
Debugging latency and error issues
Defining SLOs for service communication
Visualizing service dependencies
Troubleshooting mesh connectivity

Core Concepts

1. Three Pillars of Observability

┌─────────────────────────────────────────────────────┐
│                  Observability                       │
├─────────────────┬─────────────────┬─────────────────┤
│     Metrics     │     Traces      │      Logs       │
│                 │                 │                 │
│ • Request rate  │ • Span context  │ • Access logs   │
│ • Error rate    │ • Latency       │ • Error details │
│ • Latency P50   │ • Dependencies  │ • Debug info    │
│ • Saturation    │ • Bottlenecks   │ • Audit trail   │
└─────────────────┴─────────────────┴─────────────────┘

2. Golden Signals for Mesh

Signal	Description	Alert Threshold
Latency	Request duration P50, P99	P99 > 500ms
Traffic	Requests per second	Anomaly detection
Errors	5xx error rate	> 1%
Saturation	Resource utilization	> 80%

Templates and detailed worked examples

Full template library and detailed worked examples live in references/details.md. Read that file when you need the concrete templates.

Best Practices

Do's

Sample appropriately - 100% in dev, 1-10% in prod
Use trace context - Propagate headers consistently
Set up alerts - For golden signals
Correlate metrics/traces - Use exemplars
Retain strategically - Hot/cold storage tiers

Don'ts

Don't over-sample - Storage costs add up
Don't ignore cardinality - Limit label values
Don't skip dashboards - Visualize dependencies
Don't forget costs - Monitor observability costs

Files

ATTRIBUTION.mdAdded by AutoGineer1018 B
LICENSELicence file added by AutoGineer1.0 KiB
SKILL.md2.8 KiB
references/details.md7.5 KiB

SKILL.md