AutoGineer
Generated banner for Distributed Tracing
Generated logo for Distributed Tracing

Distributed Tracing

Seth HobsonCommunityCreated by Seth Hobson

  • Skill
  • Tool integration
  • Cloud and DevOps
  • MIT
  • No issues found
Download ZIP

From the author

Implement distributed tracing with Jaeger and Tempo to track requests across microservices and identify performance bottlenecks. Use when debugging microservices, analyzing request flows, or implementing observability for distributed systems.
— the description by Seth Hobson

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a framework for implementing distributed tracing in microservice architectures. By leveraging Jaeger and Tempo, developers can track end-to-end request journeys, visualize service dependencies, and pinpoint performance bottlenecks. It covers core concepts like traces, spans, and context propagation, alongside practical implementation patterns for Python and Node.js using OpenTelemetry. The guide also includes best practices for sampling, instrumentation, and log correlation to ensure efficient observability with minimal overhead.

What it does

  • Provides patterns for instrumenting microservices with OpenTelemetry.
  • Explains how to deploy Jaeger for trace collection and visualization.
  • Demonstrates how to correlate application logs with trace IDs.
  • Offers best practices for sampling rates and span management.
  • Includes troubleshooting steps for common tracing connectivity and latency issues.
  • Defines standard structures for traces, spans, and metadata tags.

When to use it

  • When debugging latency issues in distributed systems.
  • When you need to visualize service dependencies and request paths.
  • When identifying failure points or error propagation across services.
  • When implementing observability standards for production microservices.

Advantages

  • Enables end-to-end visibility into complex request flows.
  • Helps isolate performance bottlenecks to specific services or database queries.
  • Supports standard instrumentation practices via OpenTelemetry.
  • Provides actionable guidance on balancing observability with system performance.

Usage examples

“How do I instrument my Python Flask application for distributed tracing?”
The skill provides a code example using OpenTelemetry's TracerProvider and JaegerExporter to automatically instrument Flask routes and manual spans.
“What are the best practices for sampling traces in production?”
The skill recommends a 1-10% sampling rate in production and using batch span processors to maintain performance overhead below 1%.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/wshobson/distributed-tracing/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip distributed-tracing.zip -d .agents/skills

Documentation by Seth Hobson

Distributed Tracing

Implement distributed tracing with Jaeger and Tempo for request flow visibility across microservices.

Purpose

Track requests across distributed systems to understand latency, dependencies, and failure points.

When to Use

  • Debug latency issues

  • Understand service dependencies

  • Identify bottlenecks

  • Trace error propagation

  • Analyze request paths

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Best Practices

  1. Sample appropriately (1-10% in production)

  2. Add meaningful tags (user_id, request_id)

  3. Propagate context across all service boundaries

  4. Log exceptions in spans

  5. Use consistent naming for operations

  6. Monitor tracing overhead (<1% CPU impact)

  7. Set up alerts for trace errors

  8. Implement distributed context (baggage)

  9. Use span events for important milestones

  10. Document instrumentation standards

Integration with Logging

Correlated Logs
import logging
from opentelemetry import trace

logger = logging.getLogger(__name__)

def process_request():
    span = trace.get_current_span()
    trace_id = span.get_span_context().trace_id

    logger.info(
        "Processing request",
        extra={"trace_id": format(trace_id, '032x')}
    )

Troubleshooting

No traces appearing:

  • Check collector endpoint

  • Verify network connectivity

  • Check sampling configuration

  • Review application logs

High latency overhead:

  • Reduce sampling rate

  • Use batch span processor

  • Check exporter configuration

Related Skills

  • prometheus-configuration - For metrics

  • grafana-dashboards - For visualization

  • slo-implementation - For latency SLOs

Files

references/details.md

distributed-tracing — detailed patterns and worked examples

Distributed Tracing Concepts

Trace Structure
Trace (Request ID: abc123)
  ↓
Span (frontend) [100ms]
  ↓
Span (api-gateway) [80ms]
  ├→ Span (auth-service) [10ms]
  └→ Span (user-service) [60ms]
      └→ Span (database) [40ms]
Key Components

  • Trace - End-to-end request journey

  • Span - Single operation within a trace

  • Context - Metadata propagated between services

  • Tags - Key-value pairs for filtering

  • Logs - Timestamped events within a span

Jaeger Setup

Kubernetes Deployment
# Deploy Jaeger Operator
kubectl create namespace observability
kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.51.0/jaeger-operator.yaml -n observability

# Deploy Jaeger instance
kubectl apply -f - <<EOF
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
  namespace: observability
spec:
  strategy: production
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elasticsearch:9200
  ingress:
    enabled: true
EOF
Docker Compose
version: "3.8"
services:
  jaeger:
    image: jaegertracing/all-in-one:1.62
    ports:
      - "5775:5775/udp"
      - "6831:6831/udp"
      - "6832:6832/udp"
      - "5778:5778"
      - "16686:16686" # UI
      - "14268:14268" # Collector
      - "14250:14250" # gRPC
      - "9411:9411" # Zipkin
    environment:
      - COLLECTOR_ZIPKIN_HOST_PORT=:9411

Application Instrumentation

OpenTelemetry (Recommended)
Python (Flask)
from opentelemetry import trace
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from flask import Flask

# Initialize tracer
resource = Resource(attributes={SERVICE_NAME: "my-service"})
provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(JaegerExporter(
    agent_host_name="jaeger",
    agent_port=6831,
))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

# Instrument Flask
app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)

@app.route('/api/users')
def get_users():
    tracer = trace.get_tracer(__name__)

    with tracer.start_as_current_span("get_users") as span:
        span.set_attribute("user.count", 100)
        # Business logic
        users = fetch_users_from_db()
        return {"users": users}

def fetch_users_from_db():
    tracer = trace.get_tracer(__name__)

    with tracer.start_as_current_span("database_query") as span:
        span.set_attribute("db.system", "postgresql")
        span.set_attribute("db.statement", "SELECT * FROM users")
        # Database query
        return query_database()
Node.js (Express)
const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");
const { JaegerExporter } = require("@opentelemetry/exporter-jaeger");
const { BatchSpanProcessor } = require("@opentelemetry/sdk-trace-base");
const { registerInstrumentations } = require("@opentelemetry/instrumentation");
const { HttpInstrumentation } = require("@opentelemetry/instrumentation-http");
const {
  ExpressInstrumentation,
} = require("@opentelemetry/instrumentation-express");

// Initialize tracer
const provider = new NodeTracerProvider({
  resource: { attributes: { "service.name": "my-service" } },
});

const exporter = new JaegerExporter({
  endpoint: "http://jaeger:14268/api/traces",
});

provider.addSpanProcessor(new BatchSpanProcessor(exporter));
provider.register();

// Instrument libraries
registerInstrumentations({
  instrumentations: [new HttpInstrumentation(), new ExpressInstrumentation()],
});

const express = require("express");
const app = express();

app.get("/api/users", async (req, res) => {
  const tracer = trace.getTracer("my-service");
  const span = tracer.startSpan("get_users");

  try {
    const users = await fetchUsers();
    span.setAttributes({ "user.count": users.length });
    res.json({ users });
  } finally {
    span.end();
  }
});
Go
package main

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/jaeger"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
)

func initTracer() (*sdktrace.TracerProvider, error) {
    exporter, err := jaeger.New(jaeger.WithCollectorEndpoint(
        jaeger.WithEndpoint("http://jaeger:14268/api/traces"),
    ))
    if err != nil {
        return nil, err
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String("my-service"),
        )),
    )

    otel.SetTracerProvider(tp)
    return tp, nil
}

func getUsers(ctx context.Context) ([]User, error) {
    tracer := otel.Tracer("my-service")
    ctx, span := tracer.Start(ctx, "get_users")
    defer span.End()

    span.SetAttributes(attribute.String("user.filter", "active"))

    users, err := fetchUsersFromDB(ctx)
    if err != nil {
        span.RecordError(err)
        return nil, err
    }

    span.SetAttributes(attribute.Int("user.count", len(users)))
    return users, nil
}

Context Propagation

HTTP Headers
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
tracestate: congo=t61rcWkgMzE
Propagation in HTTP Requests
Python
from opentelemetry.propagate import inject

headers = {}
inject(headers)  # Injects trace context

response = requests.get('http://downstream-service/api', headers=headers)
Node.js
const { propagation } = require("@opentelemetry/api");

const headers = {};
propagation.inject(context.active(), headers);

axios.get("http://downstream-service/api", { headers });

Tempo Setup (Grafana)

Kubernetes Deployment
apiVersion: v1
kind: ConfigMap
metadata:
  name: tempo-config
data:
  tempo.yaml: |
    server:
      http_listen_port: 3200

    distributor:
      receivers:
        jaeger:
          protocols:
            thrift_http:
            grpc:
        otlp:
          protocols:
            http:
            grpc:

    storage:
      trace:
        backend: s3
        s3:
          bucket: tempo-traces
          endpoint: s3.amazonaws.com

    querier:
      frontend_worker:
        frontend_address: tempo-query-frontend:9095
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tempo
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: tempo
          image: grafana/tempo:2.7
          args:
            - -config.file=/etc/tempo/tempo.yaml
          volumeMounts:
            - name: config
              mountPath: /etc/tempo
      volumes:
        - name: config
          configMap:
            name: tempo-config

Sampling Strategies

Probabilistic Sampling
# Sample 1% of traces
sampler:
  type: probabilistic
  param: 0.01
Rate Limiting Sampling
# Sample max 100 traces per second
sampler:
  type: ratelimiting
  param: 100
Adaptive Sampling
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased

# Sample based on trace ID (deterministic)
sampler = ParentBased(root=TraceIdRatioBased(0.01))

Trace Analysis

Finding Slow Requests

Jaeger Query:

service=my-service
duration > 1s
Finding Errors

Jaeger Query:

service=my-service
error=true
tags.http.status_code >= 500
Service Dependency Graph

Jaeger automatically generates service dependency graphs showing:

  • Service relationships

  • Request rates

  • Error rates

  • Average latencies

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs Python

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
Seth Hobson
Publisher
Seth Hobson (Community)
Source
wshobson/agents · plugins/observability-monitoring/skills/distributed-tracing · commit 4bde316
Licence
MIT License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by Seth Hobson or Seth Hobson.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download distributed-tracing.zip