Tempo

Grafana LabsOfficialPublished by Grafana Labs

Skill
Tool integration
Cloud and DevOps
Apache-2.0
Low risk

From the author

Stand up Grafana Tempo as a cost-efficient distributed-tracing backend that only needs object storage, and write TraceQL queries against it. Covers OTLP + Jaeger + Zipkin ingestion, the distributor → live-store → block-builder → object-storage write path, metrics-generator for RED spanmetrics + service graphs, Helm `tempo-distributed` deployment, multi-tenant `X-Scope-OrgID`, TraceQL span / resource / event scopes, structural operators (`>>`, `<<`), `rate()` + `quantile_over_time` metrics, and the traces-to-logs / metrics / profiles datasource links. Use when deploying Tempo, writing a TraceQL query for slow / errored requests, debugging "no traces showing in Explore", sizing queriers / compactors, configuring S3 / GCS / Azure block storage, or wiring trace ↔ log ↔ profile correlation — even when the user says "tracing backend", "find slow requests", "show me the service graph", "store traces in S3", "Jaeger compatible store", or "what called this span" without naming Tempo.

— the description by Grafana Labs

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

Grafana Tempo is a high-scale, distributed tracing backend designed for cost-efficiency by utilizing object storage (S3, GCS, Azure) for long-term data retention. This skill covers the full operational lifecycle of Tempo, including ingestion from OTLP, Jaeger, and Zipkin, deployment via Helm, and advanced query capabilities using TraceQL. It provides guidance on configuring the metrics-generator for RED metrics and service graphs, managing multi-tenancy, and optimizing performance through proper sizing of queriers and compactors.

What it does

Deploys Tempo using Helm for Kubernetes or Docker Compose for local development.
Configures object storage backends for scalable trace storage.
Enables ingestion of OTLP, Jaeger, and Zipkin trace data.
Generates RED metrics and service graphs from trace data.
Executes TraceQL queries to identify slow or errored requests.
Wires trace-to-log and trace-to-profile correlations in Grafana.

When to use it

When setting up a new distributed tracing backend.
When you need to query traces for performance bottlenecks or errors.
When configuring storage backends like S3, GCS, or Azure for trace data.
When debugging missing traces in Grafana Explore.
When implementing multi-tenant tracing environments.
When you need to derive metrics from span data.

Advantages

Highly cost-efficient by leveraging object storage.
Supports multiple ingestion protocols including OTLP and Jaeger.
Powerful TraceQL language for complex structural queries.
Native integration with Grafana for visualization and correlation.
Scalable architecture suitable for production environments.

Usage examples

“How do I find slow requests in my frontend service using Tempo?”: The agent provides a TraceQL query: { resource.service.name = "frontend" && duration > 1s } to filter and identify high-latency spans.
“How can I deploy Tempo on Kubernetes with S3 storage?”: The agent provides the Helm command using the grafana/tempo-distributed chart with specific flags for S3 bucket and region configuration.
“How do I calculate the error rate per service using TraceQL?”: The agent provides the query: { status = error } | rate() by (resource.service.name) to aggregate error metrics.
“What should I do if my traces are not showing up in Grafana?”: The agent suggests verifying that the X-Scope-OrgID header matches between the trace writer and the Grafana datasource configuration.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/grafana/tempo/download
Unzip it into the cross-client skills folder of your project:
```
mkdir -p .agents/skills && unzip tempo.zip -d .agents/skills
```

Documentation by Grafana Labs

Grafana Tempo

Docs: https://grafana.com/docs/tempo/latest/

Cost-efficient distributed tracing. Accepts OTLP / Jaeger / Zipkin / OpenCensus / Kafka. Stores Parquet blocks in S3/GCS/Azure.

Prerequisites

Docker (quick start) or Kubernetes (production)
Object storage bucket (S3/GCS/Azure) for distributed deployments
An OTLP-emitting app or tempo-cli for synthetic traffic
A Grafana stack with a Tempo datasource for querying

Common Workflows

1. Stand up Tempo locally + verify ingestion

# 1. Start the official Docker Compose example
git clone https://github.com/grafana/tempo.git
cd tempo/example/docker-compose/local
mkdir -p tempo-data
docker compose up -d

# 2. Verify readiness
curl -sf http://localhost:3200/ready                              # → "ready"

# 3. Send a synthetic OTLP span (full payload in scratch terminal)
curl -X POST -H 'Content-Type: application/json' \
  http://localhost:4318/v1/traces \
  -d '{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"my-service"}}]},
       "scopeSpans":[{"spans":[{"traceId":"5B8EFFF798038103D269B633813FC700","spanId":"EEE19B7EC3C1B100",
       "name":"my-op","startTimeUnixNano":1689969302000000000,"endTimeUnixNano":1689969302500000000,"kind":2}]}]}]}'

# 4. Verify the trace landed (ingestion-counter > 0 and the trace is fetchable)
curl -s http://localhost:3200/metrics | grep tempo_distributor_spans_received_total | head
curl -s http://localhost:3200/api/v2/traces/5B8EFFF798038103D269B633813FC700 | jq '.batches | length'
# Expect > 0.

# 5. In Grafana → Explore → Tempo, run TraceQL: {resource.service.name="my-service"}

2. Send traces from an app via Alloy

// alloy.river
otelcol.receiver.otlp "default" {
  grpc { endpoint = "0.0.0.0:4317" }
  http { endpoint = "0.0.0.0:4318" }
  output { traces = [otelcol.exporter.otlp.tempo.input] }
}

otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo:4317"
    tls { insecure = true }
  }
}

# Verify Alloy forwarded successfully
curl -s http://localhost:12345/metrics | grep otelcol_exporter_sent_spans
# Then: same Grafana → Explore → Tempo check.

3. Write + run TraceQL

# Slow requests from a service
{ resource.service.name = "frontend" && duration > 1s }

# Server span that has a downstream error (structural)
{ kind = server } >> { status = error }

# Error rate per service (metrics)
{ status = error } | rate() by (resource.service.name)

Full operator + scope cheat sheet, intrinsics list, metric functions: references/traceql.md.

# Via the API
curl -sG --data-urlencode 'q={resource.service.name="frontend" && duration > 1s}' \
  --data-urlencode "start=$(date -d '1h ago' +%s)" --data-urlencode "end=$(date +%s)" \
  http://localhost:3200/api/search | jq '.traces | length'

4. Deploy on Kubernetes (Helm)

helm repo add grafana https://grafana.github.io/helm-charts
helm install tempo grafana/tempo-distributed --version 1.61.3 \
  --set storage.trace.backend=s3 \
  --set storage.trace.s3.bucket=my-tempo-bucket \
  --set storage.trace.s3.region=us-east-1

# Verify every pod is Ready (distributor, ingester, querier, query-frontend, compactor)
kubectl get pods -n default -l app.kubernetes.io/instance=tempo
kubectl port-forward svc/tempo-query-frontend 3200:3200 &
curl -sf http://localhost:3200/ready

Multi-tenancy

multitenancy_enabled: true
# All requests must include header: X-Scope-OrgID: <tenant-id>

Full architecture, ports, performance tuning, metrics-generator config, multi-tenant client snippets, traces-to-logs/metrics/profiles datasource: references/architecture-and-operations.md.

Troubleshooting

/ready → 503 → ingester still joining; check tempo_ingester_* metrics + logs
429 on push → raise max_outstanding_per_tenant or per-tenant ingest limits
"no traces showing in Explore" → confirm X-Scope-OrgID matches between writer and Grafana datasource
TraceQL slow → narrow start/end, add a service.name filter, enable dedicated Parquet columns for hot attributes

Resources

Tempo docs
TraceQL reference
references/traceql.md — full TraceQL cheat sheet
references/architecture-and-operations.md — components, ports, Helm, tuning, datasource links

Files

ATTRIBUTION.mdAdded by AutoGineer830 B
LICENSELicence file added by AutoGineer11.1 KiB
SKILL.md5.5 KiB
references/architecture-and-operations.md5.0 KiB
references/traceql.md2.0 KiB

references/architecture-and-operations.md

Architecture, deployment, performance

Component map

Applications
   |
   v   (OTLP 4317/4318, Jaeger 14250/14268, Zipkin 9411)
[Distributor]  →  [Kafka]  →  [Live Stores]   (recent)
                            →  [Block Builders] (Parquet → object storage)
                            →  [Metrics Generator] (RED metrics → Prometheus)

Grafana → [Query Frontend] → [Querier pool] → Live Stores + Object Storage

Component	Role	Default Port
Distributor	Receives spans, routes by traceID hash	4317 (gRPC), 4318 (HTTP)
Live Store	Buffers recent data on local disk	-
Query Frontend	Query orchestrator + sharding	3200 (HTTP)
Querier	Executes search against storage	-
Compactor	Merges blocks + retention	-
Block Builder	Final Parquet blocks → object storage	-
Metrics Generator	RED metrics from spans	-

Deployment

Docker Compose quick start

git clone https://github.com/grafana/tempo.git
cd tempo/example/docker-compose/local
mkdir tempo-data
docker compose up -d
# Grafana → http://localhost:3000  |  Tempo API → http://localhost:3200

Kubernetes (Helm `tempo-distributed`)

helm repo add grafana https://grafana.github.io/helm-charts
helm install tempo grafana/tempo-distributed --version 1.61.3 \
  --set storage.trace.backend=s3 \
  --set storage.trace.s3.bucket=my-tempo-bucket \
  --set storage.trace.s3.region=us-east-1

Multi-tenancy

multitenancy_enabled: true
# All requests need X-Scope-OrgID: <tenant-id>

OTel Collector header:

exporters:
  otlp:
    headers:
      x-scope-orgid: tenant-id

Grafana datasource:

jsonData:
  httpHeaderName1: "X-Scope-OrgID"
secureJsonData:
  httpHeaderValue1: "tenant-id"

Ports

Port	Protocol	Purpose
3200	HTTP	Tempo API (queries, search, health)
9095	gRPC	Internal component communication
4317	gRPC	OTLP trace ingestion
4318	HTTP	OTLP trace ingestion
14268	HTTP	Jaeger Thrift HTTP
14250	gRPC	Jaeger gRPC
6831 / 6832	UDP	Jaeger Thrift Compact / Binary
9411	HTTP	Zipkin ingestion
7946	TCP/UDP	Memberlist gossip

Performance tuning

Problem	Solution
Slow searches	Scale queriers; scale compactors to reduce block count
High querier memory	Reduce `max_concurrent_queries`; lower `target_bytes_per_job`
High ingester memory	Reduce `max_block_bytes`; lower per-tenant trace limits
Slow attribute queries	Add dedicated Parquet columns for frequent attributes
Cache miss high	Increase cache size; tune `cache_min_compaction_level`
429 rate-limited	Raise `max_outstanding_per_tenant` or per-tenant ingest limits
Memcached errors	Increase memcached conns (`-c 4096`)

Best-practice notes

Replication factor 3 in prod for HA
Object storage required for distributed deployments
Enable dedicated attribute columns for most-queried attributes
Set block retention per-tenant via overrides
Watch tempo_ingester_live_traces for ingester pressure
Follow OpenTelemetry semantic conventions (span.*, resource.*)
~128 attrs/span max; use span linking for batch processing

Metrics from traces

metrics_generator:
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true

overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics]

Processors:

Service Graphs → traces_service_graph_request_total, traces_service_graph_request_failed_total, duration histograms
Span Metrics (RED) → traces_spanmetrics_calls_total, traces_spanmetrics_duration_seconds_* with service / span_name / span_kind / status_code
Local Blocks → enables TraceQL metrics queries on recent data

Grafana datasource (traces-to-logs/metrics/profiles)

datasources:
  - name: Tempo
    type: tempo
    url: http://tempo:3200
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki-uid
        filterByTraceID: true
        tags: [{key: "service.name", value: "app"}]
      tracesToMetrics:
        datasourceUid: prometheus-uid
        tags: [{key: "service.name", value: "service"}]
        queries:
          - name: Error Rate
            query: 'sum(rate(traces_spanmetrics_calls_total{$$__tags, status_code="STATUS_CODE_ERROR"}[5m]))'
      tracesToProfiles:
        datasourceUid: pyroscope-uid
        tags: [{key: "service.name", value: "service_name"}]
      serviceMap:
        datasourceUid: prometheus-uid

HTTP API

# Search
GET /api/search?q={status=error}&limit=20&start=<unix>&end=<unix>

# Trace by ID
GET /api/traces/<traceID>
GET /api/v2/traces/<traceID>

# Tags
GET /api/search/tags
GET /api/search/tag/service.name/values

# TraceQL metrics
GET /api/metrics/query_range?q={status=error}|rate()&start=...&end=...&step=60

# Health
GET /ready

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs network access, Needs a command-line tool

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by

Grafana Labs

Publisher

Grafana Labs (Official)

Source

grafana/skills · skills/grafana-lgtm/tempo · commit 8fb72a5

Licence

Apache License 2.0 (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by Grafana Labs or Grafana Labs.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download tempo.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by Grafana Labs

Grafana Tempo

Prerequisites

Common Workflows

1. Stand up Tempo locally + verify ingestion

2. Send traces from an app via Alloy

3. Write + run TraceQL

4. Deploy on Kubernetes (Helm)

Multi-tenancy

Troubleshooting

Resources

Files

references/architecture-and-operations.md

Architecture, deployment, performance

Component map

Deployment

Docker Compose quick start

Kubernetes (Helm tempo-distributed)

Multi-tenancy

Ports

Performance tuning

Best-practice notes

Metrics from traces

Grafana datasource (traces-to-logs/metrics/profiles)

HTTP API

Requirements and compatibility

Security

Attribution and licence

Related skills

Observability And Instrumentation

Spark Memory Thermal Ops

Gke Alert Configuration

Application Performance Observability Engineer

Distributed Debugging Devops Troubleshooter

Distributed Tracing

Download ZIP

Kubernetes (Helm `tempo-distributed`)