App Observability
Grafana LabsOfficialPublished by Grafana Labs
- Skill
- Tool integration
- Cloud and DevOps
- Apache-2.0
- Low risk
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
This skill provides a unified approach to monitoring modern software stacks using Grafana Cloud. It covers three core pillars: Application Observability for backend APM via OTel spanmetrics, Frontend Observability using the Faro Web SDK for RUM and session replay, and AI Observability via OpenLIT for monitoring LLM performance, costs, and model evaluations. The guide details the necessary plumbing, including Grafana Alloy configuration, OTel resource attribute requirements, and correlation strategies across metrics, logs, and traces.
What it does
- Configures backend APM using OpenTelemetry and Grafana Alloy for RED metrics and service maps.
- Implements frontend RUM and session replay using the Faro Web SDK with React integration.
- Monitors LLM and GenAI applications for token usage, latency, costs, and hallucination evals via OpenLIT.
- Establishes cross-signal correlation between frontend errors, backend traces, and logs.
- Provides troubleshooting steps for missing service map edges and OTLP ingestion issues.
- Offers standardized environment variable and resource attribute configurations for consistent observability.
When to use it
- When setting up full-stack APM for a new or existing service.
- When you need to monitor browser performance and user sessions in a web application.
- When tracking LLM API costs, latency, and model quality in AI-powered applications.
- When debugging missing service map edges or correlating frontend errors to backend traces.
- When migrating to or configuring Grafana Cloud as your primary observability backend.
Advantages
- Unified observability across frontend, backend, and AI components.
- Leverages open standards like OpenTelemetry for vendor-agnostic instrumentation.
- Reduces manual configuration through pre-built dashboards and auto-instrumentation.
- Enables deep correlation between metrics, logs, traces, and profiles.
- Provides clear paths for monitoring modern GenAI workloads alongside traditional services.
Usage examples
- “How do I set up APM for my Node.js service to show up in Grafana Cloud?”
- The agent provides the OTel SDK installation steps, the required resource attributes (service.name, service.namespace), and the Grafana Alloy configuration to forward traces to your Cloud instance.
- “I need to monitor my OpenAI API usage and costs in Grafana.”
- The agent guides you through installing OpenLIT, initializing it in your Python app, and configuring the OTLP exporter to send GenAI metrics to your Grafana Cloud instance.
- “How can I link my React frontend errors to backend traces?”
- The agent explains how to initialize the Faro Web SDK with TracingInstrumentation, ensuring the traceparent header is injected to correlate browser events with backend Tempo traces.
Install
Download the skill as a ZIP:
https://autogineer.com/agent-skills/grafana/app-observability/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip app-observability.zip -d .agents/skills
Documentation by Grafana Labs
Grafana Cloud Application Observability
Docs: https://grafana.com/docs/grafana-cloud/monitor-applications/
Three products that share the same OTLP + Mimir / Loki / Tempo / Pyroscope plumbing:
Application Observability — APM from OTel spanmetrics
Frontend Observability — Faro Web SDK, RUM + session replay
AI Observability — LLM / vector-DB monitoring via OpenLIT
Prerequisites
Grafana Cloud stack + OTLP endpoint + numeric instance ID + API key with
MetricsPublisher+LogsPublisher+TracesPublisherFor APM: app instrumented with OTel SDK; for Frontend: a web app + Faro app key; for AI: Python ≥ 3.10
Grafana Alloy as the local OTLP receiver (recommended)
Common Workflows
1. Stand up APM — Alloy receiver → Grafana Cloud + verify
# 1. Set Cloud creds + start Alloy with config from references/apm.md
export GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
export GRAFANA_CLOUD_INSTANCE_ID=123456
export GRAFANA_CLOUD_API_KEY=glc_eyJ...
alloy fmt /etc/alloy/config.alloy # syntax check
alloy run /etc/alloy/config.alloy
# 2. Verify Alloy is receiving + forwarding
curl -s http://localhost:12345/api/v0/web/components \
| jq '.[] | select(.id|test("otelcol\\.exporter\\.otlphttp"))
| {id, health:.health.state}'
# Expect health.state == "healthy"
curl -s http://localhost:12345/metrics \
| grep -E 'otelcol_(receiver_accepted_spans|exporter_sent_spans)'
# 3. Point your app at Alloy (with required attributes!)
export OTEL_SERVICE_NAME="my-api"
export OTEL_RESOURCE_ATTRIBUTES="service.namespace=myteam,deployment.environment=production"
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
# 4. Verify spans landed in Tempo + spanmetrics generated
# Tempo (TraceQL): { resource.service.name = "my-api" }
# Mimir (PromQL): sum by (job) (rate(traces_spanmetrics_calls_total{service_name="my-api"}[5m]))
# Expect > 0 within ~1 minute.
# 5. Verify it's wired to App Observability
# Grafana → Application → Service Inventory: "my-api" should appear with RED metrics
# Click into it → Service Map edges visible (requires span.kind on outbound calls)Full Alloy block + required resource attributes + spanmetric names + correlation links: references/apm.md.
2. Instrument a React frontend with Faro
# 1. Install
npm install @grafana/faro-react @grafana/faro-web-tracing// 2. initializeFaro with TracingInstrumentation + ReactIntegration (see references/faro.md)
// Push a smoketest event so we have a known signal:
faro.api.pushEvent('faro_smoketest', { ts: Date.now().toString() });# 3. Verify in DevTools Network — POST to /collect returns 202
# (401 → wrong app key; 404 → wrong url region)
# 4. Verify in Grafana Cloud
# - Frontend Observability → your app → Sessions: your session appears
# - LogQL on Loki: {kind="event"} |= "faro_smoketest"
# - With TracingInstrumentation: open the session → the trace ID links to TempoFull React example, CDN setup, session config: references/faro.md.
3. Add AI / LLM observability
pip install openlit==1.42.0# At app startup
import openlit
openlit.init(application_name="my-ai-app", environment="production")
# Your existing OpenAI / Anthropic / Cohere calls now emit OTel spans + metrics.# Env (same OTLP endpoint as APM)
export OTEL_SERVICE_NAME="my-ai-app"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-gateway-<region>.grafana.net/otlp"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(echo -n $ID:$KEY | base64)"
# Verify after a few LLM calls:
# PromQL: sum by (gen_ai_request_model) (rate(gen_ai_usage_input_tokens_total[5m]))
# Dashboard: Grafana → AI Observability → "GenAI Observability" auto-populatesFull OpenLIT install, evals/guards, GenAI metric list, dashboard names: references/ai-observability.md.
Full-stack correlation cheat sheet
| Signal | Product | Storage | Query |
|---|---|---|---|
| RED metrics | App Observability | Mimir | PromQL |
| Traces | Tempo | Tempo | TraceQL |
| Logs | Loki | Loki | LogQL |
| Profiles | Pyroscope | Pyroscope | ProfileQL |
| Browser RUM | Frontend Observability | Loki + Tempo | LogQL / TraceQL |
| LLM metrics | AI Observability | Mimir | PromQL |
Correlation keys: service.name joins all signals; trace exemplars embed trace IDs in metric points; traceID in logs and traceparent injected by Faro for FE → BE linking.
Troubleshooting
Service missing from Service Inventory → missing
service.namespace(job label) ordeployment.environmentresource attributeService Map edges missing →
span.kindnot set on outbound calls (must be CLIENT) or inbound (SERVER)Faro
/collectreturns 401 → wrong app key; 404 → region in URL doesn't match the Faro appNo GenAI metrics → confirm OpenLIT version matches OTel semantic-conv version expected by Cloud; verify auth with curl as in workflow #3
References
references/apm.md— APM essentials: how RED metrics are generated, required OTel resource attributes, Alloy config, correlation linksreferences/apm-setup.md— deep dive: full per-language OTel SDK setup (Node / Python / Java / Go), span-metrics options, complete Alloy configreferences/faro.md— Faro essentials: SDK init, instrumentations, session replayreferences/frontend-observability.md— deep dive: full Faro SDK reference, React/Vue/Angular integration, custom events, source mapsreferences/ai-observability.md— OpenLIT auto-instrumentation for OpenAI / Anthropic / Bedrock / Vertex AI
Resources
Files
- ATTRIBUTION.mdAdded by AutoGineer855 B
- LICENSELicence file added by AutoGineer11.1 KiB
- SKILL.md7.2 KiB
- references/ai-observability.md2.4 KiB
- references/apm-setup.md23.1 KiB
- references/apm.md3.2 KiB
- references/faro.md3.2 KiB
- references/frontend-observability.md14.7 KiB
references/frontend-observability.md
Grafana Frontend Observability - Faro SDK Reference
What is Faro?
Grafana Faro is an open-source JavaScript/TypeScript SDK for Real User Monitoring (RUM). It instruments browser frontend applications to capture observability signals and correlate them with backend telemetry.
GitHub: https://github.com/grafana/faro-web-sdk Docs: https://grafana.com/docs/grafana-cloud/monitor-applications/frontend-observability/
Package Structure
| Package | Purpose |
|---|---|
@grafana/faro-core | Core SDK foundation: signals, transports, API surface |
@grafana/faro-web-sdk | Web instrumentations, web vitals, fetch/XHR, transports |
@grafana/faro-web-tracing | OpenTelemetry-JS distributed tracing for browsers |
@grafana/faro-react | React error boundary, router integration, component profiler |
Installation
NPM / Yarn
# Core SDK only
npm install @grafana/faro-web-sdk
# With distributed tracing
npm install @grafana/faro-web-sdk @grafana/faro-web-tracing
# With React integration
npm install @grafana/faro-web-sdk @grafana/faro-web-tracing @grafana/faro-reactCDN (no bundler)
The library is distributed as IIFE and exposes a global GrafanaFaroWebSdk:
<!-- Web SDK -->
<script src="https://unpkg.com/@grafana/faro-web-sdk@latest/dist/library/faro-web-sdk.iife.js"></script>
<!-- Web Tracing (optional) -->
<script src="https://unpkg.com/@grafana/faro-web-tracing@latest/dist/library/faro-web-tracing.iife.js"></script>
<script>
const { initializeFaro, getWebInstrumentations } = GrafanaFaroWebSdk;
const { TracingInstrumentation } = GrafanaFaroWebTracing;
initializeFaro({
url: 'https://faro-collector-prod-<region>.grafana.net/collect/<app-key>',
app: { name: 'my-app', version: '1.0.0' },
instrumentations: [
...getWebInstrumentations(),
new TracingInstrumentation(),
],
});
</script>Core Configuration - initializeFaro()
interface FaroConfig {
// Required: Collector endpoint URL from Grafana Cloud Frontend Observability
url: string;
// Application metadata
app: {
name: string; // Required - identifies your app
version?: string; // Recommended - for release tracking
environment?: string; // e.g. 'production', 'staging'
namespace?: string; // Optional grouping
};
// Instrumentations to enable
instrumentations?: Instrumentation[];
// Session tracking options
sessionTracking?: {
enabled?: boolean; // default: true
persistent?: boolean; // survives tab close; default: false
maxSessionPersistenceTime?: number; // ms; default: 4 hours
samplingRate?: number; // 0-1; default: 1 (100%)
generateSessionId?: () => string; // custom ID generator
onSessionChange?: (oldSession: Session | null, newSession: Session) => void;
};
// Optional: override or add metadata
user?: {
id?: string;
username?: string;
email?: string;
attributes?: Record<string, string>;
};
// Optional: global attributes added to all signals
globalObjectKey?: string; // window key; default: '__faroBridge'
// Batch send options
batchSendCount?: number; // signals per batch; default: 50
batchSendTimeout?: number; // ms before forced flush; default: 250
}Minimal Setup
import { initializeFaro, getWebInstrumentations } from '@grafana/faro-web-sdk';
initializeFaro({
url: 'https://faro-collector-prod-us-east-0.grafana.net/collect/abc123',
app: {
name: 'my-app',
version: '1.0.0',
environment: 'production',
},
instrumentations: [...getWebInstrumentations()],
});getWebInstrumentations() Options
getWebInstrumentations({
// Capture console.log, console.warn, console.error as Faro logs
captureConsole: true, // default: false
captureConsoleDisabledLevels: ['log'], // exclude certain levels
// Capture page visibility changes
capturePageVisibility: true, // default: true
})What's automatically included in getWebInstrumentations():
PerformanceInstrumentation- navigation timing, resource loading, web vitalsErrorsInstrumentation- unhandled errors and promise rejectionsConsoleInstrumentation- console capture (ifcaptureConsole: true)SessionInstrumentation- session lifecycle trackingViewInstrumentation- page view tracking
Web Vitals Captured
Faro v2 uses Web Vitals v5:
LCP (Largest Contentful Paint) - loading performance
CLS (Cumulative Layout Shift) - visual stability
INP (Interaction to Next Paint) - interactivity (replaces FID)
FCP (First Contentful Paint)
TTFB (Time to First Byte)
Manual API Usage
const faro = initializeFaro({ ... });
// Push a log message
faro.api.pushLog(['User completed checkout'], {
level: LogLevel.INFO,
context: { orderId: '12345' },
});
// Push an error manually
faro.api.pushError(new Error('Payment gateway timeout'), {
type: 'NetworkError',
context: { endpoint: '/api/pay' },
});
// Push a custom event
faro.api.pushEvent('feature_used', {
feature: 'dark_mode',
source: 'settings_page',
});
// Push a measurement (custom metric)
faro.api.pushMeasurement({
type: 'custom_timing',
values: {
checkout_duration_ms: 1234,
},
});
// Set user context
faro.api.setUser({
id: 'user-456',
username: 'jane.doe',
attributes: { plan: 'premium' },
});
// Set custom global attributes
faro.api.setSession({ attributes: { experiment: 'checkout-v2' } });Distributed Tracing Setup
Adds OpenTelemetry browser tracing to correlate frontend spans with backend traces.
import { initializeFaro, getWebInstrumentations } from '@grafana/faro-web-sdk';
import { TracingInstrumentation } from '@grafana/faro-web-tracing';
initializeFaro({
url: '...',
app: { name: 'my-app' },
instrumentations: [
...getWebInstrumentations(),
new TracingInstrumentation({
// Propagate trace context to these URLs (supports regex)
propagateTraceHeaderCorsUrls: [
/https:\/\/api\.myapp\.com\/.*/,
'https://internal.service.com',
],
// Custom sampling rate for traces (0-1)
// samplingRate: 0.5,
}),
],
});When enabled:
traceparentandtracestateheaders are injected into all matching fetch/XHR requestsBrowser creates spans for each HTTP call
Traces appear in Grafana Tempo correlated to backend spans
Session ID is propagated as a baggage item
React Integration
React Router v6 (data router API - recommended)
import { initializeFaro, getWebInstrumentations } from '@grafana/faro-web-sdk';
import { TracingInstrumentation } from '@grafana/faro-web-tracing';
import {
createReactRouterV6DataOptions,
ReactIntegration,
withFaroRouterInstrumentation,
} from '@grafana/faro-react';
import { createBrowserRouter, RouterProvider } from 'react-router-dom';
// Must call BEFORE rendering
initializeFaro({
url: 'https://faro-collector-prod-<region>.grafana.net/collect/<key>',
app: { name: 'my-react-app', version: '2.0.0', environment: 'production' },
instrumentations: [
...getWebInstrumentations({ captureConsole: true }),
new TracingInstrumentation(),
new ReactIntegration({
router: createReactRouterV6DataOptions({}),
}),
],
});
// Instrument the router
const router = withFaroRouterInstrumentation(
createBrowserRouter([
{ path: '/', element: <HomePage /> },
{ path: '/products/:id', element: <ProductPage /> },
{ path: '/checkout', element: <CheckoutPage /> },
])
);
function App() {
return <RouterProvider router={router} />;
}React Router v6 (non-data router - Routes component)
import {
createReactRouterV6Options,
ReactIntegration,
FaroRoutes,
} from '@grafana/faro-react';
import { useLocation, useNavigationType, createRoutesFromChildren, matchRoutes } from 'react-router-dom';
initializeFaro({
// ...
instrumentations: [
...getWebInstrumentations(),
new TracingInstrumentation(),
new ReactIntegration({
router: createReactRouterV6Options({
useLocation,
useNavigationType,
createRoutesFromChildren,
matchRoutes,
}),
}),
],
});
// In your component tree, use FaroRoutes instead of Routes:
function AppRoutes() {
return (
<FaroRoutes>
<Route path="/" element={<HomePage />} />
<Route path="/about" element={<AboutPage />} />
</FaroRoutes>
);
}React Router v4/v5
import {
createReactRouterV5Options,
ReactIntegration,
} from '@grafana/faro-react';
import { createBrowserHistory } from 'history';
import { Router } from 'react-router-dom';
const history = createBrowserHistory();
initializeFaro({
// ...
instrumentations: [
...getWebInstrumentations(),
new TracingInstrumentation(),
new ReactIntegration({
router: createReactRouterV5Options({ history }),
}),
],
});
// Wrap app with Router using the same history instance
function App() {
return (
<Router history={history}>
<YourRoutes />
</Router>
);
}React Error Boundary
import { FaroErrorBoundary, withErrorBoundary } from '@grafana/faro-react';
// As a component
function App() {
return (
<FaroErrorBoundary
fallback={<ErrorFallback />}
pushErrorAsFaro={true} // auto-pushes to Faro (default: true)
>
<MainContent />
</FaroErrorBoundary>
);
}
// As a HOC
const SafeComponent = withErrorBoundary(MyComponent, {
fallback: <p>Something went wrong</p>,
});Session Tracking Details
import {
initializeFaro,
getWebInstrumentations,
SessionInstrumentation,
} from '@grafana/faro-web-sdk';
initializeFaro({
url: '...',
app: { name: 'my-app' },
sessionTracking: {
enabled: true,
// Persist session across browser restarts (uses localStorage)
persistent: true,
// Max time a persistent session lives (milliseconds)
// Default: 4 hours
maxSessionPersistenceTime: 4 * 60 * 60 * 1000,
// Sample only 50% of sessions to reduce data volume
samplingRate: 0.5,
// Custom session ID (useful for server-side session correlation)
generateSessionId: () => crypto.randomUUID(),
// Hook for analytics or logging on session rotation
onSessionChange: (oldSession, newSession) => {
console.log(`New session: ${newSession.id}`);
analytics.track('session_started', { id: newSession.id });
},
},
instrumentations: [...getWebInstrumentations()],
});Session Replay
Session Replay records DOM mutations using rrweb, allowing visual playback of user sessions.
Setup steps:
In Grafana Cloud > Connections > Frontend Observability > Web SDK Configuration
Enable "Session Replay" toggle to get a replay-enabled collector URL
Install the SDK - the collector key controls what's recorded server-side
The basic initializeFaro setup works - session replay is controlled by the collector URL/key:
initializeFaro({
// Use the replay-enabled collector URL from Grafana Cloud UI
url: 'https://faro-collector-prod-<region>.grafana.net/collect/<replay-enabled-key>',
app: { name: 'my-app', version: '1.0.0' },
instrumentations: [...getWebInstrumentations()],
sessionTracking: {
persistent: true,
samplingRate: 0.1, // Record 10% of sessions for replay (cost control)
},
});In Grafana Cloud Frontend Observability:
Sessions panel shows timeline of user actions
"Play" button on any session opens the replay viewer
Filters: by error, by duration, by user, by URL
Getting Collector URL
The collector URL is unique per application and environment:
Navigate to Grafana Cloud stack
Left menu: Connections > search "Frontend Observability"
Click the Frontend Observability tile
Click Web SDK Configuration tab
Copy the
urlfield shown (format:https://faro-collector-prod-<region>.grafana.net/collect/<app-key>)
The same page shows:
Your App Name (pre-configured)
Sample rate settings
Replay toggle
CDN script tags for quick integration
Fetch Instrumentation Customization
Control which requests are traced and what headers are captured:
import { FetchInstrumentation } from '@grafana/faro-web-sdk';
initializeFaro({
// ...
instrumentations: [
new FetchInstrumentation({
// Capture specific request headers
requestHeaders: ['x-request-id', 'x-correlation-id'],
// Capture specific response headers
responseHeaders: ['x-trace-id'],
// Ignore certain URLs from tracking
ignoredUrls: [/.*\/health/, 'https://analytics.internal/'],
}),
],
});Data Flow Architecture
Browser App
|
| (HTTP POST, batched)
v
Faro Collector (Grafana Cloud)
|
|-- Logs -------> Grafana Loki
|-- Traces -----> Grafana Tempo
|-- Metrics ----> Grafana Mimir
|
Visualized in Grafana Cloud Frontend Observability dashboardsAlternative with self-hosted Alloy:
Browser App --> Grafana Alloy (faro.receiver) --> Grafana Cloud backendsAlloy Configuration (self-hosted)
If routing through your own Alloy instance:
faro.receiver "default" {
server {
listen_address = "0.0.0.0"
listen_port = 12347
cors_allowed_origins = ["https://myapp.com"]
api_key = env("FARO_API_KEY")
max_allowed_payload_size = "10MiB"
}
sourcemaps {
download = true
upload {
external_url = "https://myfrontend.cdn.com"
}
}
output {
logs = [loki.write.grafana_cloud.receiver]
traces = [otelcol.exporter.otlphttp.grafana_cloud.input]
}
}Correlation with Backend
Frontend-to-backend trace correlation requires:
Faro has
TracingInstrumentationenabledBackend accepts W3C
traceparentheadersBoth frontend and backend send to the same Grafana Cloud Tempo instance
In Tempo trace view, the root span will show session.id attribute linking to the Faro session.
In Frontend Observability, errors show "View trace" button when a traceId is present.
Upgrading to Faro v2
Faro v2 changes from v1:
Web Vitals v5: FID removed, INP added as Core Web Vital
Console instrumentation: Simplified -
captureConsoleis now an option ingetWebInstrumentations()Tracing APIs: Some deprecated span attributes removed
Session attributes: New structured approach
Prebuilt Dashboards in Grafana Cloud
Frontend Observability auto-provisions:
Overview - session count, error rate, web vitals scores
Sessions - session list with replay links
Errors - error grouping, stack traces, affected sessions
Performance - web vitals trends, resource loading, navigation timing
User Journey - page flow, drop-off points
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs network access, Needs an API key, Needs Python, Needs Node.js
Security
What our scanner noticed· Low risk
- Instructions that use the network
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- Grafana Labs
- Publisher
- Grafana Labs (Official)
- Source
- grafana/skills · skills/grafana-cloud/app-observability · commit 8fb72a5
- Licence
- Apache License 2.0 (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by Grafana Labs or Grafana Labs.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download app-observability.zip