Instrument LLM Analytics
PostHogOfficialCreated by PostHog
- Skill
- Tool integration
- Data and AI
- MIT
- Low risk
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
This skill provides a structured approach to instrumenting LLM-powered applications with PostHog AI Observability. It guides the agent through detecting the existing LLM stack, installing necessary SDKs, and configuring event capture for token counts, latency, and costs. The skill supports a wide range of LLM providers and frameworks, ensuring that observability is added with minimal disruption to existing codebases.
What it does
- Detects LLM providers and frameworks in the codebase.
- Installs the PostHog SDK and required integration packages.
- Instruments LLM calls to capture tokens, latency, and costs.
- Configures environment variables for secure credential management.
- Associates LLM generations with specific user IDs.
- Provides fallback manual capture methods for unsupported providers.
When to use it
- After implementing new LLM features in an application.
- During code reviews to ensure proper observability coverage.
- When setting up AI monitoring for the first time in a project.
- When needing to track costs and performance metrics for AI model usage.
Advantages
- Standardizes observability across diverse LLM stacks.
- Ensures minimal impact on existing application architecture.
- Provides granular visibility into token usage and latency.
- Follows best practices for environment variable management.
- Supports a broad ecosystem of AI frameworks and providers.
Usage examples
- “Add PostHog AI observability to my LangChain project.”
- The agent will scan the project for LangChain dependencies, install the necessary PostHog packages, and instrument the LLM chains to report usage metrics to PostHog.
- “I need to track costs and latency for my OpenAI API calls.”
- The agent will identify the OpenAI implementation, configure the appropriate middleware or callback, and ensure all generations are captured with the required metadata.
Install
Download the skill as a ZIP:
https://autogineer.com/agent-skills/posthog/instrument-llm-analytics/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip instrument-llm-analytics.zip -d .agents/skills
Documentation by PostHog
Add PostHog AI Observability
Use this skill to add PostHog AI Observability that traces AI model usage in new or changed code. Use it after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. If PostHog is not yet installed, this skill also covers initial SDK setup. Supports any provider or framework.
Supported providers: OpenAI, Azure OpenAI, Anthropic, Google, Cohere, Mistral, Perplexity, DeepSeek, Groq, Together AI, Fireworks AI, xAI, Cerebras, Hugging Face, Ollama, OpenRouter.
Supported frameworks: LangChain, LlamaIndex, CrewAI, AutoGen, DSPy, LangGraph, Pydantic AI, Vercel AI, LiteLLM, Instructor, Semantic Kernel, Mirascope, Mastra, SmolAgents, OpenAI Agents.
Proxy/gateway: Portkey, Helicone.
Instructions
Follow these steps IN ORDER:
STEP 1: Analyze the codebase and detect the LLM stack.
Look for LLM provider SDKs (openai, anthropic, google-generativeai, etc.) and AI frameworks (langchain, llamaindex, crewai, etc.) in dependency files and imports.
Look for lockfiles to determine the package manager.
Check for existing PostHog or observability setup. If PostHog is already installed and LLM tracing is configured, skip to STEP 4 to add tracing for any new LLM calls.
STEP 2: Research instrumentation. (Skip if PostHog LLM tracing is already set up.) 2.1. Find the reference file below that matches the detected provider or framework — it is the source of truth for callback setup, middleware configuration, and event capture. Read it now. 2.2. If no reference matches, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.
STEP 3: Install the PostHog SDK. (Skip if PostHog is already set up.)
Add the PostHog SDK and any required callback/integration packages.
Do not manually edit dependency files — use the package manager's install command.
STEP 4: Add LLM tracing.
Instrument LLM calls to capture input tokens, output tokens, model name, latency, and costs for every generation.
Follow the provider-specific reference for the exact callback/middleware setup.
Do not alter the fundamental architecture of existing files. Make additions minimal and targeted.
You must read a file immediately before attempting to write it.
STEP 5: Link to users.
Associate LLM generations with identified users via distinct IDs when possible.
STEP 6: Set up environment variables.
Check if the project already has PostHog environment variables configured (e.g. in
.env,.env.local, or framework-specific env files). If valid values already exist, skip this step.If the PostHog project token is missing, use the PostHog MCP server's
projects-gettool to retrieve the project'sapi_token. If multiple projects are returned, ask the user which project to use. If the MCP server is not connected or not authenticated, ask the user for their PostHog project token instead.For the PostHog host URL: check the
projects-getMCP response for aregionfield —USmaps tohttps://us.i.posthog.com,EUmaps tohttps://eu.i.posthog.com. If the region is not available from the MCP response or from existing project configuration, ask the user: "Are you on PostHog US Cloud or EU Cloud?" Do not assume US Cloud.Write these values to the appropriate env file using the framework's naming convention.
Reference these environment variables in code instead of hardcoding them.
Reference files
references/openai.md- Openai observability installationreferences/azure-openai.md- Azure openai observability installationreferences/README.md- PostHog.ai for .netreferences/anthropic.md- Anthropic ai observability installationreferences/google.md- Google ai observability installationreferences/cohere.md- Cohere ai observability installationreferences/mistral.md- Mistral ai observability installationreferences/perplexity.md- Perplexity ai observability installationreferences/deepseek.md- Deepseek ai observability installationreferences/groq.md- Groq ai observability installationreferences/together-ai.md- Together ai observability installationreferences/fireworks-ai.md- Fireworks ai observability installationreferences/xai.md- Xai observability installationreferences/cerebras.md- Cerebras ai observability installationreferences/hugging-face.md- Hugging face ai observability installationreferences/ollama.md- Ollama ai observability installationreferences/openrouter.md- Openrouter ai observability installationreferences/langchain.md- Langchain ai observability installationreferences/llamaindex.md- Llamaindex ai observability installationreferences/crewai.md- Crewai observability installationreferences/autogen.md- Autogen ai observability installationreferences/dspy.md- Dspy ai observability installationreferences/langgraph.md- Langgraph ai observability installationreferences/pydantic-ai.md- Pydantic ai observability installationreferences/vercel-ai.md- Vercel ai SDK observability installationreferences/litellm.md- Litellm ai observability installationreferences/instructor.md- Instructor ai observability installationreferences/semantic-kernel.md- Semantic kernel ai observability installationreferences/mirascope.md- Mirascope ai observability installationreferences/mastra.md- Mastra ai observability installationreferences/smolagents.md- Smolagents ai observability installationreferences/openai-agents.md- Openai agents SDK observability installationreferences/portkey.md- Portkey ai observability installationreferences/helicone.md- Helicone ai observability installationreferences/manual-capture.md- Manual capture ai observability installationreferences/basics.md- Ai observability basicsreferences/traces.md- Tracesreferences/calculating-costs.md- Calculating llm costsreferences/COMMANDMENTS.md- Framework-specific rules the integration must follow
Each provider reference contains installation instructions, SDK setup, and code examples specific to that provider or framework. Find the reference that matches the user's stack.
If the user's provider isn't listed, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.
Key principles
Environment variables: Always use environment variables for PostHog and LLM provider keys. Never hardcode them.
Minimal changes: Add LLM analytics alongside existing LLM calls. Don't replace or restructure existing code.
Trace all generations: Capture input tokens, output tokens, model name, latency, and costs for every LLM call.
Link to users: Associate LLM generations with identified users via distinct IDs when possible.
One provider at a time: Only instrument the provider(s) the user is actually using. Don't add instrumentation for providers not present in the codebase.
Files
- ATTRIBUTION.mdAdded by AutoGineer958 B
- LICENSELicence file added by AutoGineer1.0 KiB
- SKILL.md7.3 KiB
- references/COMMANDMENTS.md1.5 KiB
- references/README.md543 B
- references/anthropic.md27.1 KiB
- references/autogen.md7.7 KiB
- references/azure-openai.md10.7 KiB
- references/basics.md5.6 KiB
- references/calculating-costs.md8.8 KiB
- references/cerebras.md8.3 KiB
- references/cohere.md8.3 KiB
- references/crewai.md7.2 KiB
- references/deepseek.md8.2 KiB
- references/dspy.md7.4 KiB
- references/fireworks-ai.md8.4 KiB
- references/google.md10.0 KiB
- references/groq.md8.2 KiB
- references/helicone.md8.3 KiB
- references/hugging-face.md8.4 KiB
- references/instructor.md10.5 KiB
- references/langchain.md8.5 KiB
- references/langgraph.md8.3 KiB
- references/litellm.md9.5 KiB
- references/llamaindex.md8.1 KiB
- references/manual-capture.md23.7 KiB
- references/mastra.md6.7 KiB
- references/mirascope.md8.0 KiB
- references/mistral.md8.3 KiB
- references/ollama.md8.2 KiB
- references/openai-agents.md8.2 KiB
- references/openai.md10.2 KiB
- references/openrouter.md8.3 KiB
- references/perplexity.md8.3 KiB
- references/portkey.md8.2 KiB
- references/pydantic-ai.md7.4 KiB
- references/semantic-kernel.md7.7 KiB
- references/smolagents.md7.5 KiB
- references/together-ai.md8.3 KiB
- references/traces.md8.5 KiB
- references/vercel-ai.md10.0 KiB
- references/xai.md8.2 KiB
references/basics.md
AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt
AI Observability basics
Canonical URL: https://posthog.com/docs/ai-observability/basics
On this page
How LLM calls become events
The $ai_ prefix
Generation properties
Grouping related events
See it in action
This page covers how your LLM calls become analytics in PostHog and defines key concepts for AI Observability.
How LLM calls become events
PostHog's AI Observability works by wrapping your existing LLM provider's SDK to capture requests and responses. Your API calls still go directly to your provider, but the wrapper extracts metadata from each call and sends it to PostHog as an event.
graph LR Call["LLM call"] --> Wrapper["Wrapped SDK"] Wrapper --> Provider["LLM provider API"] Wrapper --> Event["Event"] Event --> PostHog["PostHog"]
Events sent to PostHog are called generations. A generation represents a single LLM call. For example, when you send a prompt to Claude and get a response back, that's one generation.
Generations are represented using the event name $ai_generation. Each generation captures the model, provider, input, output, token counts, latency, and cost.
The $ai_ prefix
Every event name that starts with $ai_ is an AI Observability event. This includes the names the SDKs send, like $ai_generation, $ai_span, $ai_trace, and $ai_embedding, and any other $ai_ name you send yourself.
PostHog processes every $ai_ event as an AI Observability event rather than a Product Analytics event:
It doesn't create or update a person profile.
$setand$set_onceon the event are ignored.The full event, including large properties like
$ai_inputand$ai_output, is kept for 30 days. After that, only a copy without those properties remains. See data retention.It's billed as an AI Observability event, not as a Product Analytics event.
Don't use the $ai_ prefix for events that need person profiles or Product Analytics pricing.
Generation properties
Tokens and costs
Tokens are the units LLMs use to process text. LLM providers charge based on token usage:
Input tokens are tokens used in the message you send to an LLM
Output tokens are tokens used in the message you receive from an LLM
PostHog automatically calculates costs by matching your model and provider against pricing data. We use OpenRouter's pricing as our primary source, with fallback to manually maintained pricing for additional models.
You can also set custom pricing if you have negotiated rates or use unsupported models.
Message roles
When you send messages to an LLM, each message has a role that tells the model how to interpret it:
| Role | Purpose | Example |
|---|---|---|
system | Instructions that define the assistant's behavior | "You are a helpful assistant that speaks like a pirate" |
user | Messages from the end user | "What's the weather today?" |
assistant | Previous model responses, used for conversation history | "Arrr, it be sunny with a chance of scurvy!" |
PostHog captures the full message array with roles intact, so you can see exactly what context the model had when it generated a response.
Grouping related events
Most LLM applications involve multiple calls. Traces, spans, and sessions let you see how they connect:
graph TD Session["Session"] Trace["Trace"] Generation1["Generation"] Span["Span"] Generation2["Generation"] Session -.-> Trace Trace --> Generation1 Trace --> Span Span --> Generation2
Here's a breakdown of this hierarchy:
| Term | Definition | Example |
|---|---|---|
| Session | Groups multiple traces together | A user's conversation thread |
| Trace | Contains generations and spans for a single request | One chatbot message and response |
| Span | Tracks an operation within a trace | A retrieval step or function call |
| Generation | An LLM call, tracked as $ai_generation events | Sending a prompt to Claude |
| Embedding | Converts text into vectors | Vectorizing documents for RAG |
A session here usually means a conversation
In AI Observability, a session ($ai_session_id) typically represents a single conversation – one thread of back-and-forth turns with your assistant. This is different from a standard PostHog session ($session_id), like the ones you see in Session replay, which represents one continuous visit to your app.
A single PostHog session can contain multiple AI Observability sessions: a user might start several separate conversations during one visit. See sessions for more on the distinction.
See it in action
Want to see these concepts applied in real projects? Check out our demos and example projects or browse AI Observability side projects built by the PostHog team.
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs an MCP server
Security
What our scanner noticed· Low risk
- Instructions that use the network
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Created by
- PostHog
- Publisher
- PostHog (Official)
- Source
- PostHog/skills · skills/omnibus/instrument-llm-analytics · commit a5d30e7
- Licence
- MIT License (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by PostHog or PostHog.
Related skills
Claude API
AnthropicOfficial
Comprehensive reference for integrating Anthropic's Claude API, SDKs, and Managed Agents into your applications.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreApache-2.0Medium riskHas scriptsGitHub stars of the source repository188.6k
Transformers
K-Dense· Community
Integrate Hugging Face Transformers for model loading, inference, and fine-tuning across NLP, vision, and audio tasks.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITLow riskGitHub stars of the source repository56.4k
Open Notebook
K-Dense· Community
Integrate with the self-hosted Open Notebook platform to automate research ingestion, context-aware chat, and AI-driven analysis.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITLow riskHas scriptsGitHub stars of the source repository56.4k
Vision Sft
Seth Hobson· Community
A structured workflow for supervised fine-tuning of vision-language models, focusing on adapter configuration and silent failure prevention.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository42.3k
LLM Evaluation
Seth Hobson· Community
A comprehensive framework for evaluating LLM application performance using automated metrics, human feedback, and LLM-as-a-judge techniques.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository42.3k
Lora Qlora Recipes
Seth Hobson· Community
Provides configuration recipes and best-practice hyperparameters for LoRA and QLoRA supervised fine-tuning.
Data and AIClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository42.3k
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download instrument-llm-analytics.zip