AutoGineer
Generated banner for LLM Analytics Setup
Generated logo for LLM Analytics Setup

LLM Analytics Setup

PostHogOfficialCreated by PostHog

  • Skill
  • Tool integration
  • Data and AI
  • MIT
  • Low risk
Download ZIP

From the author

PostHog LLM analytics for all supported providers
— the description by PostHog

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a comprehensive guide for integrating PostHog's LLM analytics into applications using various AI providers and frameworks. By wrapping existing LLM clients with PostHog's instrumentation, developers can automatically capture detailed telemetry—including input/output tokens, latency, model names, and costs—without proxying their API calls. The skill covers a wide range of providers and frameworks, offering specific implementation patterns for each, and includes a fallback manual capture method for custom setups.

What it does

  • Instruments LLM calls to automatically capture $ai_generation events.
  • Tracks key metrics like latency, token usage, and total cost per generation.
  • Supports a broad ecosystem of LLM providers and AI frameworks.
  • Enables linking LLM generations to specific users for better observability.
  • Provides a manual capture interface for unsupported or custom LLM providers.
  • Ensures data privacy by distinguishing between event properties and PII.

When to use it

  • When you need to monitor the performance and cost of LLM calls in production.
  • When you want to debug AI agent behavior by tracing input and output sequences.
  • When you need to associate AI interactions with specific user sessions.
  • When you are building applications using frameworks like LangChain, AutoGen, or LlamaIndex.

Advantages

  • Non-intrusive integration that does not proxy your LLM API calls.
  • Centralized visibility into LLM costs and usage across different providers.
  • Automatic capture of critical performance metrics like time-to-first-token.
  • Flexible configuration allows for custom properties and trace IDs.

Usage examples

“How do I add PostHog analytics to my Anthropic Claude implementation in Python?”
The agent will provide the code to initialize the PostHog client and wrap the Anthropic client, enabling automatic event capture for all messages.create calls.
“I am using AutoGen for my agents, how can I track their LLM usage in PostHog?”
The agent will guide you to install the necessary extensions and show how to pass a wrapped OpenAI client into the AutoGen OpenAIChatCompletionClient.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/posthog/llm-analytics-setup/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip llm-analytics-setup.zip -d .agents/skills

Documentation by PostHog

PostHog LLM analytics

This skill helps you add PostHog LLM analytics to any application using AI/LLM providers.

Reference files

  • references/openai.md - Openai llm analytics installation - docs

  • references/azure-openai.md - Azure openai llm analytics installation - docs

  • references/anthropic.md - Anthropic llm analytics installation - docs

  • references/google.md - Google llm analytics installation - docs

  • references/cohere.md - Cohere llm analytics installation - docs

  • references/mistral.md - Mistral llm analytics installation - docs

  • references/perplexity.md - Perplexity llm analytics installation - docs

  • references/deepseek.md - Deepseek llm analytics installation - docs

  • references/groq.md - Groq llm analytics installation - docs

  • references/together-ai.md - Together ai llm analytics installation - docs

  • references/fireworks-ai.md - Fireworks ai llm analytics installation - docs

  • references/xai.md - Xai llm analytics installation - docs

  • references/cerebras.md - Cerebras llm analytics installation - docs

  • references/hugging-face.md - Hugging face llm analytics installation - docs

  • references/ollama.md - Ollama llm analytics installation - docs

  • references/openrouter.md - Openrouter llm analytics installation - docs

  • references/langchain.md - Langchain llm analytics installation - docs

  • references/llamaindex.md - Llamaindex llm analytics installation - docs

  • references/crewai.md - Crewai llm analytics installation - docs

  • references/autogen.md - Autogen llm analytics installation - docs

  • references/dspy.md - Dspy llm analytics installation - docs

  • references/langgraph.md - Langgraph llm analytics installation - docs

  • references/pydantic-ai.md - Pydantic ai llm analytics installation - docs

  • references/vercel-ai.md - Vercel ai SDK llm analytics installation - docs

  • references/litellm.md - Litellm llm analytics installation - docs

  • references/instructor.md - Instructor llm analytics installation - docs

  • references/semantic-kernel.md - Semantic kernel llm analytics installation - docs

  • references/mirascope.md - Mirascope llm analytics installation - docs

  • references/mastra.md - Mastra llm analytics installation - docs

  • references/smolagents.md - Smolagents llm analytics installation - docs

  • references/openai-agents.md - Openai agents SDK llm analytics installation - docs

  • references/portkey.md - Portkey llm analytics installation - docs

  • references/helicone.md - Helicone llm analytics installation - docs

  • references/manual-capture.md - Manual capture llm analytics installation - docs

  • references/basics.md - Llm analytics basics - docs

  • references/traces.md - Traces - docs

  • references/calculating-costs.md - Calculating llm costs - docs

Each provider reference contains installation instructions, SDK setup, and code examples specific to that provider or framework. Find the reference that matches the user's stack and follow its instructions.

If the user's provider isn't listed, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

Key principles

  • Environment variables: Always use environment variables for PostHog and LLM provider keys. Never hardcode them.

  • Minimal changes: Add LLM analytics alongside existing LLM calls. Don't replace or restructure existing code.

  • Trace all generations: Capture input tokens, output tokens, model name, latency, and costs for every LLM call.

  • Link to users: Associate LLM generations with identified users via distinct IDs when possible.

  • One provider at a time: Only instrument the provider(s) the user is actually using. Don't add instrumentation for providers not present in the codebase.

Framework guidelines

  • Remember that source code is available in the venv/site-packages directory

  • posthog is the Python SDK package name

  • Install dependencies with pip install posthog or pip install -r requirements.txt and do NOT use unquoted version specifiers like >= directly in shell commands

  • In CLIs and scripts: MUST call posthog.shutdown() before exit or all events are lost

  • Always use the Posthog() class constructor (instance-based API) instead of module-level posthog.api_key config

  • Always include enable_exception_autocapture=True in the Posthog() constructor to automatically track exceptions

  • NEVER send PII in capture() event properties — no emails, full names, phone numbers, physical addresses, IP addresses, or user-generated content

  • PII belongs in identify() person properties, NOT in capture() event properties. Safe event properties are metadata like message_length, form_type, boolean flags.

  • Register posthog_client.shutdown with atexit.register() to ensure all events are flushed on exit

  • The Python SDK has NO identify() method — use posthog_client.set(distinct_id=user_id, properties={...}) to set person properties, or use identify_context(user_id) within a context

Files

references/traces.md

Traces - Docs

Traces are a collection of generations and spans that capture a full interaction between a user and an LLM. The traces tab lists them along with the properties autocaptured by PostHog like the person, total cost, total latency, and more.

Sessions vs Traces

  • Trace ($ai_trace_id): Groups related generations and spans together. Required for all LLM analytics events.

  • Session ($ai_session_id): Optional property that groups multiple traces together based on your chosen grouping strategy.

See the Sessions documentation for more details on how to use $ai_session_id.

Trace timeline

Clicking on a trace opens a timeline of the interaction with all the generation and span events. The trace timeline enables you to see the entire conversation, profiling details, and the individual generations and spans.

Image: LLM tracesImage: LLM traces

A trace presents LLM event data in a timeline, tree-structured view

Conversation display options

When viewing a trace, you can control how conversation messages are displayed using the display options dropdown. The available options are:

  • Expand all - Shows the full content of all messages in the conversation

  • Expand user only - Expands only user messages, keeping system and assistant messages collapsed for easier scanning of user inputs

  • Collapse except output and last input - The default view that shows the model's output and the most recent user input, keeping earlier messages collapsed

Tool calls

Traces display any tools called by the generations within them, shown as tags in the traces list. This makes it easy to see which conversations involved tool use at a glance.

Sentiment classification

PostHog can classify the sentiment of user messages in a trace as negative, neutral, or positive. Sentiment is computed on-demand using a local model when you view a trace — no data is sent to third-party services. Each trace gets an overall sentiment label and score, with a per-generation and per-message breakdown. See Sentiment classification for more details.

Search traces with PostHog AI

PostHog AI can search and analyze your LLM traces using natural language. When you're on an LLM Analytics page, PostHog AI automatically switches to its LLM analytics mode, giving it access to tools for searching traces by date range, model, cost, error status, and other properties.

Example prompts you can try:

  • "Show me recent LLM traces from the past week"

  • "What are the most expensive LLM calls from today?"

  • "Find traces with errors in the last 30 days"

  • "What's happening in my most expensive trace?"

PostHog AI returns trace details including name, latency, cost, token counts, and error count. It can also read individual traces to provide a detailed summary of what happened.

AI event hierarchy

flowchart TD S["<strong>$ai_session_id</strong><br/>(optional)"] A[<strong>$ai_trace</strong>] A2[<strong>$ai_trace</strong>] B[<strong>$ai_generation</strong>] C@{ shape: processes, label: "<strong>$ai_spans</strong>" } D[<strong>$ai_generation</strong>] E@{ shape: processes, label: "<strong>$ai_spans</strong>" } F[<strong>$ai_generation</strong>] S -.-> A S -.-> A2 A --> B A --> C C --> D C --> E E --> F

Traces consist of the following event hierarchy:

  1. (Optional) A session ($ai_session_id) can group multiple traces together.

  2. A trace ($ai_trace_id) is the top-level required grouping for LLM events.

  3. A trace can contain multiple spans and generations.

  4. A span can be the parent of other spans.

  5. A generation can be the child of a span or trace.

Event properties

A trace is a group that contains multiple spans, generations, and embeddings. Traces can be manually sent as events or appear as pseudo-events automatically created from child events.

Event name: $ai_trace

Core properties
PropertyDescription
$ai_trace_idThe trace ID (a UUID to group related AI events together)Must contain only letters, numbers, and special characters: -, _, ~, ., @, (, ), !, ', :, |Example: d9222e05-8708-41b8-98ea-d4a21849e761
$ai_session_id(Optional) Groups related traces together. Use this to organize traces by whatever grouping makes sense for your application (user sessions, workflows, conversations, or other logical boundaries).Example: session-abc-123, conv-user-456
$ai_latency(Optional) The latency of the trace in seconds
$ai_span_name(Optional) The name of the traceExample: chat_completion, rag_pipeline
$ai_is_error(Optional) Boolean to indicate if the trace encountered an error
$ai_error(Optional) The error message or object if the trace failed
Pseudo-trace Events

When you send generation ($ai_generation), span ($ai_span), or embedding ($ai_embedding) events with a $ai_trace_id, PostHog automatically creates a pseudo-trace event that appears in the dashboard as a parent grouping. These pseudo-traces:

  • Are not actual events in your data

  • Automatically aggregate metrics from child events (latency, tokens, costs)

  • Provide a hierarchical view of your AI operations

  • Do not require sending an explicit $ai_trace event

This means you can either:

  1. Send explicit $ai_trace events to control the trace metadata

  2. Let PostHog automatically create pseudo-traces from your generation/span events

Community questions

Ask a question

Was this page useful?

HelpfulCould be better

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an API key, Needs Python

Security

What our scanner noticed· Low risk

  • Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
PostHog
Publisher
PostHog (Official)
Source
PostHog/skills · skills/posthog/llm-analytics/skills/setup · commit a5d30e7
Licence
MIT License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced
Also distributed in
PostHog/skills

AutoGineer is not affiliated with or endorsed by PostHog or PostHog.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download llm-analytics-setup.zip