AutoGineer
Generated banner for LLM Analytics Setup
Generated logo for LLM Analytics Setup

LLM Analytics Setup

PostHogOfficialCreated by PostHog

  • Skill
  • Tool integration
  • Data and AI
  • MIT
  • Low risk
Download ZIP

From the author

PostHog LLM analytics for all supported providers
— the description by PostHog

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a comprehensive guide for integrating PostHog's LLM analytics into applications using various AI providers and frameworks. By wrapping existing LLM clients with PostHog's instrumentation, developers can automatically capture detailed telemetry—including input/output tokens, latency, model names, and costs—without proxying their API calls. The skill covers a wide range of providers and frameworks, offering specific implementation patterns for each, and includes a fallback manual capture method for custom setups.

What it does

  • Instruments LLM calls to automatically capture $ai_generation events.
  • Tracks key metrics like latency, token usage, and total cost per generation.
  • Supports a broad ecosystem of LLM providers and AI frameworks.
  • Enables linking LLM generations to specific users for better observability.
  • Provides a manual capture interface for unsupported or custom LLM providers.
  • Ensures data privacy by distinguishing between event properties and PII.

When to use it

  • When you need to monitor the performance and cost of LLM calls in production.
  • When you want to debug AI agent behavior by tracing input and output sequences.
  • When you need to associate AI interactions with specific user sessions.
  • When you are building applications using frameworks like LangChain, AutoGen, or LlamaIndex.

Advantages

  • Non-intrusive integration that does not proxy your LLM API calls.
  • Centralized visibility into LLM costs and usage across different providers.
  • Automatic capture of critical performance metrics like time-to-first-token.
  • Flexible configuration allows for custom properties and trace IDs.

Usage examples

“How do I add PostHog analytics to my Anthropic Claude implementation in Python?”
The agent will provide the code to initialize the PostHog client and wrap the Anthropic client, enabling automatic event capture for all messages.create calls.
“I am using AutoGen for my agents, how can I track their LLM usage in PostHog?”
The agent will guide you to install the necessary extensions and show how to pass a wrapped OpenAI client into the AutoGen OpenAIChatCompletionClient.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/posthog/llm-analytics-setup/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip llm-analytics-setup.zip -d .agents/skills

Documentation by PostHog

PostHog LLM analytics

This skill helps you add PostHog LLM analytics to any application using AI/LLM providers.

Reference files

  • references/openai.md - Openai llm analytics installation - docs

  • references/azure-openai.md - Azure openai llm analytics installation - docs

  • references/anthropic.md - Anthropic llm analytics installation - docs

  • references/google.md - Google llm analytics installation - docs

  • references/cohere.md - Cohere llm analytics installation - docs

  • references/mistral.md - Mistral llm analytics installation - docs

  • references/perplexity.md - Perplexity llm analytics installation - docs

  • references/deepseek.md - Deepseek llm analytics installation - docs

  • references/groq.md - Groq llm analytics installation - docs

  • references/together-ai.md - Together ai llm analytics installation - docs

  • references/fireworks-ai.md - Fireworks ai llm analytics installation - docs

  • references/xai.md - Xai llm analytics installation - docs

  • references/cerebras.md - Cerebras llm analytics installation - docs

  • references/hugging-face.md - Hugging face llm analytics installation - docs

  • references/ollama.md - Ollama llm analytics installation - docs

  • references/openrouter.md - Openrouter llm analytics installation - docs

  • references/langchain.md - Langchain llm analytics installation - docs

  • references/llamaindex.md - Llamaindex llm analytics installation - docs

  • references/crewai.md - Crewai llm analytics installation - docs

  • references/autogen.md - Autogen llm analytics installation - docs

  • references/dspy.md - Dspy llm analytics installation - docs

  • references/langgraph.md - Langgraph llm analytics installation - docs

  • references/pydantic-ai.md - Pydantic ai llm analytics installation - docs

  • references/vercel-ai.md - Vercel ai SDK llm analytics installation - docs

  • references/litellm.md - Litellm llm analytics installation - docs

  • references/instructor.md - Instructor llm analytics installation - docs

  • references/semantic-kernel.md - Semantic kernel llm analytics installation - docs

  • references/mirascope.md - Mirascope llm analytics installation - docs

  • references/mastra.md - Mastra llm analytics installation - docs

  • references/smolagents.md - Smolagents llm analytics installation - docs

  • references/openai-agents.md - Openai agents SDK llm analytics installation - docs

  • references/portkey.md - Portkey llm analytics installation - docs

  • references/helicone.md - Helicone llm analytics installation - docs

  • references/manual-capture.md - Manual capture llm analytics installation - docs

  • references/basics.md - Llm analytics basics - docs

  • references/traces.md - Traces - docs

  • references/calculating-costs.md - Calculating llm costs - docs

Each provider reference contains installation instructions, SDK setup, and code examples specific to that provider or framework. Find the reference that matches the user's stack and follow its instructions.

If the user's provider isn't listed, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

Key principles

  • Environment variables: Always use environment variables for PostHog and LLM provider keys. Never hardcode them.

  • Minimal changes: Add LLM analytics alongside existing LLM calls. Don't replace or restructure existing code.

  • Trace all generations: Capture input tokens, output tokens, model name, latency, and costs for every LLM call.

  • Link to users: Associate LLM generations with identified users via distinct IDs when possible.

  • One provider at a time: Only instrument the provider(s) the user is actually using. Don't add instrumentation for providers not present in the codebase.

Framework guidelines

  • Remember that source code is available in the venv/site-packages directory

  • posthog is the Python SDK package name

  • Install dependencies with pip install posthog or pip install -r requirements.txt and do NOT use unquoted version specifiers like >= directly in shell commands

  • In CLIs and scripts: MUST call posthog.shutdown() before exit or all events are lost

  • Always use the Posthog() class constructor (instance-based API) instead of module-level posthog.api_key config

  • Always include enable_exception_autocapture=True in the Posthog() constructor to automatically track exceptions

  • NEVER send PII in capture() event properties — no emails, full names, phone numbers, physical addresses, IP addresses, or user-generated content

  • PII belongs in identify() person properties, NOT in capture() event properties. Safe event properties are metadata like message_length, form_type, boolean flags.

  • Register posthog_client.shutdown with atexit.register() to ensure all events are flushed on exit

  • The Python SDK has NO identify() method — use posthog_client.set(distinct_id=user_id, properties={...}) to set person properties, or use identify_context(user_id) within a context

Files

references/basics.md

LLM Analytics basics - Docs

This page covers how your LLM calls become analytics in PostHog and defines key concepts for LLM Analytics.

How LLM calls become events

PostHog's LLM Analytics works by wrapping your existing LLM provider's SDK to capture requests and responses. Your API calls still go directly to your provider, but the wrapper extracts metadata from each call and sends it to PostHog as an event.

graph LR Call["LLM call"] --> Wrapper["Wrapped SDK"] Wrapper --> Provider["LLM provider API"] Wrapper --> Event["Event"] Event --> PostHog["PostHog"]

Events sent to PostHog are called generations. A generation represents a single LLM call. For example, when you send a prompt to Claude and get a response back, that's one generation.

Generations are represented using the event name $ai_generation. Each generation captures the model, provider, input, output, token counts, latency, and cost.

Generation properties

Tokens and costs

Tokens are the units LLMs use to process text. LLM providers charge based on token usage:

  • Input tokens are tokens used in the message you send to an LLM

  • Output tokens are tokens used in the message you receive from an LLM

PostHog automatically calculates costs by matching your model and provider against pricing data. We use OpenRouter's pricing as our primary source, with fallback to manually maintained pricing for additional models.

You can also set custom pricing if you have negotiated rates or use unsupported models.

Message roles

When you send messages to an LLM, each message has a role that tells the model how to interpret it:

RolePurposeExample
systemInstructions that define the assistant's behavior"You are a helpful assistant that speaks like a pirate"
userMessages from the end user"What's the weather today?"
assistantPrevious model responses, used for conversation history"Arrr, it be sunny with a chance of scurvy!"

PostHog captures the full message array with roles intact, so you can see exactly what context the model had when it generated a response.

Grouping related events

Most LLM applications involve multiple calls. Traces, spans, and sessions let you see how they connect:

graph TD Session["Session"] Trace["Trace"] Generation1["Generation"] Span["Span"] Generation2["Generation"] Session -.-> Trace Trace --> Generation1 Trace --> Span Span --> Generation2

Here's a breakdown of this hierarchy:

TermDefinitionExample
SessionGroups multiple traces togetherA user's conversation thread
TraceContains generations and spans for a single requestOne chatbot message and response
SpanTracks an operation within a traceA retrieval step or function call
GenerationAn LLM call, tracked as $ai_generation eventsSending a prompt to Claude
EmbeddingConverts text into vectorsVectorizing documents for RAG
Community questions

Ask a question

Was this page useful?

HelpfulCould be better

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an API key, Needs Python

Security

What our scanner noticed· Low risk

  • Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
PostHog
Publisher
PostHog (Official)
Source
PostHog/skills · skills/posthog/llm-analytics/skills/setup · commit a5d30e7
Licence
MIT License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced
Also distributed in
PostHog/skills

AutoGineer is not affiliated with or endorsed by PostHog or PostHog.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download llm-analytics-setup.zip