LLM Analytics Setup — Agent Skill by PostHog

From the author

PostHog LLM analytics for all supported providers

— the description by PostHog

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a comprehensive guide for integrating PostHog's LLM analytics into applications using various AI providers and frameworks. By wrapping existing LLM clients with PostHog's instrumentation, developers can automatically capture detailed telemetry—including input/output tokens, latency, model names, and costs—without proxying their API calls. The skill covers a wide range of providers and frameworks, offering specific implementation patterns for each, and includes a fallback manual capture method for custom setups.

What it does

Instruments LLM calls to automatically capture $ai_generation events.
Tracks key metrics like latency, token usage, and total cost per generation.
Supports a broad ecosystem of LLM providers and AI frameworks.
Enables linking LLM generations to specific users for better observability.
Provides a manual capture interface for unsupported or custom LLM providers.
Ensures data privacy by distinguishing between event properties and PII.

When to use it

When you need to monitor the performance and cost of LLM calls in production.
When you want to debug AI agent behavior by tracing input and output sequences.
When you need to associate AI interactions with specific user sessions.
When you are building applications using frameworks like LangChain, AutoGen, or LlamaIndex.

Advantages

Non-intrusive integration that does not proxy your LLM API calls.
Centralized visibility into LLM costs and usage across different providers.
Automatic capture of critical performance metrics like time-to-first-token.
Flexible configuration allows for custom properties and trace IDs.

Usage examples

“How do I add PostHog analytics to my Anthropic Claude implementation in Python?”: The agent will provide the code to initialize the PostHog client and wrap the Anthropic client, enabling automatic event capture for all messages.create calls.
“I am using AutoGen for my agents, how can I track their LLM usage in PostHog?”: The agent will guide you to install the necessary extensions and show how to pass a wrapped OpenAI client into the AutoGen OpenAIChatCompletionClient.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/posthog/llm-analytics-setup/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip llm-analytics-setup.zip -d .agents/skills

Documentation by PostHog

PostHog LLM analytics

This skill helps you add PostHog LLM analytics to any application using AI/LLM providers.

Reference files

references/openai.md - Openai llm analytics installation - docs
references/azure-openai.md - Azure openai llm analytics installation - docs
references/anthropic.md - Anthropic llm analytics installation - docs
references/google.md - Google llm analytics installation - docs
references/cohere.md - Cohere llm analytics installation - docs
references/mistral.md - Mistral llm analytics installation - docs
references/perplexity.md - Perplexity llm analytics installation - docs
references/deepseek.md - Deepseek llm analytics installation - docs
references/groq.md - Groq llm analytics installation - docs
references/together-ai.md - Together ai llm analytics installation - docs
references/fireworks-ai.md - Fireworks ai llm analytics installation - docs
references/xai.md - Xai llm analytics installation - docs
references/cerebras.md - Cerebras llm analytics installation - docs
references/hugging-face.md - Hugging face llm analytics installation - docs
references/ollama.md - Ollama llm analytics installation - docs
references/openrouter.md - Openrouter llm analytics installation - docs
references/langchain.md - Langchain llm analytics installation - docs
references/llamaindex.md - Llamaindex llm analytics installation - docs
references/crewai.md - Crewai llm analytics installation - docs
references/autogen.md - Autogen llm analytics installation - docs
references/dspy.md - Dspy llm analytics installation - docs
references/langgraph.md - Langgraph llm analytics installation - docs
references/pydantic-ai.md - Pydantic ai llm analytics installation - docs
references/vercel-ai.md - Vercel ai SDK llm analytics installation - docs
references/litellm.md - Litellm llm analytics installation - docs
references/instructor.md - Instructor llm analytics installation - docs
references/semantic-kernel.md - Semantic kernel llm analytics installation - docs
references/mirascope.md - Mirascope llm analytics installation - docs
references/mastra.md - Mastra llm analytics installation - docs
references/smolagents.md - Smolagents llm analytics installation - docs
references/openai-agents.md - Openai agents SDK llm analytics installation - docs
references/portkey.md - Portkey llm analytics installation - docs
references/helicone.md - Helicone llm analytics installation - docs
references/manual-capture.md - Manual capture llm analytics installation - docs
references/basics.md - Llm analytics basics - docs
references/traces.md - Traces - docs
references/calculating-costs.md - Calculating llm costs - docs

Each provider reference contains installation instructions, SDK setup, and code examples specific to that provider or framework. Find the reference that matches the user's stack and follow its instructions.

If the user's provider isn't listed, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

Key principles

Environment variables: Always use environment variables for PostHog and LLM provider keys. Never hardcode them.
Minimal changes: Add LLM analytics alongside existing LLM calls. Don't replace or restructure existing code.
Trace all generations: Capture input tokens, output tokens, model name, latency, and costs for every LLM call.
Link to users: Associate LLM generations with identified users via distinct IDs when possible.
One provider at a time: Only instrument the provider(s) the user is actually using. Don't add instrumentation for providers not present in the codebase.

Framework guidelines

Remember that source code is available in the venv/site-packages directory
posthog is the Python SDK package name
Install dependencies with pip install posthog or pip install -r requirements.txt and do NOT use unquoted version specifiers like >= directly in shell commands
In CLIs and scripts: MUST call posthog.shutdown() before exit or all events are lost
Always use the Posthog() class constructor (instance-based API) instead of module-level posthog.api_key config
Always include enable_exception_autocapture=True in the Posthog() constructor to automatically track exceptions
NEVER send PII in capture() event properties — no emails, full names, phone numbers, physical addresses, IP addresses, or user-generated content
PII belongs in identify() person properties, NOT in capture() event properties. Safe event properties are metadata like message_length, form_type, boolean flags.
Register posthog_client.shutdown with atexit.register() to ensure all events are flushed on exit
The Python SDK has NO identify() method — use posthog_client.set(distinct_id=user_id, properties={...}) to set person properties, or use identify_context(user_id) within a context

Files

ATTRIBUTION.mdAdded by AutoGineer957 B
LICENSELicence file added by AutoGineer1.0 KiB
SKILL.md5.0 KiB
references/anthropic.md6.6 KiB
references/autogen.md4.5 KiB
references/azure-openai.md6.4 KiB
references/basics.md3.2 KiB
references/calculating-costs.md8.3 KiB
references/cerebras.md6.7 KiB
references/cohere.md6.7 KiB
references/crewai.md4.4 KiB
references/deepseek.md6.7 KiB
references/dspy.md4.4 KiB
references/fireworks-ai.md6.8 KiB
references/google.md7.6 KiB
references/groq.md6.7 KiB
references/helicone.md6.9 KiB
references/hugging-face.md6.8 KiB
references/instructor.md5.8 KiB
references/langchain.md6.9 KiB
references/langgraph.md6.6 KiB
references/litellm.md6.6 KiB
references/llamaindex.md4.7 KiB
references/manual-capture.md15.6 KiB
references/mastra.md5.1 KiB
references/mirascope.md4.5 KiB
references/mistral.md6.7 KiB
references/ollama.md6.7 KiB
references/openai-agents.md5.7 KiB
references/openai.md6.2 KiB
references/openrouter.md7.1 KiB
references/perplexity.md6.7 KiB
references/portkey.md7.0 KiB
references/pydantic-ai.md4.5 KiB
references/semantic-kernel.md4.5 KiB
references/smolagents.md4.3 KiB
references/together-ai.md6.7 KiB
references/traces.md5.8 KiB
references/vercel-ai.md4.5 KiB
references/xai.md6.6 KiB

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an API key, Needs Python

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Claude API
AnthropicOfficial
Comprehensive reference for integrating Anthropic's Claude API, SDKs, and Managed Agents into your applications.
Data and AIClaude CodeClaude appsOpenAI Codex+14 more
Apache-2.0Medium riskHas scriptsGitHub stars of the source repository188.6k
Transformers
K-Dense· Community
Integrate Hugging Face Transformers for model loading, inference, and fine-tuning across NLP, vision, and audio tasks.
Data and AIClaude CodeClaude appsOpenAI Codex+14 more
MITLow riskGitHub stars of the source repository56.4k
Open Notebook
K-Dense· Community
Integrate with the self-hosted Open Notebook platform to automate research ingestion, context-aware chat, and AI-driven analysis.
Data and AIClaude CodeClaude appsOpenAI Codex+14 more
MITLow riskHas scriptsGitHub stars of the source repository56.4k
Vision Sft
Seth Hobson· Community
A structured workflow for supervised fine-tuning of vision-language models, focusing on adapter configuration and silent failure prevention.
Data and AIClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository42.3k
LLM Evaluation
Seth Hobson· Community
A comprehensive framework for evaluating LLM application performance using automated metrics, human feedback, and LLM-as-a-judge techniques.
Data and AIClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository42.3k
Lora Qlora Recipes
Seth Hobson· Community
Provides configuration recipes and best-practice hyperparameters for LoRA and QLoRA supervised fine-tuning.
Data and AIClaude CodeClaude appsOpenAI Codex+14 more
MITGitHub stars of the source repository42.3k