Instrument LLM Analytics

PostHogOfficialCreated by PostHog

Skill
Tool integration
Data and AI
MIT
Low risk

From the author

Add PostHog AI Observability to trace AI model usage. Use after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. Also handles initial PostHog SDK setup if not yet installed.

— the description by PostHog

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured approach to instrumenting LLM-powered applications with PostHog AI Observability. It guides the agent through detecting the existing LLM stack, installing necessary SDKs, and configuring event capture for token counts, latency, and costs. The skill supports a wide range of LLM providers and frameworks, ensuring that observability is added with minimal disruption to existing codebases.

What it does

Detects LLM providers and frameworks in the codebase.
Installs the PostHog SDK and required integration packages.
Instruments LLM calls to capture tokens, latency, and costs.
Configures environment variables for secure credential management.
Associates LLM generations with specific user IDs.
Provides fallback manual capture methods for unsupported providers.

When to use it

After implementing new LLM features in an application.
During code reviews to ensure proper observability coverage.
When setting up AI monitoring for the first time in a project.
When needing to track costs and performance metrics for AI model usage.

Advantages

Standardizes observability across diverse LLM stacks.
Ensures minimal impact on existing application architecture.
Provides granular visibility into token usage and latency.
Follows best practices for environment variable management.
Supports a broad ecosystem of AI frameworks and providers.

Usage examples

“Add PostHog AI observability to my LangChain project.”: The agent will scan the project for LangChain dependencies, install the necessary PostHog packages, and instrument the LLM chains to report usage metrics to PostHog.
“I need to track costs and latency for my OpenAI API calls.”: The agent will identify the OpenAI implementation, configure the appropriate middleware or callback, and ensure all generations are captured with the required metadata.

Install

Download the skill as a ZIP:
https://autogineer.com/agent-skills/posthog/instrument-llm-analytics/download

Unzip it into the cross-client skills folder of your project:

mkdir -p .agents/skills && unzip instrument-llm-analytics.zip -d .agents/skills

Documentation by PostHog

Add PostHog AI Observability

Use this skill to add PostHog AI Observability that traces AI model usage in new or changed code. Use it after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. If PostHog is not yet installed, this skill also covers initial SDK setup. Supports any provider or framework.

Supported providers: OpenAI, Azure OpenAI, Anthropic, Google, Cohere, Mistral, Perplexity, DeepSeek, Groq, Together AI, Fireworks AI, xAI, Cerebras, Hugging Face, Ollama, OpenRouter.

Supported frameworks: LangChain, LlamaIndex, CrewAI, AutoGen, DSPy, LangGraph, Pydantic AI, Vercel AI, LiteLLM, Instructor, Semantic Kernel, Mirascope, Mastra, SmolAgents, OpenAI Agents.

Proxy/gateway: Portkey, Helicone.

Instructions

Follow these steps IN ORDER:

STEP 1: Analyze the codebase and detect the LLM stack.

Look for LLM provider SDKs (openai, anthropic, google-generativeai, etc.) and AI frameworks (langchain, llamaindex, crewai, etc.) in dependency files and imports.
Look for lockfiles to determine the package manager.
Check for existing PostHog or observability setup. If PostHog is already installed and LLM tracing is configured, skip to STEP 4 to add tracing for any new LLM calls.

STEP 2: Research instrumentation. (Skip if PostHog LLM tracing is already set up.) 2.1. Find the reference file below that matches the detected provider or framework — it is the source of truth for callback setup, middleware configuration, and event capture. Read it now. 2.2. If no reference matches, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

STEP 3: Install the PostHog SDK. (Skip if PostHog is already set up.)

Add the PostHog SDK and any required callback/integration packages.
Do not manually edit dependency files — use the package manager's install command.

STEP 4: Add LLM tracing.

Instrument LLM calls to capture input tokens, output tokens, model name, latency, and costs for every generation.
Follow the provider-specific reference for the exact callback/middleware setup.
Do not alter the fundamental architecture of existing files. Make additions minimal and targeted.
You must read a file immediately before attempting to write it.

STEP 5: Link to users.

Associate LLM generations with identified users via distinct IDs when possible.

STEP 6: Set up environment variables.

Check if the project already has PostHog environment variables configured (e.g. in .env, .env.local, or framework-specific env files). If valid values already exist, skip this step.
If the PostHog project token is missing, use the PostHog MCP server's projects-get tool to retrieve the project's api_token. If multiple projects are returned, ask the user which project to use. If the MCP server is not connected or not authenticated, ask the user for their PostHog project token instead.
For the PostHog host URL: check the projects-get MCP response for a region field — US maps to https://us.i.posthog.com, EU maps to https://eu.i.posthog.com. If the region is not available from the MCP response or from existing project configuration, ask the user: "Are you on PostHog US Cloud or EU Cloud?" Do not assume US Cloud.
Write these values to the appropriate env file using the framework's naming convention.
Reference these environment variables in code instead of hardcoding them.

Reference files

references/openai.md - Openai observability installation
references/azure-openai.md - Azure openai observability installation
references/README.md - PostHog.ai for .net
references/anthropic.md - Anthropic ai observability installation
references/google.md - Google ai observability installation
references/cohere.md - Cohere ai observability installation
references/mistral.md - Mistral ai observability installation
references/perplexity.md - Perplexity ai observability installation
references/deepseek.md - Deepseek ai observability installation
references/groq.md - Groq ai observability installation
references/together-ai.md - Together ai observability installation
references/fireworks-ai.md - Fireworks ai observability installation
references/xai.md - Xai observability installation
references/cerebras.md - Cerebras ai observability installation
references/hugging-face.md - Hugging face ai observability installation
references/ollama.md - Ollama ai observability installation
references/openrouter.md - Openrouter ai observability installation
references/langchain.md - Langchain ai observability installation
references/llamaindex.md - Llamaindex ai observability installation
references/crewai.md - Crewai observability installation
references/autogen.md - Autogen ai observability installation
references/dspy.md - Dspy ai observability installation
references/langgraph.md - Langgraph ai observability installation
references/pydantic-ai.md - Pydantic ai observability installation
references/vercel-ai.md - Vercel ai SDK observability installation
references/litellm.md - Litellm ai observability installation
references/instructor.md - Instructor ai observability installation
references/semantic-kernel.md - Semantic kernel ai observability installation
references/mirascope.md - Mirascope ai observability installation
references/mastra.md - Mastra ai observability installation
references/smolagents.md - Smolagents ai observability installation
references/openai-agents.md - Openai agents SDK observability installation
references/portkey.md - Portkey ai observability installation
references/helicone.md - Helicone ai observability installation
references/manual-capture.md - Manual capture ai observability installation
references/basics.md - Ai observability basics
references/traces.md - Traces
references/calculating-costs.md - Calculating llm costs
references/COMMANDMENTS.md - Framework-specific rules the integration must follow

Each provider reference contains installation instructions, SDK setup, and code examples specific to that provider or framework. Find the reference that matches the user's stack.

If the user's provider isn't listed, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.

Key principles

Environment variables: Always use environment variables for PostHog and LLM provider keys. Never hardcode them.
Minimal changes: Add LLM analytics alongside existing LLM calls. Don't replace or restructure existing code.
Trace all generations: Capture input tokens, output tokens, model name, latency, and costs for every LLM call.
Link to users: Associate LLM generations with identified users via distinct IDs when possible.
One provider at a time: Only instrument the provider(s) the user is actually using. Don't add instrumentation for providers not present in the codebase.

Files

ATTRIBUTION.mdAdded by AutoGineer958 B
LICENSELicence file added by AutoGineer1.0 KiB
SKILL.md7.3 KiB
references/COMMANDMENTS.md1.5 KiB
references/README.md543 B
references/anthropic.md27.1 KiB
references/autogen.md7.7 KiB
references/azure-openai.md10.7 KiB
references/basics.md5.6 KiB
references/calculating-costs.md8.8 KiB
references/cerebras.md8.3 KiB
references/cohere.md8.3 KiB
references/crewai.md7.2 KiB
references/deepseek.md8.2 KiB
references/dspy.md7.4 KiB
references/fireworks-ai.md8.4 KiB
references/google.md10.0 KiB
references/groq.md8.2 KiB
references/helicone.md8.3 KiB
references/hugging-face.md8.4 KiB
references/instructor.md10.5 KiB
references/langchain.md8.5 KiB
references/langgraph.md8.3 KiB
references/litellm.md9.5 KiB
references/llamaindex.md8.1 KiB
references/manual-capture.md23.7 KiB
references/mastra.md6.7 KiB
references/mirascope.md8.0 KiB
references/mistral.md8.3 KiB
references/ollama.md8.2 KiB
references/openai-agents.md8.2 KiB
references/openai.md10.2 KiB
references/openrouter.md8.3 KiB
references/perplexity.md8.3 KiB
references/portkey.md8.2 KiB
references/pydantic-ai.md7.4 KiB
references/semantic-kernel.md7.7 KiB
references/smolagents.md7.5 KiB
references/together-ai.md8.3 KiB
references/traces.md8.5 KiB
references/vercel-ai.md10.0 KiB
references/xai.md8.2 KiB

references/openai-agents.md

AI agents: this is one page from PostHog's docs. Full index of Markdown docs for LLMs: https://posthog.com/llms.txt

OpenAI Agents SDK observability installation

Canonical URL: https://posthog.com/docs/ai-observability/installation/openai-agents

Cloud region: This page uses US Cloud hosts. For an EU Cloud project, use https://eu.i.posthog.com instead of https://us.i.posthog.com (ingestion host), and use https://eu.posthog.com instead of https://app.posthog.com (app host for links).

1
Install the PostHog SDK
Required
Setting up analytics starts with installing the PostHog Python SDK.
```
pip install posthog
```
2
Install the OpenAI Agents SDK
Required
Install the OpenAI Agents SDK. PostHog instruments your agent runs by registering a tracing processor. The PostHog SDK does not proxy your calls.
```
pip install openai-agents
```
Proxy note
These SDKs do not proxy your calls. They only fire off an async call to PostHog in the background to send the data. You can also use AI observability with other SDKs or our API, but you will need to capture the data in the right format. See the schema in the manual capture section for more details.
3
Initialize PostHog tracing
Required
Initialize PostHog with your project token and host from your project settings. Then call instrument() to register PostHog tracing with the OpenAI Agents SDK. This automatically captures all agent traces, spans, and LLM generations.
```
from posthog import Posthog
from posthog.ai.openai_agents import instrument

posthog = Posthog(
    "<ph_project_token>",
    host="https://us.i.posthog.com"
)

instrument(
    client=posthog,
    distinct_id=lambda trace: (trace.metadata or {}).get("posthog_distinct_id"),
    privacy_mode=False, # optional
    groups={"company": "company_id_in_your_db"}, # optional
)
```
Note: If you want to capture LLM events anonymously, do not pass a distinct ID — here or per run. See our docs on anonymous vs identified events to learn more.

Run your agents

Required

Run your OpenAI agents as normal. PostHog automatically captures $ai_generation events for LLM calls and $ai_span events for agent execution, tool calls, and handoffs. Pass the user and conversation on the run's RunConfig:

group_id groups the run's traces into a conversation — it becomes $ai_session_id.
trace_metadata["posthog_distinct_id"] attributes the run's events to a user — the distinct_id lambda from the previous step reads it off each trace. Any other trace_metadata keys land on the trace as $ai_trace_metadata.

The example below defines a tool and lets the agent call it.

from agents import Agent, Runner, RunConfig, function_tool

@function_tool
def get_weather(city: str) -> str:
    """Get the weather for a city."""
    return f"The weather in {city} is sunny, 72F"

agent = Agent(
    name="Assistant",
    instructions="You are a helpful assistant.",
    tools=[get_weather],
)

result = Runner.run_sync(
    agent,
    "What's the weather in Paris?",
    run_config=RunConfig(
        group_id="conversation_abc",
        trace_metadata={"posthog_distinct_id": "user_123"},
    ),
)
print(result.final_output)

You can expect captured $ai_generation events to have the following properties:

Property	Description
`$ai_model`	The specific model, like `gpt-5-mini` or `claude-4-sonnet`
`$ai_latency`	The latency of the LLM call in seconds
`$ai_time_to_first_token`	Time to first token in seconds (streaming only)
`$ai_tools`	Tools and functions available to the LLM
`$ai_input`	List of messages sent to the LLM
`$ai_input_tokens`	The number of tokens in the input (often found in response.usage)
`$ai_output_choices`	List of response choices from the LLM
`$ai_output_tokens`	The number of tokens in the output (often found in `response.usage`)
`$ai_total_cost_usd`	The total cost in USD (input + output)
[...]	See full list of properties

Multi-agent and tool usage

Optional

PostHog captures the full trace hierarchy for complex agent workflows, including handoffs between multiple agents.

from agents import Agent, Runner, function_tool

@function_tool
def get_weather(city: str) -> str:
    """Get the weather for a city."""
    return f"The weather in {city} is sunny, 72F"

weather_agent = Agent(
    name="WeatherAgent",
    instructions="You help with weather queries.",
    tools=[get_weather]
)

triage_agent = Agent(
    name="TriageAgent",
    instructions="Route weather questions to the weather agent.",
    handoffs=[weather_agent]
)

result = Runner.run_sync(triage_agent, "What's the weather in San Francisco?")

This captures:

Agent spans for TriageAgent and WeatherAgent
Handoff spans showing the routing between agents
Tool spans for get_weather function calls
Generation spans for all LLM calls

As with the single-agent example above, PostHog captures every span in that list automatically. You write no extra code for the handoff itself.

Verify traces and generations
Recommended
Confirm LLM events are being sent to PostHog
Let's make sure LLM events are being captured and sent to PostHog. Under AI Observability, you should see rows of data appear in the Traces and Generations tabs.
Image: LLM generations in PostHog
Check for LLM events in PostHog

Next steps

Recommended

Now that you're capturing AI conversations, continue with the resources below to learn what else AI Observability enables within the PostHog platform.

Resource	Description
Basics	Learn the basics of how LLM calls become events in PostHog.
Generations	Read about the `$ai_generation` event and its properties.
Traces	Explore the trace hierarchy and how to use it to debug LLM calls.
Spans	Review spans and their role in representing individual operations.
Anaylze LLM performance	Learn how to create dashboards to analyze LLM performance.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Needs an MCP server

Security

What our scanner noticed· Low risk

Instructions that use the network

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by

PostHog

Publisher

PostHog (Official)

Source

PostHog/skills · skills/omnibus/instrument-llm-analytics · commit a5d30e7

Licence

MIT License (redistribution allowed)

LICENSE (added by AutoGineer from LICENSE)
ATTRIBUTION.md (written by AutoGineer)

Synced

2026-11-09

AutoGineer is not affiliated with or endorsed by PostHog or PostHog.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download instrument-llm-analytics.zip

From the author

AutoGineer summary

What it does

When to use it

Advantages

Usage examples

Install

Documentation by PostHog

Add PostHog AI Observability

Instructions

Reference files

Key principles

Files

references/openai-agents.md

OpenAI Agents SDK observability installation

On this page

Install the PostHog SDK

Install the OpenAI Agents SDK

Initialize PostHog tracing

Run your agents

Multi-agent and tool usage

Verify traces and generations

Next steps

Requirements and compatibility

Security

Attribution and licence

Related skills

Claude API

Transformers

Open Notebook

Vision Sft

LLM Evaluation

Lora Qlora Recipes

Download ZIP