AutoGineer
Generated banner for Auditing Experiments Flags
Generated logo for Auditing Experiments Flags

Auditing Experiments Flags

PostHogOfficialPublished by PostHog

  • Skill
  • Review and audit
  • Data and AI
  • MIT
  • No issues found
Download ZIP

From the author

Audit PostHog experiments and feature flags for configuration issues, staleness, and best-practice violations. Read when the user asks to audit, health-check, or review experiments or feature flags, check flag hygiene, or verify experiment setup.
— the description by PostHog

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

This skill provides a structured approach to auditing PostHog experiments and feature flags. It enables users to perform quick checks on individual entities, scoped audits of specific domains, or comprehensive full-project audits. The skill uses defined check taxonomies to identify configuration errors, waste, and process violations, providing clear remediation actions for each finding.

What it does

  • Performs configuration audits on PostHog experiments and feature flags.
  • Identifies critical correctness issues, potential waste, and process violations.
  • Supports single-entity checks, scoped domain audits, and full-project reviews.
  • Categorizes findings by severity (CRITICAL, WARNING, INFO) and domain.
  • Provides actionable remediation steps for every identified issue.
  • Handles data gaps gracefully by noting skipped checks when activity logs are unavailable.

When to use it

  • When you need to verify the health and setup of a specific experiment or feature flag.
  • When performing routine hygiene checks to identify stale or misconfigured flags.
  • When preparing for a launch to ensure experiment metrics and flag rollouts are correctly configured.
  • When cleaning up project resources to reduce technical debt and waste.

Advantages

  • Standardizes the auditing process across different experiments and flags.
  • Prioritizes findings by severity to help focus on the most impactful issues first.
  • Provides clear, actionable guidance for resolving configuration problems.
  • Maintains consistency in reporting through a structured output format.

Usage examples

“Audit the experiment with ID 42.”
The agent fetches the experiment, runs all relevant configuration checks, and reports findings grouped by severity with actionable remediation steps.
“Perform a health check on all active feature flags.”
The agent retrieves all flags, evaluates them against best-practice criteria, and provides a summary report of any issues found.
“Run a comprehensive audit of all experiments and flags in this project.”
The agent performs a full audit, synthesizes recurring patterns across the project, and presents the findings in a structured report or notebook.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/posthog/auditing-experiments-flags/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip auditing-experiments-flags.zip -d .agents/skills

Documentation by PostHog

Auditing experiments and feature flags

This skill teaches you how to run configuration audits on experiments and feature flags. All checks use the experiment and feature flag read tools (experiment-get, experiment-list, feature-flag-get-definition, feature-flag-get-all) — no SQL queries are needed for Phase 1 checks.

Usage modes

Quick check (single entity)

When the user asks about a specific experiment or flag:

  1. Fetch the entity via experiment-get (experiment ID) or feature-flag-get-definition (numeric flag ID).

  2. Apply the relevant checks from experiment checks or flag checks.

  3. Report findings inline as markdown, grouped by severity (CRITICAL first, then WARNING, then INFO).

  4. Include entity links as [Experiment: name](/experiments/id) or [Flag: key](/feature_flags/id).

Scoped audit (one domain)

When the user asks to audit all experiments or all flags:

  1. Bulk-fetch via experiment-list or feature-flag-get-all.

  2. Run all checks for that domain against each entity.

  3. Group findings by severity, then by entity.

  4. Report as inline markdown.

Full audit (comprehensive)

When the user asks for a comprehensive audit of both experiments and flags:

  1. Fetch all experiments via experiment-list and all flags via feature-flag-get-all.

  2. Run all experiment checks and all flag checks.

  3. Apply recurring patterns to identify patterns across multiple findings.

  4. If there are more than 5 entities with findings, write them to a notebook for easier navigation. Otherwise report inline. Create the notebook from the project's own notebook tools. Run search notebooks?- to load them and read the titles.

Output format

For each finding, include:

  • Severity badge: 🔴 CRITICAL, 🟡 WARNING, or 🔵 INFO

  • Check name: Which check produced this finding

  • Entity link: Markdown link to the entity

  • What's wrong: One-sentence description

  • Action: What to do about it (see remediation actions)

Example:

🟡 WARNING — Flag integration · Experiment: checkout-redesign The linked feature flag is inactive (paused). Traffic is not being split. Action: Re-enable the flag or end the experiment.

Handling unavailable data

Some checks require activity logs (feature-flags-activity-retrieve for flags), which may not be available in every session. If activity log data is unavailable:

  • Skip checkActivityHistory (experiment check) entirely.

  • Skip the "toggle instability" and "never activated" sub-checks in flag lifecycle checks.

  • In your report, note which checks were skipped and why:

    Skipped: Activity history checks (activity logs not available via current tools)

Partial failures

If a fetch call fails for some entities:

  • Continue with the entities you could fetch.

  • Report which entities could not be assessed and why.

  • Do not silently omit entities from the audit.

Reference files

Files

references/experiment-checks.md

Experiment checks

Run these checks against each experiment fetched via experiment-get or experiment-list.

For each check, the "Look at" section tells you which fields to inspect on the experiment object. The "Findings" section lists what to report and at what severity.

1. Metric setup

Verifies the experiment has a valid primary metric configuration.

Look at: metrics, metrics_secondary

Findings:

  • No metrics at all: Both metrics and metrics_secondary are empty or missing.

    • Severity: CRITICAL · Category: Correctness

    • Report: "This experiment has no metrics configured. Results cannot be measured."

    • Action: Add at least one primary metric before launching.

  • Secondary metrics only: metrics is empty but metrics_secondary has entries.

    • Severity: WARNING · Category: Process

    • Report: "This experiment has secondary metrics but no primary metric. There is no primary success criterion."

    • Action: Promote one secondary metric to primary or add a new primary metric.

2. Flag integration

Verifies the experiment's linked feature flag is valid and correctly configured.

Look at: feature_flag (the linked flag object or ID), and fetch the flag via feature-flag-get-definition if only an ID is available.

Findings:

  • Missing flag: feature_flag is null or missing.

    • Severity: CRITICAL · Category: Correctness

    • Report: "This experiment has no linked feature flag. Traffic cannot be split."

    • Action: Create and link a feature flag.

  • Inactive flag: The linked flag exists but active is false.

    • Severity: WARNING · Category: Correctness

    • Report: "The linked feature flag is inactive (paused). Traffic is not being split."

    • Action: Re-enable the flag or end the experiment.

  • Deleted flag: The linked flag has deleted set to true.

    • Severity: CRITICAL · Category: Correctness

    • Report: "The linked feature flag has been deleted."

    • Action: Create a new flag and re-link it, or archive the experiment.

  • Incomplete variant split: The rollout percentages in the flag's filters.multivariate.variants don't add up to 100.

    • Severity: WARNING · Category: Correctness

    • Report: "The flag's variant rollout percentages don't add up to 100%. Some matched users get no variant."

    • Action: Adjust the variant percentages on the flag so they total 100%.

  • Too few variants: The flag's filters.multivariate.variants holds fewer than two variants.

    • Severity: CRITICAL · Category: Correctness

    • Report: "The linked feature flag has fewer than two variants. There is nothing to compare."

    • Action: Add a test variant to the flag alongside the baseline.

3. State consistency

Checks for contradictions between an experiment's conclusion and its current flag state.

Look at: end_date (non-null means concluded), archived, parameters.recommended_variant, and the linked flag's active state and variant configuration.

Findings:

  • Conclusion contradicts shipped variant: The experiment concluded with a recommended variant (in parameters.recommended_variant), but the flag is rolled out to a different variant at 100%.

    • Severity: WARNING · Category: Correctness

    • Report: "The experiment concluded recommending variant 'X' but the flag is rolled out to variant 'Y'."

    • Action: Review and align the flag's rollout with the experiment conclusion.

  • Concluded but still splitting: The experiment has an end_date (it's concluded) but the linked flag still has multiple variants with non-zero rollout (traffic is still being split).

    • Severity: WARNING · Category: Waste

    • Report: "This experiment has concluded but its flag is still splitting traffic between variants."

    • Action: Roll out the winning variant or disable the flag.

4. Lifecycle

Checks for experiments stuck in unproductive states.

Look at: created_at, start_date, end_date, description (for hypothesis)

Findings:

  • Stale draft: start_date is null (never launched) and created_at is more than 7 days ago.

    • Severity: INFO · Category: Cleanup

    • Report: "This experiment has been in draft for N days without being launched."

    • Action: Launch the experiment or delete it.

  • No hypothesis: description is empty or missing, and the experiment has been launched (start_date is set).

    • Severity: INFO · Category: Process

    • Report: "This launched experiment has no hypothesis documented in its description."

    • Action: Add a hypothesis to document what you expect to learn.

5. Stopped with active flag

Checks for experiments that have ended but whose flags are still active and splitting.

Look at: end_date, archived, and the linked flag's active status and variant rollout.

Findings:

  • Ended but flag still active and splitting: end_date is set (experiment ended), but the linked flag is still active: true and has multiple variants with non-zero rollout percentages.

    • Severity: WARNING · Category: Waste

    • Report: "This experiment ended on [date] but its flag is still actively splitting traffic."

    • Action: Roll out the winning variant at 100% or disable the flag.

Note: This is related to but distinct from "concluded but still splitting" in check 3. Check 3 focuses on the contradiction with the conclusion; this check focuses on the resource waste of an ended experiment still consuming flag evaluations.

6. Minimum duration

Checks whether a running experiment has collected enough data.

Look at: start_date, end_date

Findings:

  • Very short run: start_date is set, end_date is set, and the duration is less than 7 days.

    • Severity: WARNING · Category: Process

    • Report: "This experiment ran for only N days. Results may not be statistically significant."

    • Action: Consider whether the sample size was sufficient before drawing conclusions.

  • Short run: Duration is between 7 and 14 days.

    • Severity: INFO · Category: Process

    • Report: "This experiment ran for N days. Consider whether the sample size is sufficient."

    • Action: Review statistical significance before concluding.

7. Stats config

Checks for unusual statistical configuration.

Look at: start_date, end_date (or current date if still running), parameters.stats_config

Findings:

  • Long-running experiment: The experiment has been running for more than 30 days (calculated from start_date to end_date or today if still running).

    • Severity: INFO · Category: Process

    • Report: "This experiment has been running for N days. Long-running experiments can accumulate confounding factors."

    • Action: Review whether this experiment still needs to run or if a conclusion can be drawn.

8. Activity history

Checks for flag modifications that may have affected experiment integrity. These checks require activity logs. If activity logs are not available, skip this entire check and note it was skipped.

Look at: Activity log entries for the linked feature flag, filtered by the experiment's run period (start_date to end_date or today).

Findings:

  • Pre-run flag changes: The flag was modified between experiment creation and launch.

    • Severity: INFO · Category: Process

    • Report: "The flag was modified N times before the experiment launched."

    • Action: Informational — verify the flag was in the intended state at launch.

  • Mid-run rollout changes: The flag's rollout percentages were changed while the experiment was running.

    • Severity: WARNING · Category: Correctness

    • Report: "The flag's rollout percentages were changed during the experiment run."

    • Action: This may have affected results. Note the change date and consider its impact on the data.

  • Mid-run variant changes: Variants were added or removed from the flag while the experiment was running.

    • Severity: CRITICAL · Category: Correctness

    • Report: "Variants were added or removed from the flag during the experiment run."

    • Action: This likely invalidated the experiment. Consider restarting with a clean flag.

  • Mid-run flag toggles: The flag was toggled on/off during the experiment run.

    • Severity: WARNING · Category: Correctness

    • Report: "The flag was toggled on/off during the experiment run, creating periods with no traffic splitting."

    • Action: Review whether the interruption affected results significantly.

  • Mid-run targeting changes: The flag's targeting conditions (properties, groups) were modified during the run.

    • Severity: WARNING · Category: Correctness

    • Report: "The flag's targeting conditions were changed mid-experiment, altering the eligible population."

    • Action: Review whether the targeting change affected the experiment's statistical validity.

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Published by
PostHog
Publisher
PostHog (Official)
Source
PostHog/skills · skills/omnibus/auditing-experiments-flags · commit a5d30e7
Licence
MIT License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by PostHog or PostHog.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download auditing-experiments-flags.zip