Diagnosing Experiment Health

Diagnosing experiment health

This skill answers: My PostHog experiment results look wrong, biased, or empty — what's going on? It is the diagnostics and health check for one experiment: its setup, its exposures, its results and what changed during the run.

Match the user's complaint in the dispatch table, then read the matching reference file for the diagnostic.

Each diagnostic in the reference files is tagged [HIGH], [MEDIUM], or [LOW] based on how strongly it's verified — [HIGH] is verified directly in PostHog code, [MEDIUM] is partially or team-source verified, [LOW] describes SDK/external behavior that wasn't verified here. Treat [LOW] items as hypotheses to test, not facts to assert.

Step 1 — Resolve the experiment

If the user refers to an experiment by name or description, load the finding-experiments skill first to resolve it to a concrete ID.

Call experiment-get and pull these fields. They are inputs for almost every diagnostic:

• status (draft / running / paused / exposure_frozen / stopped), start_date, end_date, is_legacy

• feature_flag.active, and feature_flag.filters.multivariate.variants[] — the variant keys and the split (rollout_percentage). This is the flag as it is now. After a ship or an edit, the split of the run is in the change history (Step 1.5).

• feature_flag.filters.aggregation_group_type_index — when set, the experiment's unit is a group, not a person

• feature_flag.filters.groups[] — for each group read variant, properties, and rollout_percentage (that group's rollout, % of the matched bucketing units — persons, or groups when the flag or the group is aggregated by a group type — that enter the experiment; each group has its own, so there is no single overall rollout unless the flag has one unconditional group). A non-null variant that is one of the flag's variant keys is a forced-variant override on the matched cohort (release-condition assignment, not randomized) — surfaces A7. Watch for the severe shape (A7b): a variant-pinned group with broad/empty properties at high rollout, or no group left randomized (variant: null) / no release path to one arm — that starves the other variant (one arm gets ~0 analyzable exposures). See references/bias-and-skew.md.

• feature_flag.bucketing_identifier and feature_flag.ensure_experience_continuity — what the flag hashes (A3, A6, A8, A10)

• holdout and excluded_variants — people and arms that are outside the analysis on purpose

• exposure_criteria.multiple_variant_handling — defaults to "exclude" if absent

• exposure_criteria.exposure_config — the exposure event and its property filters. Unset, or naming $feature_flag_called, means the default exposure event: read which one from resolved_exposure_event. When it is $feature_flag_called, also read $experiment_exposure (event IN ('$feature_flag_called', '$experiment_exposure')), and count persons, not rows. A config that names $experiment_exposure counts that event, whatever resolved_exposure_event says. Any other event, or an action, is a custom exposure event. Property filters in the config apply to either (B12). The table in references/diagnostic-snapshot.md says which property carries the variant.

• exposure_criteria.activation_config — an activation event, or an action, on top of the default exposure. When it is set, a person counts as exposed only from their first activation event at or after their first flag exposure, and the time of that event is the exposure time. It does not combine with a custom exposure event.

• exposure_criteria.filterTestAccounts — defaults to true

• metrics and metrics_secondary, with shared metrics — for each: the name, the type and the definition. Describe a metric from its definition, not from its name (D13).

• stats_config — method (Bayesian or Frequentist), the confidence level (bayesian.ci_level, frequentist.alpha), frequentist.sequential_testing_enabled, cuped, baseline_variant_key. A missing method reads as Bayesian and a missing confidence level as 95%. Sequential testing and CUPED fall back to the project default (C13).

• running_time_calculation — the target sample (a total across all variants), the days still missing to reach it, and the minimum detectable effect. When the estimate comes from numbers the user typed, or was written at creation and never refreshed by the page, the days are the whole estimated length (C12). A duplicated experiment starts with the values of its source until the page refreshes them.

• only_count_matured_users — E11

Step 1.5 — Pull a diagnostic snapshot (verify before asking)

Before asking the user clarifying questions, pull the diagnostic snapshot in references/diagnostic-snapshot.md. Most diagnostics in this skill can be confirmed or ruled out from that data without an interview.

Read the cheap sources first, in every diagnosis: the stored results, then the change history. When the user reports a surprising change, read the change history before the stored results. Then the exposure data, when the diagnosis needs what only it holds. Query raw events only for what those cannot show.

Step 2 — Match symptom to diagnostic

If the symptom is unclear, ask one clarifying question before picking. Most diagnostics have different fixes — do not guess. A complaint that matches rows of several groups gets each of them read (Step 3).

A report from a scout or another automated check names its finding: route by the finding, and reuse the report's numbers before pulling them again.

Step 3 — Surface every diagnostic the evidence supports

After matching the symptom in Step 2 and reading the relevant reference file(s), list each diagnostic that applies before recommending an action.

Surface co-occurring mechanisms independently — even when one is more salient, don't collapse them into a single "wait" or "fix" recommendation. Different mechanisms have different fixes: a systematic bias (e.g. uneven-split + Exclude) doesn't resolve by waiting; a statistical pattern (e.g. small-sample variance) does. Bundling them leaves the bias in place after the user follows the bundled advice.

Only list mechanisms that have a path to verification in the project state — config (from experiment-get), snapshot data, activity log, or repo source (when the agent has the repository). Config-derived mechanisms count: an 80/20 split with default multiple_variant_handling="exclude" is visible in experiment-get and is therefore enumerable. Naming a mechanism with no source (e.g. SRM when the snapshot shows a clean variant ratio) is not.

Two checks belong to every diagnosis, whatever the complaint:

• How young is the experiment? State its age (start_date) and its share of the target sample (the exposed total against running_time_calculation.recommended_sample_size) before reading a result (C1 in references/interpretation.md). The page takes the exposed total from the first primary metric: the sum of its number_of_samples over all variants in the stored results, not exposures.total_exposures. recommended_sample_size is the target the page saved when it last computed the estimate. When it is absent, no target was saved: say so.

• Who is exposed? Compare the exposed population (exposure_criteria and the flag's release conditions) with the people the decision is about before calling the setup sound (D12 in references/numbers-vs-sql.md). Default exposure settings are not evidence that the right people are in the experiment. When nothing says who the decision is about, ask.

Diagnostic groups

A — Bias & skew

Variants don't look balanced, one variant looks biased, the in-app warning banner appeared, or users are showing up under multiple variants. Covers the uneven-split + Exclude interaction, the sample ratio mismatch and its causes, exposure events that are lost or sent unevenly between the arms, flags that return no variant for part of the audience, identity fragmentation, bootstrap × /flags mismatch, mid-run flag edits, forced variants, and flag/experiment state inconsistency.

→ See references/bias-and-skew.md

B — Empty experiment / 0 exposures / "not enough data"

A frequent pain point. Covers SDK call (wrong evaluation method, identify() timing, dedup), exposure capture (custom event missing variant property, required properties, ad-blockers), and exposure-criteria match (test-account filter, filters that cannot match, eligibility ordering, events firing before exposure).

→ See references/empty-experiment.md

C — Significance / interpretation traps

Significance flipping, A/A test showing significance, Bayesian vs Frequentist confusion, multiple comparisons, low-volume variance, peeking / early stopping. Includes the running-time estimate ("target reached" is not a result) and the statistics settings that change how a result reads: sequential testing, CUPED, the confidence level and the baseline variant.

→ See references/interpretation.md

D — Numbers don't match (PostHog vs the user's SQL / raw count)

The experiment page applies an exposure scope, $multiple exclusion, test-account filter, and date range that ad-hoc SQL almost never replicates. Covers funnel attribution (only first→last step counts for stats), breakdowns (read from the exposure event, not the metric event), the "sum of revenue" mean-of-per-user confusion, stale stored results, the Recordings tab against the stats, an insight that counts a narrower population than the experiment, and a metric whose title does not match its definition.

→ See references/numbers-vs-sql.md

E — Surprises after mid-run changes (incl. lifecycle and retention quirks)

Increasing rollout is safe; decreasing is caution; changing the variant split is an anti-pattern; adding metrics mid-run is p-hacking; ship-variant rewrites the flag and ending alone does not; reset clears results not the flag (apart from removing an exposure freeze); a pause, a freeze and an unfreeze can each bias the result; a changed start date moves the whole analysis. Also covers retention-metric quirks, "matured users" filtering, and long-term vs short-term metric divergence.

→ See references/mid-run-changes.md

F — Qualitative feedback: how the change landed, not how far the number moved

Groups A–E find mechanisms. F is for the question they can't reach: what the people in the experiment made of the change. A short survey, shown when users finish the experimented flow, adds that qualitative half — a rating and an optional comment, readable per variant — and works over MCP today.

It suits some experiments and not others. The gate is whether a user could describe the change without being shown both versions: a reworked flow, layout, or process work; a threshold or ranking tweak don't, however large its measured effect. Don't offer it while a mechanical diagnostic is still open — a survey on top of a broken flag gate collects opinions about a feature half the audience never received — and check whether an existing or recent survey already covers the window before proposing a new one.

Two things to get right before creating one: a survey shown to a single variant is itself a difference between the variants, and a running survey linked to the flag can generate exposure events.

→ See references/qualitative-feedback.md

Step 4 — Calibrate recommendations to experiment state

Surface diagnostics first (Step 3). Then recommend — but scope what you recommend to what the experiment's current state permits.

• Draft — config changes are free; recommend and apply. The flag of a draft can still serve users: a draft made by a reset, or created on an existing flag, can have feature_flag.active true, and an edit of that flag reaches those users at once.

• Running (and paused or exposure_frozen, which are launched and not ended) — every change has a tradeoff. Explain the mid-run impact (anti-pattern? safe? user-visible?) before recommending. See configuring-experiment-rollout and its reference file references/changing-distribution-after-launch.md for the mid-run rules. A pause, a freeze, an unfreeze and a reset are changes too: each one interrupts the run and can bias the result (E8, E9 and E16 in references/mid-run-changes.md).

• Stopped / archived — the experiment AND its feature flag represent the documented outcome of the run. Recommendations are scoped to (a) interpretation of the existing data, (b) what to do for the next experiment, or (c) explaining what happened.

A diagnosis ends with findings and a recommendation. A change to a running experiment or to its flag (a new metric, new exposure criteria, a reset, a relaunch, a flag edit) is the user's decision: say what the change does to the data collected so far, and make it only on an explicit instruction.

On a stopped or archived experiment, don't preemptively offer reversal of a state mutation (ship-variant flag rewrite, manual flag edit, reset, archive). If the user asks "why did X happen?", explain X — don't append a "here's how to undo it" coda. That pattern assumes intent the user didn't signal. Conditional offers like "if this wasn't intended, you could…" or "want me to revert it?" count as preemptive too — only the user explicitly naming the reversal action ("how do I undo this?", "can I roll back ship-variant?", "how do I get the 50/50 split back?") is a request to surface reversal mechanics.

Use consistent terminology: variant split (between variants) is distinct from rollout (the share of a release condition's audience that enters); the default exposure event (resolved_exposure_event) is distinct from a custom exposure event; the Exclude from analysis / Use first seen variant options of Multiple variant handling (exposure_criteria.multiple_variant_handling) decide how people exposed to several variants are counted, not which event is the exposure.

Related skills

• configuring-experiment-analytics — fix the exposure or metric configuration the diagnosis points at

• configuring-experiment-rollout — split-change anti-patterns and safe rollout adjustments

• managing-experiment-lifecycle — reset, end, or restart when the experiment can't be salvaged in place

• analyzing-experiment-session-replays — when the numbers are fine but you need to see the behavior behind them

• debugging-experiments — when the question arrives as a support request about someone else's experiment and the answer is a written reply