Diagnosing Experiment Health
PostHogOfficialPublished by PostHog
- Skill
- Review and audit
- Data and AI
- MIT
- No issues found
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
This skill provides a structured diagnostic framework for investigating PostHog experiment results. It helps users identify why experiments might appear biased, empty, or statistically inconsistent by analyzing configuration, exposure data, and activity logs. It guides the user through a systematic verification process, moving from high-level configuration checks to detailed statistical interpretation, ensuring that reported issues are grounded in verifiable data rather than assumptions.
What it does
- Resolves experiment IDs and retrieves current configuration settings.
- Pulls diagnostic snapshots to verify experiment state without user interviews.
- Identifies causes for bias, sample ratio mismatches, and uneven variant splits.
- Troubleshoots empty experiments and zero-exposure scenarios.
- Interprets statistical significance, A/A test results, and potential interpretation traps.
- Compares PostHog experiment numbers against raw SQL or custom dashboard counts.
When to use it
- When an experiment shows unexpected bias or a sample ratio mismatch warning.
- When an experiment reports zero exposures or results that seem empty.
- When results appear statistically significant or insignificant in ways that contradict expectations.
- When PostHog numbers disagree with external SQL queries or internal dashboards.
- When results change unexpectedly after mid-run configuration edits.
Advantages
- Reduces guesswork by prioritizing data-backed verification over user reports.
- Provides a clear, step-by-step methodology for complex experiment debugging.
- Distinguishes between systematic bias and statistical noise.
- Ensures recommendations are scoped to the current state of the experiment.
Usage examples
- “Why is my experiment showing a sample ratio mismatch?”
- The agent will pull the experiment configuration and exposure snapshot to check the p-value of the sample ratio test and identify if the split changed during the run.
- “My experiment has 0 exposures, what is wrong?”
- The agent will investigate the exposure criteria, SDK call timing, and potential filtering issues to determine why users are not being captured.
- “The results changed after I edited the experiment, can you explain why?”
- The agent will review the experiment activity log and stored results to explain the impact of the mid-run change on the data collected.
- “Is my A/B test statistically significant?”
- The agent will evaluate the current sample size, significance method, and potential interpretation traps like peeking or low-volume variance.
Install
Download the skill as a ZIP:
https://autogineer.com/agent-skills/posthog/diagnosing-experiment-health/downloadUnzip it into the cross-client skills folder of your project:
mkdir -p .agents/skills && unzip diagnosing-experiment-health.zip -d .agents/skills
Documentation by PostHog
Diagnosing experiment health
This skill answers: My PostHog experiment results look wrong, biased, or empty — what's going on? It is the diagnostics and health check for one experiment: its setup, its exposures, its results and what changed during the run.
Match the user's complaint in the dispatch table, then read the matching reference file for the diagnostic.
Each diagnostic in the reference files is tagged [HIGH], [MEDIUM], or [LOW] based on how
strongly it's verified — [HIGH] is verified directly in PostHog code, [MEDIUM] is partially or
team-source verified, [LOW] describes SDK/external behavior that wasn't verified here. Treat [LOW]
items as hypotheses to test, not facts to assert.
Step 1 — Resolve the experiment
If the user refers to an experiment by name or description, load the finding-experiments skill first to
resolve it to a concrete ID.
Call experiment-get and pull these fields. They are inputs for almost every diagnostic:
status(draft/running/paused/exposure_frozen/stopped),start_date,end_date,is_legacyfeature_flag.active, andfeature_flag.filters.multivariate.variants[]— the variant keys and the split (rollout_percentage). This is the flag as it is now. After a ship or an edit, the split of the run is in the change history (Step 1.5).feature_flag.filters.aggregation_group_type_index— when set, the experiment's unit is a group, not a personfeature_flag.filters.groups[]— for each group readvariant,properties, androllout_percentage(that group's rollout, % of the matched bucketing units — persons, or groups when the flag or the group is aggregated by a group type — that enter the experiment; each group has its own, so there is no single overall rollout unless the flag has one unconditional group). A non-nullvariantthat is one of the flag's variant keys is a forced-variant override on the matched cohort (release-condition assignment, not randomized) — surfaces A7. Watch for the severe shape (A7b): a variant-pinned group with broad/emptypropertiesat high rollout, or no group left randomized (variant: null) / no release path to one arm — that starves the other variant (one arm gets ~0 analyzable exposures). Seereferences/bias-and-skew.md.feature_flag.bucketing_identifierandfeature_flag.ensure_experience_continuity— what the flag hashes (A3, A6, A8, A10)holdoutandexcluded_variants— people and arms that are outside the analysis on purposeexposure_criteria.multiple_variant_handling— defaults to"exclude"if absentexposure_criteria.exposure_config— the exposure event and its property filters. Unset, or naming$feature_flag_called, means the default exposure event: read which one fromresolved_exposure_event. When it is$feature_flag_called, also read$experiment_exposure(event IN ('$feature_flag_called', '$experiment_exposure')), and count persons, not rows. A config that names$experiment_exposurecounts that event, whateverresolved_exposure_eventsays. Any other event, or an action, is a custom exposure event. Property filters in the config apply to either (B12). The table inreferences/diagnostic-snapshot.mdsays which property carries the variant.exposure_criteria.activation_config— an activation event, or an action, on top of the default exposure. When it is set, a person counts as exposed only from their first activation event at or after their first flag exposure, and the time of that event is the exposure time. It does not combine with a custom exposure event.exposure_criteria.filterTestAccounts— defaults totruemetricsandmetrics_secondary, with shared metrics — for each: the name, the type and the definition. Describe a metric from its definition, not from its name (D13).stats_config—method(Bayesian or Frequentist), the confidence level (bayesian.ci_level,frequentist.alpha),frequentist.sequential_testing_enabled,cuped,baseline_variant_key. A missingmethodreads as Bayesian and a missing confidence level as 95%. Sequential testing and CUPED fall back to the project default (C13).running_time_calculation— the target sample (a total across all variants), the days still missing to reach it, and the minimum detectable effect. When the estimate comes from numbers the user typed, or was written at creation and never refreshed by the page, the days are the whole estimated length (C12). A duplicated experiment starts with the values of its source until the page refreshes them.only_count_matured_users— E11
Step 1.5 — Pull a diagnostic snapshot (verify before asking)
Before asking the user clarifying questions, pull the diagnostic snapshot in references/diagnostic-snapshot.md. Most diagnostics in this skill can be confirmed or ruled out from that data without an interview.
Read the cheap sources first, in every diagnosis: the stored results, then the change history. When the user reports a surprising change, read the change history before the stored results. Then the exposure data, when the diagnosis needs what only it holds. Query raw events only for what those cannot show.
Step 2 — Match symptom to diagnostic
| User says... | Diagnostic group |
|---|---|
| "Smaller variant looks biased" / banner says bias | A — bias & skew |
| "Variant ratio doesn't match my split" / mismatch warning / "is 58/42 normal?" | A — bias & skew |
| "Why isn't it 50/50?" / "one variant gets fewer users every day" | A — bias & skew |
"Users in both control and test" / high $multiple % | A — bias & skew |
| "Our own analytics show an even split and PostHog doesn't" | A — bias & skew |
| "Most responses on the flag are false" / "users saw the new UI for a second, then lost it" | A — bias & skew |
| Multi-variant exposure on a server-rendered app | A — bias & skew |
| Banner about feature-flag/experiment state mismatch | A — bias & skew |
| "Migrating distinct_id" / "switching from anonymous to user_id" mid-run | A — bias & skew |
| Metric count is much smaller than exposures (e.g. 10× or 100× gap) | A — bias & skew (route here before D) |
| "Experiment shows 0 / not enough data" / empty | B — empty experiment |
| "Variant always undefined / false" | B — empty experiment |
| "$feature_flag_called fires but no exposures show up" | B — empty experiment |
| "Exposures dropped to zero after I added a filter" / "the copied experiment is empty" | B — empty experiment |
| "Plenty of exposures, zero conversions" | B — empty experiment (then D11) |
| "Experiment says running but exposures haven't moved in weeks/months" | B — empty experiment |
| "Significance keeps flipping as we run longer" | C — interpretation traps |
| "Significance was declared, then it wasn't significant anymore" | C — interpretation traps |
| "It's been running one day, can we read it?" | C — interpretation traps |
| "30/16 split at 46 exposures, is this broken?" | C — interpretation traps |
| "It says complete / target reached and nothing is significant" | C — interpretation traps |
| "A/A test is showing significant results" | C — interpretation traps |
| "Many metrics — some significant, some not" | C — interpretation traps |
| "The test variant does worse, is it set up correctly?" / "can you check this experiment?" | Step 3 — the two checks, then C and D12 |
| "Bayesian says 96% chance to win — should we ship?" | C — interpretation traps |
| "Confidence intervals overlap — does that mean not significant?" | C — interpretation traps |
| "The p-value is exactly 1" / "control shows a lift" | C — interpretation traps |
| "An external tool (significance calculator or AI agent) disagrees with PostHog" | C — interpretation traps |
| "Should I ship? Primary is up but a secondary is down" | C — interpretation traps |
| "PostHog numbers ≠ my SQL count" | D — numbers vs SQL |
| "Funnel says X% but my raw event count says Y" | D — numbers vs SQL |
| "The experiment says flat and our dashboard shows a drop" | D — numbers vs SQL |
| "Sum of revenue looks wrong" / "breakdown shows 'none'" | D — numbers vs SQL |
| "The metric's name says one thing and the number says another" | D — numbers vs SQL |
| "Recordings don't match the stats" / "no recordings for exposed users" | D — numbers vs SQL |
| "I applied a filter but the user count didn't change" | D — numbers vs SQL |
| "I want to slice results by current person properties (as of now, not as of exposure)" | D — numbers vs SQL |
| "The results haven't updated" / "refresh changes nothing" | D — numbers vs SQL |
| "Changed split / rollout / metric / criteria mid-run, now odd" | E — mid-run changes |
| "The numbers changed overnight and nobody touched it" | E — mid-run changes, then D8 in D |
| "We paused it for a few days, can we still trust it?" | E — mid-run changes |
| "Froze the experiment" / "after the freeze users lost the variant" / "freeze was refused" | E — mid-run changes |
| "Ended/shipped — flag now flipped to 0/100 unexpectedly" | E — mid-run changes |
| "We shipped the variant and some users still don't get it" | E — mid-run changes |
| "Long-term metric moves opposite from primary" | E — mid-run changes |
| "Retention metric counts users I didn't expect" | E — mid-run changes |
| "Can't convert the feature flag back to a simple (boolean) flag after the experiment ends" | E — mid-run changes |
| "How do I restart an experiment with new variants?" | E — mid-run changes |
| "It says this is a legacy experiment" / metrics can't be edited | E — mid-run changes (E13 legacy experiments) |
"Results won't load" / many metric rows show data: null (not a legacy experiment) | Step 1.5 — diagnostic snapshot (null rows) |
| "What do users think of the new flow?" / wants qualitative feedback on an experiment | F — qualitative feedback |
| "Why did users prefer control?" / "what did they dislike about the test variant?" | F — qualitative feedback |
If the symptom is unclear, ask one clarifying question before picking. Most diagnostics have different fixes — do not guess. A complaint that matches rows of several groups gets each of them read (Step 3).
A report from a scout or another automated check names its finding: route by the finding, and reuse the report's numbers before pulling them again.
Step 3 — Surface every diagnostic the evidence supports
After matching the symptom in Step 2 and reading the relevant reference file(s), list each diagnostic that applies before recommending an action.
Surface co-occurring mechanisms independently — even when one is more salient, don't collapse them into a single "wait" or "fix" recommendation. Different mechanisms have different fixes: a systematic bias (e.g. uneven-split + Exclude) doesn't resolve by waiting; a statistical pattern (e.g. small-sample variance) does. Bundling them leaves the bias in place after the user follows the bundled advice.
Only list mechanisms that have a path to verification in the project state — config (from
experiment-get), snapshot data, activity log, or repo source (when the agent has the repository). Config-derived mechanisms count: an
80/20 split with default multiple_variant_handling="exclude" is visible in experiment-get and is
therefore enumerable. Naming a mechanism with no source (e.g. SRM when the snapshot shows a clean
variant ratio) is not.
Two checks belong to every diagnosis, whatever the complaint:
How young is the experiment? State its age (
start_date) and its share of the target sample (the exposed total againstrunning_time_calculation.recommended_sample_size) before reading a result (C1 inreferences/interpretation.md). The page takes the exposed total from the first primary metric: the sum of itsnumber_of_samplesover all variants in the stored results, notexposures.total_exposures.recommended_sample_sizeis the target the page saved when it last computed the estimate. When it is absent, no target was saved: say so.Who is exposed? Compare the exposed population (
exposure_criteriaand the flag's release conditions) with the people the decision is about before calling the setup sound (D12 inreferences/numbers-vs-sql.md). Default exposure settings are not evidence that the right people are in the experiment. When nothing says who the decision is about, ask.
Diagnostic groups
A — Bias & skew
Variants don't look balanced, one variant looks biased, the in-app warning banner appeared, or users are
showing up under multiple variants. Covers the uneven-split + Exclude interaction, the sample ratio mismatch and its causes, exposure events that are lost or sent unevenly between the arms, flags that return no variant for part of the audience, identity
fragmentation, bootstrap × /flags mismatch, mid-run flag edits, forced variants, and flag/experiment state inconsistency.
→ See references/bias-and-skew.md
B — Empty experiment / 0 exposures / "not enough data"
A frequent pain point. Covers SDK call (wrong evaluation method, identify() timing, dedup),
exposure capture (custom event missing variant property, required properties, ad-blockers), and
exposure-criteria match (test-account filter, filters that cannot match, eligibility ordering, events firing before exposure).
→ See references/empty-experiment.md
C — Significance / interpretation traps
Significance flipping, A/A test showing significance, Bayesian vs Frequentist confusion, multiple comparisons, low-volume variance, peeking / early stopping. Includes the running-time estimate ("target reached" is not a result) and the statistics settings that change how a result reads: sequential testing, CUPED, the confidence level and the baseline variant.
→ See references/interpretation.md
D — Numbers don't match (PostHog vs the user's SQL / raw count)
The experiment page applies an exposure scope, $multiple exclusion, test-account filter, and date range
that ad-hoc SQL almost never replicates. Covers funnel attribution (only first→last step counts for stats),
breakdowns (read from the exposure event, not the metric event), the "sum of revenue" mean-of-per-user
confusion, stale stored results, the Recordings tab against the stats, an insight that counts a narrower population than the experiment, and a metric whose title does not match its definition.
→ See references/numbers-vs-sql.md
E — Surprises after mid-run changes (incl. lifecycle and retention quirks)
Increasing rollout is safe; decreasing is caution; changing the variant split is an anti-pattern; adding metrics mid-run is p-hacking; ship-variant rewrites the flag and ending alone does not; reset clears results not the flag (apart from removing an exposure freeze); a pause, a freeze and an unfreeze can each bias the result; a changed start date moves the whole analysis. Also covers retention-metric quirks, "matured users" filtering, and long-term vs short-term metric divergence.
→ See references/mid-run-changes.md
F — Qualitative feedback: how the change landed, not how far the number moved
Groups A–E find mechanisms. F is for the question they can't reach: what the people in the experiment made of the change. A short survey, shown when users finish the experimented flow, adds that qualitative half — a rating and an optional comment, readable per variant — and works over MCP today.
It suits some experiments and not others. The gate is whether a user could describe the change without being shown both versions: a reworked flow, layout, or process work; a threshold or ranking tweak don't, however large its measured effect. Don't offer it while a mechanical diagnostic is still open — a survey on top of a broken flag gate collects opinions about a feature half the audience never received — and check whether an existing or recent survey already covers the window before proposing a new one.
Two things to get right before creating one: a survey shown to a single variant is itself a difference between the variants, and a running survey linked to the flag can generate exposure events.
→ See references/qualitative-feedback.md
Step 4 — Calibrate recommendations to experiment state
Surface diagnostics first (Step 3). Then recommend — but scope what you recommend to what the experiment's current state permits.
Draft — config changes are free; recommend and apply. The flag of a draft can still serve users: a draft made by a reset, or created on an existing flag, can have
feature_flag.activetrue, and an edit of that flag reaches those users at once.Running (and
pausedorexposure_frozen, which are launched and not ended) — every change has a tradeoff. Explain the mid-run impact (anti-pattern? safe? user-visible?) before recommending. Seeconfiguring-experiment-rolloutand its reference filereferences/changing-distribution-after-launch.mdfor the mid-run rules. A pause, a freeze, an unfreeze and a reset are changes too: each one interrupts the run and can bias the result (E8, E9 and E16 inreferences/mid-run-changes.md).Stopped / archived — the experiment AND its feature flag represent the documented outcome of the run. Recommendations are scoped to (a) interpretation of the existing data, (b) what to do for the next experiment, or (c) explaining what happened.
A diagnosis ends with findings and a recommendation. A change to a running experiment or to its flag (a new metric, new exposure criteria, a reset, a relaunch, a flag edit) is the user's decision: say what the change does to the data collected so far, and make it only on an explicit instruction.
On a stopped or archived experiment, don't preemptively offer reversal of a state mutation (ship-variant flag rewrite, manual flag edit, reset, archive). If the user asks "why did X happen?", explain X — don't append a "here's how to undo it" coda. That pattern assumes intent the user didn't signal. Conditional offers like "if this wasn't intended, you could…" or "want me to revert it?" count as preemptive too — only the user explicitly naming the reversal action ("how do I undo this?", "can I roll back ship-variant?", "how do I get the 50/50 split back?") is a request to surface reversal mechanics.
Use consistent terminology: variant split (between variants) is distinct from rollout (the share of a release condition's audience that
enters); the default exposure event (resolved_exposure_event) is distinct from a custom exposure event; the
Exclude from analysis / Use first seen variant options of Multiple variant handling (exposure_criteria.multiple_variant_handling) decide how people exposed to several variants are counted, not which event is the exposure.
Related skills
configuring-experiment-analytics— fix the exposure or metric configuration the diagnosis points atconfiguring-experiment-rollout— split-change anti-patterns and safe rollout adjustmentsmanaging-experiment-lifecycle— reset, end, or restart when the experiment can't be salvaged in placeanalyzing-experiment-session-replays— when the numbers are fine but you need to see the behavior behind themdebugging-experiments— when the question arrives as a support request about someone else's experiment and the answer is a written reply
Files
- ATTRIBUTION.mdAdded by AutoGineer875 B
- LICENSELicence file added by AutoGineer1.0 KiB
- SKILL.md23.3 KiB
- references/bias-and-skew.md49.9 KiB
- references/diagnostic-snapshot.md20.7 KiB
- references/empty-experiment.md33.7 KiB
- references/interpretation.md30.3 KiB
- references/mid-run-changes.md32.4 KiB
- references/numbers-vs-sql.md36.5 KiB
- references/qualitative-feedback.md17.8 KiB
references/empty-experiment.md
Empty experiment / 0 exposures / "not enough data"
Diagnose by walking the chain: SDK call → exposure event captured → ingested → matches the configured exposure criteria → counted.
Which exposure event? When exposure_criteria names no custom event, the experiment counts exposures on its default event.
Read it from resolved_exposure_event in experiment-get: $feature_flag_called, or $experiment_exposure for an experiment that started on or after 2026-09-01 (UTC) in a project that is in the rollout of that event.
When it is $feature_flag_called, also read $experiment_exposure (event IN ('$feature_flag_called', '$experiment_exposure')), and count persons, not rows.
An exposure_criteria.exposure_config that names $experiment_exposure counts that event, whatever resolved_exposure_event says.
A config that names $feature_flag_called counts the default event, as an absent config does.
$experiment_exposure is a copy of $feature_flag_called, written at ingestion with the same properties, and only for a string response other than true, false and the empty value.
So the SDK-side diagnostics below apply to both: an SDK that never fires $feature_flag_called produces neither event.
The reverse case, $feature_flag_called with variant responses and no $experiment_exposure, is not an SDK fault ("Which event, which property" in diagnostic-snapshot.md).
What counts as an exposure. An event counts only when all of these hold:
it is the experiment's exposure event, inside the experiment's window;
its variant property holds one of the flag's current variant keys (
$feature_flag_responseon a default event,$feature/<flag-key>on a custom event), and that key is not inexcluded_variants;on a group-aggregated flag, it carries the key of that group type;
with an activation event (
exposure_criteria.activation_config), the person also sent that event at or after their first flag exposure, and counts from then;it passes the test-account filter and every filter of the exposure criteria.
An event that fails one of them is not counted at all. There is no "no variant" bucket. A launched experiment without a counted exposure shows a total of 0 and a flat chart, and its open Exposures panel says "No exposures yet". The panel says the same for a draft, for a start date after today, and for an exposure query that failed.
The queries this file refers to are in diagnostic-snapshot.md, with their rules.
Contents
Quick triage decision tree
B0 — Fresh-launch check (launched minutes or hours ago)
B1 — Wrong flag-evaluation method (no exposure recorded)
B2 —
identify()timing and ids that are never linkedB3 —
$feature_flag_calleddeduplication per identityB4 — Custom exposure event missing variant property
B5 — Required properties on
$feature_flag_calledB6 — Ad-blockers / network drops
B7 — Test-account filter hides the data
B8 — Metric events firing before exposure
B9 — Eligibility check ordered after the flag check
B10 — "Variant always undefined / false"
B11 —
$feature/<key>is missing on metric eventsB12 — An exposure criteria filter that cannot match
If none of the above: the code path may not be running
Quick triage decision tree
The tree is for an experiment that counts 0 exposures, or 0 in one variant.
For exposures that grew and then stopped, go to the closing section.
For exposures without conversions, go to B2, B8 and B11, then to D11 in numbers-vs-sql.md.
Check directly, in this order, and ask only what the tools cannot answer:
How long ago was the experiment launched?
start_datefromexperiment-get. Null means it is a draft. Minutes or a few hours → B0 first.Has the code that calls the flag been deployed and is traffic flowing through it? If no → the experiment will be empty until that ships. Stop here.
What did the application send? Run the raw-responses query for the flag.
Events with a variant response, and the experiment still counts 0 → the criteria remove them (B4, B7, B12), the variant is in
excluded_variants, the flag is aggregated by a group and the events carry no group key, an activation event is set and nobody sent it after the flag call, or the 0 is a result computed before the events arrived (last_refresh, B0).Events whose response is no variant (
false, empty) → B10, and A10 inbias-and-skew.mdwhen only a part is affected. On a custom exposure event, values of$feature/<flag-key>that are empty or no variant key → B4.One variant has events and another has none → a variant at 0% or a pinned variant (A7b in
bias-and-skew.md, E3 inmid-run-changes.md), or code that serves one arm only (closing section).No events at all → SDK or capture (B1, B6), a read before the SDK has loaded the flags (B10), events sent without the flag's key (B5), or the code calls another flag key (closing section). When the experiment counts
$experiment_exposure(resolved_exposure_event, or anexposure_configthat names it), run the query on$feature_flag_calledtoo before you conclude that nothing is sent.
Are some people exposed and others not? → B3 (returning users who do not send the event again), a read before the SDK has loaded the flags that the code does not repeat (B10: visitors without flags from an earlier visit send nothing), or A10 in
bias-and-skew.md.
B0 — Fresh-launch check (launched minutes or hours ago) [MEDIUM]
Evidence. start_date from experiment-get, against the current time.
start_dateis null: the experiment is not launched. Recommend launching.Exposures above 0 for any variant: B0 is ruled out.
What a fresh experiment shows [HIGH].
For its first 12 hours an experiment reads exposures directly from events, and the page forces one fresh read on each load while the experiment has fewer than 50 exposures in total. So a correctly wired experiment shows its first exposures on the page within minutes of the first flag call. The delay is ingestion.
experiment-results-getis different: it can return a result up to 24 hours old, so a 0 that it read earlier can outlast the first exposures (last_refreshon the result says how old it is).A variant with fewer than 50 exposures has no result yet: the metric reads "Not enough data yet". That is the floor, not a fault (C2 in
interpretation.md).Stored results may not exist yet for a fresh experiment. A read of the stored results that returns nothing in the first hours is expected. The scheduled calculation skips an experiment younger than 12 hours or with fewer than 50 exposures in total, so an empty experiment has stored results only when a calculation was started another way, for example from the experiment page.
In a project with precomputed exposures, an experiment older than 12 hours can lag today's new exposures by up to 30 minutes. The total is 0 because of that only when every exposure is younger than 30 minutes.
<!-- Source for maintainers (may rot): MIN_PRECOMPUTATION_DURATION_SECONDS and DEFAULT_EXPOSURE_TTL_SECONDS in products/experiments/backend/hogql_queries/precompute_policy.py; ExperimentResultsCacheMixin in products/experiments/backend/hogql_queries/experiment_query_runner.py; validate_variant_result in products/experiments/backend/hogql_queries/utils.py (the per-variant floor); NEW_EXPERIMENT_FORCE_REFRESH_AFTER_MINUTES and EXPERIMENT_MIN_EXPOSURES_FOR_RESULTS in products/experiments/frontend/constants.ts (the forced page read); MIN_EXPERIMENT_AGE and MIN_TOTAL_EXPOSURES in products/experiments/backend/temporal/scheduled_recalculation_logic.py. Verify before citing. -->
Reading. Zero exposures minutes after launch, on a surface with little traffic, is waiting. Zero exposures an hour after launch, on a surface that people visit, is a real B-series case: walk B1 to B12.
B1 — Wrong flag-evaluation method (no exposure recorded) [MEDIUM]
Only a read of one flag's value records exposure. A read of all flags, of a payload, or of the
values a callback hands over does not fire $feature_flag_called.
| SDK | Records $feature_flag_called | Does NOT record |
|---|---|---|
| posthog-js | getFeatureFlag(), getFeatureFlagResult(), isFeatureEnabled(), the React hooks useFeatureFlagVariantKey(), useFeatureFlagEnabled() and useFeatureFlagResult() | getFeatureFlagPayload() (deprecated for this reason), getFlags(), getFeatureFlagDetails(), the values passed to onFeatureFlags, the React hooks useFeatureFlagPayload() and useActiveFeatureFlags() |
| posthog-node | evaluateFlags() then getFlag() or isEnabled() on the result. Older: getFeatureFlag(), getFeatureFlagResult(), isFeatureEnabled() | getFlagPayload() on the result, getFeatureFlagPayload(), getAllFlags(), getAllFlagsAndPayloads() |
| posthoganalytics (Python) | evaluate_flags() then get_flag() or is_enabled() on the result. Older: get_feature_flag(), get_feature_flag_result(), feature_enabled() | get_feature_flag_payload(), get_all_flags(), get_all_flags_and_payloads() |
The pattern across SDKs: methods that ask about one specific flag fire exposure; methods that return the whole flag bag or just a payload don't. Other SDKs (Ruby, Go, PHP, mobile) follow the same shape — when in doubt, check that SDK's docs.
An option can switch the event off for a call that would send it: send_event: false in posthog-js, sendFeatureFlagEvents: false in posthog-node, send_feature_flag_events=False in Python.
Verify: the raw-responses query shows no events for the flag while the surface has traffic. Then ask which SDK method reads the flag, and whether the value is read directly or pulled from a cached bulk result.
Fix: switch to the single-flag read. If
the user genuinely needs the bulk accessor, they must additionally fire $feature_flag_called
themselves with the right properties (see B5).
B2 — identify() timing and ids that are never linked [LOW]
The experiment joins a person's metric events to that person's exposure. Two shapes break the join or the assignment:
identify()runs after the flag is evaluated. The exposure is sent under the anonymous id. Once the two ids are linked, exposure and later events belong to one person, and the metric counts. The exception is a project that keeps each event's person id as it was written: there a merge of two persons that both existed does not move the exposure sent before it. The risk is a second evaluation under the identified id that lands in another variant (A3 inbias-and-skew.md).The ids are never linked. The exposure is sent under one id and the metric events under another that PostHog never joins to it: a second pipeline (a customer data platform, a backend) that sends events under its own anonymous id, or anonymous events sent without a person profile in a project that keeps each event's person id as it was written. The experiment then shows exposures and almost no conversions.
Evidence. Take one person who converted.
Compare the distinct_id of the exposure event with the distinct_id of the metric event, and check whether both belong to one person.
When the ids are never linked, the person who converted has no exposure event at all.
Then compare properties.$lib and the form of distinct_id on that person's metric events with those on the exposure events: another library or another form of id points to a second pipeline.
Common symptoms:
Exposure events exist but don't match later events under the same person
Variant-specific metric counts are far below exposures
Fix: call identify() before flag evaluation. Never re-identify() to a different distinct_id
mid-session.
Give every pipeline that sends events for one visitor the same id from the first event on.
B3 — $feature_flag_called deduplication per identity [MEDIUM]
PostHog SDKs deduplicate $feature_flag_called to avoid flooding ingestion with identical exposure
events. The scope of "duplicate" varies by SDK:
posthog-js dedupes per identity across sessions by default. Returning users who evaluated the flag before the experiment launched will not re-emit exposure on later visits — they look like they've never seen the flag. Enable
advanced_feature_flags_dedup_per_session: trueto reset the cache each session. The key is the flag together with its response, so a person whose response changes sends a new event.posthog-node / posthoganalytics (Python) dedupe per distinct id, flag, response and groups within the process lifetime, in memory. The cache is emptied when the process restarts. At 50,000 distinct ids posthog-node empties it (the
maxCacheSizeoption sets the limit), while Python drops only the oldest distinct id for each new one.Mobile SDKs (iOS / Android / React Native / Flutter) dedupe per session, not across sessions — the "returning user with stale dedup" shape is a web concern.
Where it bites: a flag that people evaluated before the experiment started.
The flag was active before launch.
The flag key is reused from an earlier experiment or an earlier run. The same key also gives every person the same draw as before.
The experiment was reset and relaunched on the same flag.
Returning people already sent the event with the same response, so they send nothing after launch and are missing from the experiment.
Evidence. The flag's history in experiment-activity: was the flag active before start_date?
The history holds this flag's own changes, and only when the user can view the flag.
It does not list other experiments that used the flag, nor a deleted flag with the same key: ask the user whether the key was used before.
First-time visitors are exposed and returning visitors are not.
Fix: match the dedup strategy to the user's complaint with a concrete change:
Web with returning users (
posthog-js). In the SDK init config, setadvanced_feature_flags_dedup_per_session: true. The cache resets each session, so returning users re-emit exposure once per session and the experiment captures them.A new run: a new flag key (E15 in
mid-run-changes.md).Server-side long-lived workers (
posthog-node,posthoganalytics). Two paths, pick one: (a) restart workers more frequently so the in-memory cache flushes more often, or (b) bypass SDK dedup by firing a custom exposure event yourself (see B4) — the experiment can then use that event as its exposure criterion instead of$feature_flag_called. Option (b) is the cleaner fix when you also want the exposure to carry custom properties.
Repeated exposure events with the same variant are harmless: the experiment counts each person once, at the first exposure.
A person whose exposure events carry two variants is $multiple, and under the default multiple_variant_handling (exclude) leaves the analysis (A3 in bias-and-skew.md).
B4 — Custom exposure event missing variant property [HIGH]
If the experiment uses a custom exposure event instead of the default one, the event must
include $feature/<flag-key> with the variant value (e.g. $feature/new-checkout: 'control').
Without it the event is not an exposure.
The same holds for a value that is not exactly one of the flag's variant keys: another spelling, another case, true, a key of another flag.
Evidence. The custom-exposure query of the snapshot lists the values of the property with their counts.
Compare them with feature_flag.filters.multivariate.variants[].key:
every row empty → the sender does not set the property;
values that differ from the variant keys → the sender sets another value.
Fix: set the property on the event in the tracking code.
posthog-js adds it to every event once flags have loaded, except while its flag cache is older than feature_flag_cache_ttl_ms when that option is set [MEDIUM].
An event that arrives another way carries it only when the sender adds it: a server SDK adds it only when the capture call is given the evaluated flags (flags) or asked to evaluate them (sendFeatureFlags: true in posthog-node, send_feature_flags=True in Python), and a customer data platform or a hand-written event never adds it [MEDIUM].
Placebo / variant-less experiments still need the property. A "no UX impact" experiment
(common for instrumentation-only or breakdown-driven analyses) requires $feature/<flag-key> on
the custom exposure event just like any other experiment. PostHog uses the property for variant
attribution, not for product behavior.
B5 — Required properties on $feature_flag_called [HIGH]
An exposure event that the user's own code sends (a bulk accessor plus a hand-written event, a third-party path) must carry:
$feature_flag— the experiment's flag key$feature_flag_response— one of the flag's variant keys
One such event per person is enough: the experiment takes each person's first exposure.
Evidence. The raw-responses query of the snapshot.
No rows for the flag key means the events carry another key or none, or that no event is sent at all (B1).
The same query without the $feature_flag condition, grouped by properties.$feature_flag, shows which keys the events carry: one day only, since it reads every flag of the project.
Rows whose response is no variant key are not counted.
Fix: set both properties on the event.
B6 — Ad-blockers / network drops [MEDIUM]
Common cause of partial or zero data. The SDK call goes out, but the request never reaches PostHog.
A consent step has the same effect [LOW]: nothing is captured before the visitor opts in, and when flags load only after consent, every read before it returns no variant.
With posthog-js opted out by default (opt_out_capturing_by_default: true) and persistence left on, a read before opt-in returns the variant and records the exposure as sent, while the event itself is dropped.
After opt-in the same read sends nothing until the distinct id changes, so that person is never exposed unless advanced_feature_flags_dedup_per_session is on (B3) [MEDIUM].
Fix: set up a reverse proxy so capture requests come from the user's own domain, which ad-blockers don't block.
B7 — Test-account filter hides the data [HIGH]
exposure_criteria.filterTestAccounts defaults to true. If the user's own traffic matches the
project's test-account filter (e.g. their email domain is in the filter), their events are excluded from
the experiment.
When a rule of the project's filter is inverted, so that it keeps what it was meant to remove, every production event is dropped.
Evidence.
The filter.
project-getwithout an id returns the active project'stest_account_filters: an array of{ key, type, value, operator }conditions, withtypeusuallyperson,eventorcohort, and sometimesgrouporelement. An event is kept only when it matches every row. Rows are normally negative (is_not,not_icontains, andnot_inon the cohort row a new project starts with): a positive row (exact,icontains,in) keeps only what it names, so check whether that is intended.The effect. Compare the people in the raw-responses query (no filter) with
exposures.total_exposures(filtered) over the same window. People in the raw events and none in the experiment, with exposure criteria that hold nothing else, points to this cause. The other causes in the same branch of the triage tree give the same picture, so confirm it: add the rows to the raw-responses query one at a time (properties.<key>for an event row,person.properties.<key>for a person row,person_id IN COHORT <id>for a cohort row), and the row after which the count falls to zero, or nearly, is the cause.Summarize the rows in plain language so the user can recognize their own traffic. Don't assume what the rows contain — they vary per project (common shapes: email-domain exclusions, localhost host filters, internal IP ranges, specific cohorts).
Fix: turn filterTestAccounts off for this experiment, or correct the project's filter.
Both change which exposures count for the whole run (E5 in mid-run-changes.md).
B8 — Metric events firing before exposure [HIGH]
Metric events that occur before a user's first exposure are ignored. Only events after exposure are included in the calculation.
Common cause: the exposure event fires too late in the user journey. For example, if the metric event is
signup_completed and the exposure event is on a checkout page that the user only reaches after signup,
exposures will lag the metric and the metric appears to barely register.
Verify directly. One scan over both events, for the people who sent the metric event in the window:
SELECT
countIf(exposure_events = 0) AS persons_with_metric_and_no_exposure,
countIf(exposure_events > 0 AND last_metric < first_exposure) AS persons_with_every_metric_event_before_exposure,
countIf(exposure_events > 0 AND last_metric >= first_exposure) AS persons_with_a_metric_event_after_exposure
FROM (
SELECT
person_id,
countIf(event = '<resolved_exposure_event>') AS exposure_events,
minIf(timestamp, event = '<resolved_exposure_event>') AS first_exposure,
maxIf(timestamp, event = '<metric-event>') AS last_metric
FROM events
WHERE timestamp >= toDateTime('<window_start>', 'UTC')
AND timestamp < toDateTime('<window_end>', 'UTC')
AND (
event = '<metric-event>'
OR (
event = '<resolved_exposure_event>'
AND properties.$feature_flag = '<flag-key>'
AND properties.$feature_flag_response IN ('<variant-key-1>', '<variant-key-2>')
)
)
GROUP BY person_id
HAVING countIf(event = '<metric-event>') > 0
)On a custom exposure event, put that event in place of <resolved_exposure_event>, drop the $feature_flag condition, and test properties.`$feature/<flag-key>` in place of $feature_flag_response.
If the second count dominates, the exposure event is firing too late in the journey — confirmed B8. The first count also holds every person outside the experiment: with a rollout under 100%, release conditions, or a metric event that also fires on other surfaces, it is large by design. If it dominates beyond that, the people who convert are not being exposed at all (back to B1, B2, B10), or they were exposed before the window.
Fix: capture exposure at the first encounter with the experimental change, not later in the flow.
An exposure placed late can also differ between the arms (A9 in bias-and-skew.md).
B9 — Eligibility check ordered after the flag check [MEDIUM]
Eligibility filtering should happen before you call the flag — otherwise unaffected users are pulled into the analysis and the picture gets noisy. This shows up as exposures being much higher than expected and metric rates unexpectedly low.
The same shape has other sources: a flag that is read on more surfaces than the one under test, and an experiment on new users whose flag existing users also evaluate.
In each, the experiment counts people the decision is not about (D12 in numbers-vs-sql.md).
Fix: structure the code as: eligibility check → flag check → render. Not: flag check → eligibility → render. Where the code cannot change, narrow who counts: a filter in the exposure criteria, which removes a person only when every exposure event of that person fails it (B12 lists what such a filter can read), or a custom exposure event sent at the point of the change.
B10 — "Variant always undefined / false" [MEDIUM]
Almost always one of:
B1 (wrong evaluation method), when the code reads a payload:
getFeatureFlagPayload()returns the payload, which is empty for a flag without oneB2 (
identify()timing), on a flag whose release conditions read person properties that are set atidentify(): a read before it gets no variantposthog is not defined(SDK init order — initialize PostHog before any flag call)The code reads the flag before the SDK has loaded the flags. posthog-js then returns
undefinedand sends no$feature_flag_called, unless the browser holds flags from an earlier visitThe flag is genuinely off —
feature_flag.active === false(statusispaused), or rollout0%, or the user is outside release conditionsThe flag buckets on the device id, and the call carries none (A10 in
bias-and-skew.md)The code reads the multivariate flag as a boolean and gets
true, or replaces "no variant" with a default of its own
Fix: walk the user through their SDK setup. Verify in this order: (a) is PostHog initialized? (b) is the flag active and rolled out? (c) is the right variant key being requested?
When only a part of the audience gets no variant, the question is who: A10 in bias-and-skew.md.
B11 — $feature/<key> is missing on metric events [HIGH]
For a current experiment this is not a fault.
Only the exposure event decides a person's variant.
Metric events are matched to that person and need no flag property, for every metric type.
A user who adds $feature/<flag-key> to metric events changes nothing in the results.
A legacy experiment shows no results at all (E13 in mid-run-changes.md).
Where the property does matter:
A custom exposure event needs it (B4).
The user's own insight or SQL that filters or breaks down by
$feature/<flag-key>. It undercounts events from a server SDK, which adds the property only when the capture call is given the evaluated flags or asked to evaluate them[MEDIUM]. It also overcounts in the other direction: the property is on events of everyone whose SDK holds the flag's value, including people who were never exposed, so it is not a filter for participants.
Verify directly. A missing conversion has another cause: B2 (ids), B8 (order), D1 in numbers-vs-sql.md (scope).
Part of the public docs still asks for the property on metric events. The exposures page states the current rule.
<!-- Source for maintainers (may rot): build_variant_property in products/experiments/backend/hogql_queries/experiment_exposure_query_builder.py reads the property on the exposure side only. Docs: https://posthog.com/docs/experiments/exposures#metric-events-dont-need-a-variant-property -->
B12 — An exposure criteria filter that cannot match [MEDIUM]
Symptom (in the user's vocabulary): "exposures dropped to zero after I added a filter", "the experiment shows far fewer users than the flag", "I copied the experiment and the new one is empty".
Behavior. A filter in the exposure criteria is evaluated on each exposure event. It reads the event's own properties, and person properties either as they were attached to that event or as the person holds them now: which of the two depends on a project setting that the tools do not show. A filter that names something the exposure event cannot carry removes every exposure, and nothing reports it.
Shapes:
A property the exposure event does not have. A browser property (device type, URL) on an exposure that a server sends.
A person property that is set after the exposure, such as a sign-up timestamp on an exposure that happens before sign-up. This shape occurs only where person properties are read as they were attached to the event.
A misspelled property name or value.
Criteria copied from another experiment that name the other experiment's flag or event.
An exposure event the project has never received.
A URL filter, or another event-property filter, that is meant to keep people out. It drops the events that do not match. A person with one matching exposure event is in, with everything they did.
Evidence to gather.
exposure_criteria.exposure_configfromexperiment-get: the event and itsproperties.The raw-responses query once without and once with the filter's condition. A count that falls to zero, or nearly, names the filter. One condition at a time when there are several.
For a person-property filter: the raw-responses query with the same condition on
person.properties.<key>. That query reads person properties the same way the experiment does, so the check holds whichever way the project reads them.
Co-occurs with: B7, and D9 in numbers-vs-sql.md.
Recommendation. Remove or correct the filter.
Filter on what the exposure event carries.
To restrict the experiment to a group of people, use the flag's release conditions.
An edit of the exposure criteria recomputes the whole run (E5 in mid-run-changes.md).
If none of the above: the code path may not be running
Two sub-cases:
Never had exposures (the experiment has shown 0 since launch). After B1–B12, check the obvious:
Has the deploy with the flag-reading code shipped to production?
Is real traffic flowing through that code path?
Does the code read the experiment's flag key, and not another key?
Is the date range correct (start_date in the future, etc.)?
Ask explicitly. The "empty experiment" shape often resolves to a feature flag still on a feature branch that hasn't merged, or a page that calls the flag not being live yet.
Exposures were healthy then stopped (the experiment ran for weeks/months, then the exposure count plateaued and never moved again). Before investigating, check the experiment's status from Step 1:
exposure_frozen— someone froze exposure. The plateau is the intended behavior: enrollment is closed, metrics still flow (E16 inmid-run-changes.md).paused— the flag is off.stopped— the results end at the end date.
Then check where the project keeps its flag calls: project-get returns flag_evaluations_mode.
At 2, $feature_flag_called no longer arrives in events for the projects that setting covers.
The default exposure keeps counting from the $experiment_exposure copies ("Which event, which property" in diagnostic-snapshot.md).
An exposure or activation config that is an action with a $feature_flag_called step can stop gaining exposures, without a warning.
That is no fault of the SDK or of the application: say so, and point the user to PostHog support.
Then read the flag's change history, before any conclusion about the application:
A change of the flag after launch can explain the plateau by itself. A split edited to 0% for one variant, a variant pinned on a broad release condition, or a rollout cut to 0 stops one arm or all of them while
statusstill readsrunning(A5, A6 and A7b inbias-and-skew.md, E3 inmid-run-changes.md).No change of the flag after launch means that an edit of the rollout, the split or the release conditions cannot explain it. What is left: the application stopped calling the flag, a release condition on a cohort or a property stopped matching new people, the audience has no new people, or the project's test-account filter changed, which the experiment's history does not show.
Verify directly:
exposures.timeseriesin the exposure data: the cumulative count per variant stops growing. One arm flat while the other grows points to the flag or to one code path. All arms flat points to the application, or to an audience without new people: the series counts each person once.last_seenper variant in the raw-responses query, over the last days: no recent events means the application stopped sending them, or that only people who already sent the event still reach the code (B3). Both variants silent is a code-path removal, or an audience without new people. One variant silent while the other still fires is a one-sided change, in the flag or in the code.
Common causes:
The flag-reading call was removed in a refactor (most common).
The page or component that hosts the flag-read was deprecated or rerouted (e.g. URL restructuring moved the eligible traffic onto a different page that doesn't read this flag).
A different flag is now serving the same UX (intentional migration that wasn't paired with ending the original experiment).
Recommend:
If the hypothesis is settled enough: end the experiment with the appropriate conclusion (won / lost / inconclusive). The metric data accumulated before the plateau is the experiment's documented outcome. Don't ship the variant unless the code path is being restored — an "end + ship" on a dormant flag flips the variant distribution to a UX that isn't being served anyway.
If you want to keep running the hypothesis: restore the flag-reading call in the application code, then either continue (and treat the pre-/post-resumption windows separately) or reset + relaunch for a clean comparison window.
SDK-side fallback. If B1, B2, and B10 are all on the table and you can't pin one down, invoke the
posthog:diagnosing-sdk-health skill — an outdated SDK is one possible cause of the "no exposures
at all" shape (missing instrumentation, broken identify() ordering, deprecated flag methods).
Requirements and compatibility
Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Security
No issues found by our static scan· No issues found
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Published by
- PostHog
- Publisher
- PostHog (Official)
- Source
- PostHog/skills · skills/omnibus/diagnosing-experiment-health · commit a5d30e7
- Licence
- MIT License (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by PostHog or PostHog.
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download diagnosing-experiment-health.zip