Peer Review

Statistical, Methods, and Reproducibility Review

This guide supports structured questions; it does not replace a statistician, methodologist, domain expert, or independent reanalysis. Sources verified on 2026-07-23 are recorded in assets/source_ledger.csv.

Evidence hierarchy for the review

Use, in order:

1. The stated research question and target population

2. Protocol, registration, analysis plan, and amendments

3. Reported design and data-generating process

4. Methods, code, tables, figures, supplements, and repository records

5. Applicable primary method or regulatory guidance

6. Current reporting guidance

7. Target venue policy

Do not reject a method merely because another method is more familiar. Explain the estimand, assumption, error, or interpretation at stake.

Core review sequence

1. Define what is being estimated

Write down:

• Unit of inference

• Population

• Intervention, exposure, test, or predictors

• Comparator or reference condition

• Outcome and time horizon

• Target effect, association, accuracy, or predictive performance

• Intercurrent events, censoring, and missing observations where relevant

Then trace whether design, data collection, analysis, result, and claim target the same quantity.

For applicable clinical trials, ICH E9(R1) provides a framework for estimands and sensitivity analyses. It is not a universal rule for every study.

2. Reconstruct the design

Identify:

• Prospective, retrospective, cross-sectional, longitudinal, experimental, or observational structure

• Recruitment or sampling frame

• Experimental/observational unit

• Pairing, nesting, clustering, repeated measures, sites, batches, and time

• Allocation, concealment, blinding, matching, or weighting

• Primary and secondary outcomes

• Prespecified versus exploratory analyses

If the design cannot be reconstructed, first request missing reporting. Do not label the design invalid solely because details are absent.

3. Trace every denominator

Reconcile:

• Eligible, enrolled, assigned, treated/exposed, followed, measured, and analyzed units

• Outcome-specific denominators

• Exclusions before and after allocation or measurement

• Missing values and reasons

• Complete-case, imputed, weighted, or model-based analysis populations

• Figure, table, abstract, text, and supplement totals

Report the exact mismatch and locations; do not infer why counts differ.

4. Assess analysis–design alignment

Ask whether the method accounts for:

• Outcome scale and distribution

• Pairing and repeated measures

• Clustering and multilevel structure

• Unequal follow-up, censoring, or competing events

• Sampling weights or matched designs

• Baseline adjustment and prespecified covariates

• Multiplicity and outcome hierarchy

• Model tuning and validation

• Missingness assumptions

The name of a statistical test is not enough. The report should state inputs, model form, uncertainty method, software/version, and relevant diagnostics.

5. Assess estimates and interpretation

Prefer:

• Effect or performance estimates with units

• Compatible uncertainty intervals

• Absolute as well as relative quantities when decision-relevant

• Exact denominators and analysis sets

• Assumption and sensitivity context

• Clinical, biological, policy, or practical relevance distinct from statistical compatibility

The ASA’s six p-value principles include:

• A p-value is about incompatibility with a specified model, not the probability a hypothesis is true.

• Threshold crossing alone should not determine scientific conclusions.

• Transparent reporting of all relevant analyses is required.

• Statistical significance does not measure effect size or importance.

• A p-value alone is not a good measure of evidence.

SAMPL provides concise biomedical statistical reporting guidance. Apply it as reporting guidance, not a universal analysis recipe.

Topic-specific checks

Sample size and precision

Look for:

• Prospective calculation or precision rationale

• Target effect or interval width

• Variance, event rate, prevalence, or accuracy assumptions

• Type I error, power, sidedness, and multiplicity when applicable

• Design effect, clustering, attrition, noncompliance, and missingness

• Model complexity and effective sample size

• Simulation details for complex designs

Do not request observed/post hoc power as a remedy for an imprecise result. Examine the estimate and uncertainty.

Randomized trials

Check:

• Allocation sequence and concealment

• Prespecified estimand and analysis population

• Protocol/registry/outcome consistency

• Baseline adjustment and stratification factors

• Intercurrent events, adherence, treatment switching, and missing data

• Harms and unintended effects

• Sensitivity and supplementary analyses

• Non-inferiority/equivalence margin and interpretation if relevant

Use CONSORT 2025 and SPIRIT 2025 for reporting. Use ICH E9/E9(R1) only when its scope and decision context fit.

Observational causal analyses

Check:

• Causal question and target contrast

• Time zero, eligibility, treatment/exposure assignment, follow-up, and outcome timing

• Confounder rationale and measurement timing

• Positivity/overlap

• Exchangeability and consistency assumptions

• Missingness, censoring, selection, and measurement error

• Model specification and balance diagnostics

• Sensitivity to unmeasured confounding or alternative specifications

Avoid judging causal identification from adjusted versus unadjusted p-values.

Diagnostic accuracy

Check:

• Intended use, setting, and participant spectrum

• Index test and reference standard

• Threshold prespecification

• Blinding and timing

• Indeterminate/missing results

• Verification and incorporation bias

• Two-by-two denominators and uncertainty

• External applicability

STARD/STARD-AI describe reporting; use an appropriate risk-of-bias framework separately.

Prediction models

Check:

• Intended use, prediction time, outcome, and target population

• Data source and participant flow

• Predictor availability at intended use

• Missing-data and preprocessing leakage

• Sample size relative to outcome information and complexity

• Internal validation and optimism correction

• Independent evaluation and dataset shift

• Calibration, discrimination, decision utility, and uncertainty

• Hyperparameter tuning separated from evaluation

• Reproducible model specification and preprocessing

• Subgroup performance tied to plausible use and harms

TRIPOD+AI replaces TRIPOD 2015 for regression and machine-learning prediction model reporting. STARD-AI is more appropriate when diagnostic accuracy of an index test is the primary aim.

Systematic reviews and meta-analyses

Check:

• Protocol and registration

• Eligibility criteria and information sources

• Reproducible search dates and strategies

• Duplicate screening/extraction processes or justified alternatives

• Risk-of-bias assessment

• Effect measure and synthesis model

• Heterogeneity and prediction intervals when appropriate

• Dependence among estimates

• Small-study and reporting biases

• Certainty assessment, if claimed

• Transparent deviations and unavailable data

PRISMA 2020 assesses reporting. Do not substitute PRISMA coverage for review-conduct appraisal.

Clustered and longitudinal data

Check:

• Level of assignment, measurement, and inference

• Within-cluster/subject correlation

• Number and distribution of clusters

• Small-cluster corrections where needed

• Time structure, nonlinear change, and irregular measurement

• Informative visit, dropout, or censoring processes

• Cluster-level versus individual-level covariates

Repeated observations do not increase independent sample size one-for-one.

Multiplicity

Identify the inferential family before recommending adjustment:

• Multiple primary outcomes

• Multiple intervention arms or contrasts

• Repeated time points

• Subgroups and interactions

• Interim analyses

• High-dimensional features

• Model selection

Possible responses include hierarchical testing, family-wise control, false-discovery control, multilevel modeling, transparent exploratory labeling, or emphasis on estimates and uncertainty. The remedy depends on the claim and decision rule.

Missing data

Check:

• Missingness by group, variable, outcome, and time

• Reasons and relation to intercurrent events

• Information used by imputation or weighting

• Number of imputations and pooling when applicable

• Compatibility of imputation and analysis models

• Uncertainty propagation

• Sensitivity to plausible departures from assumptions

Avoid demanding one preferred technique without considering the estimand and missingness process.

Reproducibility review

Materials and provenance

Check:

• Stable identifiers for datasets, samples, models, protocols, and materials

• Raw-to-processed provenance

• Exclusion and transformation records

• Versioned analysis inputs and outputs

• Repository accession numbers

• Data dictionary, units, and coding

• Domain metadata standard where applicable

Legacy domain standards and their current status are summarized in references/reporting_standards.md.

Code and computational environment

Check:

• Executable code for central analyses when sharing is permitted

• Dependency versions or lock/environment file

• Operating-system or hardware requirements that affect results

• Random seeds and nondeterminism

• Parameter, configuration, and model checkpoints

• Run order and instructions

• Tests or validation of custom code

• License and access restrictions

Code availability does not prove that the code generated the reported result. Provenance and a reproducible run record are separate evidence.

Data and access

Open sharing may be limited by consent, privacy, indigenous/community governance, security, contracts, or licensing. A useful statement should identify:

• What exists

• Where it is held

• Who can request access

• Criteria and process

• Expected timeline

• Restrictions and rationale

• Whether code or synthetic/aggregate alternatives are available

Do not request disclosure that would violate ethics, law, consent, or governance.

Independent reproduction

Claim independent reproduction only if the reviewer actually:

1. Obtained authorized inputs.

2. Recorded versions and environment.

3. Ran documented commands.

4. Preserved content hashes or equivalent provenance.

5. Compared prespecified outputs.

6. Recorded deviations and failures.

A static consistency audit is not reproduction.

When to request specialist review

Escalate when a central conclusion depends on methods outside competence, including:

• Complex adaptive, Bayesian, causal, survival, multilevel, spatial, or longitudinal methods

• High-dimensional omics or multiple-testing procedures

• Diagnostic, prediction, or AI evaluation

• Survey weighting or complex sampling

• Economic modeling

• Meta-analysis with dependent effects or network structure

• Unfamiliar qualitative or mixed-methods methodology

• Image forensics, biosecurity, privacy, or domain-specific ethics

Say what expertise is needed and which claim depends on it. Do not mask uncertainty with an automated score.

Using the local checklist

Copy assets/statistical_reproducibility_template.json, record evidence locations without pasting manuscript prose into report fields, and run:

python3 scripts/audit_statistics_reproducibility.py local-checklist.json

Statuses:

• verified_present

• partly_documented

• missing

• not_assessed

• not_applicable with rationale

The tool reports item IDs and counts. It does not calculate merit, rerun analyses, or certify reproducibility.