Scientific Critical Thinking

Core Capabilities

The seven capability areas in full: methodology critique, bias detection, statistical analysis evaluation, evidence quality assessment, logical fallacy identification, research design guidance, and claim evaluation — each with the questions to ask and what the answers imply.

Core Capabilities

1. Methodology Critique

Evaluate research methodology for rigor, validity, and potential flaws.

Apply when:

• Reviewing research papers

• Assessing experimental designs

• Evaluating study protocols

• Planning new research

Evaluation framework:

1. Study Design Assessment

   • Is the design appropriate for the research question?

   • Can the design support causal claims being made?

   • Are comparison groups appropriate and adequate?

   • Consider whether experimental, quasi-experimental, or observational design is justified

2. Validity Analysis

   • Internal validity: Can we trust the causal inference?

     • Check randomization quality

     • Evaluate confounding control

     • Assess selection bias

     • Review attrition/dropout patterns

   • External validity: Do results generalize?

     • Evaluate sample representativeness

     • Consider ecological validity of setting

     • Assess whether conditions match target application

   • Construct validity: Do measures capture intended constructs?

     • Review measurement validation

     • Check operational definitions

     • Assess whether measures are direct or proxy

   • Statistical conclusion validity: Are statistical inferences sound?

     • Verify adequate power/sample size

     • Check assumption compliance

     • Evaluate test appropriateness

3. Control and Blinding

   • Was randomization properly implemented (sequence generation, allocation concealment)?

   • Was blinding feasible and implemented (participants, providers, assessors)?

   • Are control conditions appropriate (placebo, active control, no treatment)?

   • Could performance or detection bias affect results?

4. Measurement Quality

   • Are instruments validated and reliable?

   • Are measures objective when possible, or subjective with acknowledged limitations?

   • Is outcome assessment standardized?

   • Are multiple measures used to triangulate findings?

Reference: See references/scientific_method.md for detailed principles and references/experimental_design.md for comprehensive design checklist.

2. Bias Detection

Identify and evaluate potential sources of bias that could distort findings.

Apply when:

• Reviewing published research

• Designing new studies

• Interpreting conflicting evidence

• Assessing research quality

Systematic bias review:

1. Cognitive Biases (Researcher)

   • Confirmation bias: Are only supporting findings highlighted?

   • HARKing: Were hypotheses stated a priori or formed after seeing results?

   • Publication bias: Are negative results missing from literature?

   • Cherry-picking: Is evidence selectively reported?

   • Check for preregistration and analysis plan transparency

2. Selection Biases

   • Sampling bias: Is sample representative of target population?

   • Volunteer bias: Do participants self-select in systematic ways?

   • Attrition bias: Is dropout differential between groups?

   • Survivorship bias: Are only "survivors" visible in sample?

   • Examine participant flow diagrams and compare baseline characteristics

3. Measurement Biases

   • Observer bias: Could expectations influence observations?

   • Recall bias: Are retrospective reports systematically inaccurate?

   • Social desirability: Are responses biased toward acceptability?

   • Instrument bias: Do measurement tools systematically err?

   • Evaluate blinding, validation, and measurement objectivity

4. Analysis Biases

   • P-hacking: Were multiple analyses conducted until significance emerged?

   • Outcome switching: Were non-significant outcomes replaced with significant ones?

   • Selective reporting: Are all planned analyses reported?

   • Subgroup fishing: Were subgroup analyses conducted without correction?

   • Check for study registration and compare to published outcomes

5. Confounding

   • What variables could affect both exposure and outcome?

   • Were confounders measured and controlled (statistically or by design)?

   • Could unmeasured confounding explain findings?

   • Are there plausible alternative explanations?

Reference: See references/common_biases.md for comprehensive bias taxonomy with detection and mitigation strategies.

3. Statistical Analysis Evaluation

Critically assess statistical methods, interpretation, and reporting.

Apply when:

• Reviewing quantitative research

• Evaluating data-driven claims

• Assessing clinical trial results

• Reviewing meta-analyses

Statistical review checklist:

1. Sample Size and Power

   • Was a priori power analysis conducted?

   • Is sample adequate for detecting meaningful effects?

   • Is the study underpowered (common problem)?

   • Do significant results from small samples raise flags for inflated effect sizes?

2. Statistical Tests

   • Are tests appropriate for data type and distribution?

   • Were test assumptions checked and met?

   • Are parametric tests justified, or should non-parametric alternatives be used?

   • Is the analysis matched to study design (e.g., paired vs. independent)?

3. Multiple Comparisons

   • Were multiple hypotheses tested?

   • Was correction applied (Bonferroni, FDR, other)?

   • Are primary outcomes distinguished from secondary/exploratory?

   • Could findings be false positives from multiple testing?

4. P-Value Interpretation

   • Are p-values interpreted correctly (probability of data if null is true)?

   • Is non-significance incorrectly interpreted as "no effect"?

   • Is statistical significance conflated with practical importance?

   • Are exact p-values reported, or only "p < .05"?

   • Is there suspicious clustering just below .05?

5. Effect Sizes and Confidence Intervals

   • Are effect sizes reported alongside significance?

   • Are confidence intervals provided to show precision?

   • Is the effect size meaningful in practical terms?

   • Are standardized effect sizes interpreted with field-specific context?

6. Missing Data

   • How much data is missing?

   • Is missing data mechanism considered (MCAR, MAR, MNAR)?

   • How is missing data handled (deletion, imputation, maximum likelihood)?

   • Could missing data bias results?

7. Regression and Modeling

   • Is the model overfitted (too many predictors, no cross-validation)?

   • Are predictions made outside the data range (extrapolation)?

   • Are multicollinearity issues addressed?

   • Are model assumptions checked?

8. Common Pitfalls

   • Correlation treated as causation

   • Ignoring regression to the mean

   • Base rate neglect

   • Texas sharpshooter fallacy (pattern finding in noise)

   • Simpson's paradox (confounding by subgroups)

Reference: See references/statistical_pitfalls.md for detailed pitfalls and correct practices.

4. Evidence Quality Assessment

Evaluate the strength and quality of evidence systematically.

Apply when:

• Weighing evidence for decisions

• Conducting literature reviews

• Comparing conflicting findings

• Determining confidence in conclusions

Evidence evaluation framework:

1. Study Design Hierarchy

   • Systematic reviews/meta-analyses (highest for intervention effects)

   • Randomized controlled trials

   • Cohort studies

   • Case-control studies

   • Cross-sectional studies

   • Case series/reports

   • Expert opinion (lowest)

   Important: Higher-level designs aren't always better quality. A well-designed observational study can be stronger than a poorly-conducted RCT.

2. Quality Within Design Type

   • Risk of bias assessment (use appropriate tool: Cochrane RoB 2 for RCTs, ROBINS-I for non-randomized studies, Newcastle-Ottawa, etc.)

   • Methodological rigor

   • Transparency and reporting completeness

   • Conflicts of interest

3. GRADE Considerations (if applicable)

   • Start with design type (RCT = high, observational = low)

   • Downgrade for:

     • Risk of bias

     • Inconsistency across studies

     • Indirectness (wrong population/intervention/outcome)

     • Imprecision (wide confidence intervals, small samples)

     • Publication bias

   • Upgrade for:

     • Large effect sizes

     • Dose-response relationships

     • Confounders would reduce (not increase) effect

4. Convergence of Evidence

   • Stronger when:

     • Multiple independent replications

     • Different research groups and settings

     • Different methodologies converge on same conclusion

     • Mechanistic and empirical evidence align

   • Weaker when:

     • Single study or research group

     • Contradictory findings in literature

     • Publication bias evident

     • No replication attempts

5. Contextual Factors

   • Biological/theoretical plausibility

   • Consistency with established knowledge

   • Temporality (cause precedes effect)

   • Specificity of relationship

   • Strength of association

Reference: See references/evidence_hierarchy.md for detailed hierarchy, GRADE system, and quality assessment tools.

5. Logical Fallacy Identification

Detect and name logical errors in scientific arguments and claims.

Apply when:

• Evaluating scientific claims

• Reviewing discussion/conclusion sections

• Assessing popular science communication

• Identifying flawed reasoning

Common fallacies in science:

1. Causation Fallacies

   • Post hoc ergo propter hoc: "B followed A, so A caused B"

   • Correlation = causation: Confusing association with causality

   • Reverse causation: Mistaking cause for effect

   • Single cause fallacy: Attributing complex outcomes to one factor

2. Generalization Fallacies

   • Hasty generalization: Broad conclusions from small samples

   • Anecdotal fallacy: Personal stories as proof

   • Cherry-picking: Selecting only supporting evidence

   • Ecological fallacy: Group patterns applied to individuals

3. Authority and Source Fallacies

   • Appeal to authority: "Expert said it, so it's true" (without evidence)

   • Ad hominem: Attacking person, not argument

   • Genetic fallacy: Judging by origin, not merits

   • Appeal to nature: "Natural = good/safe"

4. Statistical Fallacies

   • Base rate neglect: Ignoring prior probability

   • Texas sharpshooter: Finding patterns in random data

   • Multiple comparisons: Not correcting for multiple tests

   • Prosecutor's fallacy: Confusing P(E|H) with P(H|E)

5. Structural Fallacies

   • False dichotomy: "Either A or B" when more options exist

   • Moving goalposts: Changing evidence standards after they're met

   • Begging the question: Circular reasoning

   • Straw man: Misrepresenting arguments to attack them

6. Science-Specific Fallacies

   • Galileo gambit: "They laughed at Galileo, so my fringe idea is correct"

   • Argument from ignorance: "Not proven false, so true"

   • Nirvana fallacy: Rejecting imperfect solutions

   • Unfalsifiability: Making untestable claims

When identifying fallacies:

• Name the specific fallacy

• Explain why the reasoning is flawed

• Identify what evidence would be needed for valid inference

• Note that fallacious reasoning doesn't prove the conclusion false—just that this argument doesn't support it

Reference: See references/logical_fallacies.md for comprehensive fallacy catalog with examples and detection strategies.

6. Research Design Guidance

Provide constructive guidance for planning rigorous studies.

Apply when:

• Helping design new experiments

• Planning research projects

• Reviewing research proposals

• Improving study protocols

Design process:

1. Research Question Refinement

   • Ensure question is specific, answerable, and falsifiable

   • Verify it addresses a gap or contradiction in literature

   • Confirm feasibility (resources, ethics, time)

   • Define variables operationally

2. Design Selection

   • Match design to question (causal → experimental; associational → observational)

   • Consider feasibility and ethical constraints

   • Choose between-subjects, within-subjects, or mixed designs

   • Plan factorial designs if testing multiple factors

3. Bias Minimization Strategy

   • Implement randomization when possible

   • Plan blinding at all feasible levels (participants, providers, assessors)

   • Identify and plan to control confounds (randomization, matching, stratification, statistical adjustment)

   • Standardize all procedures

   • Plan to minimize attrition

4. Sample Planning

   • Conduct a priori power analysis (specify expected effect, desired power, alpha)

   • Account for attrition in sample size

   • Define clear inclusion/exclusion criteria

   • Consider recruitment strategy and feasibility

   • Plan for sample representativeness

5. Measurement Strategy

   • Select validated, reliable instruments

   • Use objective measures when possible

   • Plan multiple measures of key constructs (triangulation)

   • Ensure measures are sensitive to expected changes

   • Establish inter-rater reliability procedures

6. Analysis Planning

   • Prespecify all hypotheses and analyses

   • Designate primary outcome clearly

   • Plan statistical tests with assumption checks

   • Specify how missing data will be handled

   • Plan to report effect sizes and confidence intervals

   • Consider multiple comparison corrections

7. Transparency and Rigor

   • Preregister study and analysis plan

   • Use reporting guidelines (CONSORT, STROBE, PRISMA)

   • Plan to report all outcomes, not just significant ones

   • Distinguish confirmatory from exploratory analyses

   • Commit to data/code sharing

Reference: See references/experimental_design.md for comprehensive design checklist covering all stages from question to dissemination.

7. Claim Evaluation

Systematically evaluate scientific claims for validity and support.

Apply when:

• Assessing conclusions in papers

• Evaluating media reports of research

• Reviewing abstract or introduction claims

• Checking if data support conclusions

Claim evaluation process:

1. Identify the Claim

   • What exactly is being claimed?

   • Is it a causal claim, associational claim, or descriptive claim?

   • How strong is the claim (proven, likely, suggested, possible)?

2. Assess the Evidence

   • What evidence is provided?

   • Is evidence direct or indirect?

   • Is evidence sufficient for the strength of claim?

   • Are alternative explanations ruled out?

3. Check Logical Connection

   • Do conclusions follow from the data?

   • Are there logical leaps?

   • Is correlational data used to support causal claims?

   • Are limitations acknowledged?

4. Evaluate Proportionality

   • Is confidence proportional to evidence strength?

   • Are hedging words used appropriately?

   • Are limitations downplayed?

   • Is speculation clearly labeled?

5. Check for Overgeneralization

   • Do claims extend beyond the sample studied?

   • Are population restrictions acknowledged?

   • Is context-dependence recognized?

   • Are caveats about generalization included?

6. Red Flags

   • Causal language from correlational studies

   • "Proves" or absolute certainty

   • Cherry-picked citations

   • Ignoring contradictory evidence

   • Dismissing limitations

   • Extrapolation beyond data

Provide specific feedback:

• Quote the problematic claim

• Explain what evidence would be needed to support it

• Suggest appropriate hedging language if warranted

• Distinguish between data (what was found) and interpretation (what it means)