AutoGineer
Generated banner for Scientific Critical Thinking
Generated logo for Scientific Critical Thinking

Scientific Critical Thinking

K-DenseCommunityCreated by K-Dense Inc.

  • Skill
  • Research and analysis
  • Science and research
  • MIT
  • No issues found
Download ZIP

From the author

Evaluates scientific claims and evidence quality. Applies to experimental design validity, biases and confounders, statistical interpretation, evidence grading frameworks (GRADE, Cochrane Risk of Bias), and teaching critical analysis. Supports evidence appraisal and identifying flaws; formal peer review writing belongs to peer-review.
— the description by K-Dense Inc.

AutoGineer summary

Written by AutoGineer from the skill’s own files, not by the author

Scientific Critical Thinking provides a structured approach to assessing the validity of research claims. It guides users through the appraisal of experimental designs, statistical interpretations, and potential biases using established frameworks like GRADE and Cochrane Risk of Bias. The skill emphasizes constructive, evidence-based critique, helping users distinguish between data and interpretation while maintaining transparency regarding uncertainty.

What it does

  • Evaluates research methodology and experimental design validity.
  • Identifies cognitive, methodological, and statistical biases.
  • Applies formal evidence grading frameworks like GRADE and Cochrane RoB.
  • Provides structured guidance for writing traceable, constructive critiques.
  • Assists in separating supported conclusions from assumptions and recommendations.
  • Offers optional visual aids for bias and quality assessment workflows.

When to use it

  • When reviewing scientific papers or research protocols.
  • When conducting systematic reviews or meta-analyses.
  • When assessing the strength of evidence for a specific scientific claim.
  • When identifying potential flaws in experimental design or statistical analysis.
  • When teaching critical analysis of scientific literature.

Advantages

  • Promotes consistent, objective evaluation standards.
  • Reduces reliance on intuition by using systematic checklists.
  • Encourages constructive feedback that improves research quality.
  • Provides clear, actionable steps for identifying and reporting study limitations.
  • Supports rigorous, evidence-based decision making.

Usage examples

“Review this study's methodology for potential selection bias and confounding.”
The agent will analyze the study design, check for baseline imbalances, evaluate the randomization process, and identify potential threats to internal validity.
“Apply the GRADE framework to the evidence presented in this paper.”
The agent will assess the certainty of the evidence across domains like risk of bias, inconsistency, and imprecision, providing a structured quality rating.
“Create a flowchart illustrating the risk of bias assessment process for a randomized trial.”
The agent will generate a visual diagram outlining the decision steps for evaluating bias in a randomized controlled trial.
“Critique the statistical analysis used in this clinical trial report.”
The agent will examine the power, multiplicity, and model assumptions, highlighting any misuse of p-values or inappropriate effect size interpretations.

Install

  1. Download the skill as a ZIP:

    https://autogineer.com/agent-skills/k-dense-ai/scientific-critical-thinking/download

  2. Unzip it into the cross-client skills folder of your project:

    mkdir -p .agents/skills && unzip scientific-critical-thinking.zip -d .agents/skills

Documentation by K-Dense Inc.

Scientific Critical Thinking

Overview

Critical thinking is a systematic process for evaluating scientific rigor. Assess methodology, experimental design, statistical validity, biases, confounding, and evidence quality using GRADE and Cochrane ROB frameworks. Apply this skill for critical analysis of scientific claims.

When to Use This Skill

This skill should be used when:

  • Evaluating research methodology and experimental design

  • Assessing statistical validity and evidence quality

  • Identifying biases and confounding in studies

  • Reviewing scientific claims and conclusions

  • Conducting systematic reviews or meta-analyses

  • Applying GRADE or Cochrane risk of bias assessments

  • Providing critical analysis of research papers

Appraisal Workflow

  1. Fix the question and unit. Record population, intervention/exposure, comparator, outcome, time point, effect measure, and target setting. Distinguish descriptive, predictive, and causal claims. Identify the independent experimental/sampling unit.

  2. Extract evidence before judging. Locate the numerical result, uncertainty, denominators, protocol/registration, analysis plan, and relevant supplementary material. Keep missing reporting separate from evidence that a procedure was not performed.

  3. Assess design and analysis. Check selection, confounding, measurement, attrition, multiplicity, dependence, and model assumptions. Good fit or optimizer convergence does not establish a uniquely identified parameter or a causal effect.

  4. Choose the right appraisal framework. Reporting completeness, risk of bias, and certainty of a body of evidence answer different questions. Record the exact tool version and the result being assessed; do not turn checklist counts into a quality score.

  5. Synthesize with scope intact. Examine independent replication, overlapping samples, missing evidence, and applicability. Apply GRADE per outcome/comparison when appropriate, with explicit domain reasons, rather than grading a whole paper by its design label.

  6. Write a traceable critique. For each concern give the source location, observation, consequence for the claim, uncertainty, and a feasible remedy. Separate supported conclusions from assumptions and from clinical/policy recommendations.

Current framework versions, primary sources, and verification limits are in references/review_sources.md. The examples in the references are teaching examples, not empirical findings or validated patient-specific advice.

Visual Aids (Optional)

Only add figures when the user explicitly requests a diagram (for example, a GRADE flowchart, bias decision tree, or evidence-quality framework).

When figures help:

  • Critical thinking framework diagrams

  • Bias identification decision trees

  • Evidence quality assessment flowcharts

  • GRADE or risk-of-bias evaluation frameworks

How to create figures:

  • Preferred: Use the scientific-schematics skill for AI-generated diagrams from a natural-language description

  • Alternative: Build figures in your usual tools (draw.io, PowerPoint, matplotlib, etc.)

Run from the repository root, with OPENROUTER_API_KEY set:

python skills/scientific-schematics/scripts/generate_schematic.py "Illustrative GRADE appraisal: define outcome and comparison, assess certainty domains with reasons; keep recommendation decisions separate" -o figures/grade_flowchart.png --doc-type report

This optional command's CLI was checked with --help; paid generation was not exercised for this example. Follow that skill's current dependencies and review every generated label.

Disclosure: AI schematic generation sends your prompt to OpenRouter (a third-party API). Do not include unpublished sensitive details unless that transmission is appropriate for your project.

Core Capabilities

Seven capability areas, each with the questions to ask and what the answers imply, are in references/core_capabilities.md:

  1. Methodology critique — design, controls, confounding, and whether the method can answer the question asked.

  2. Bias detection — selection, measurement, publication, and cognitive biases.

  3. Statistical analysis evaluation — power, multiplicity, p-value misuse, effect sizes.

  4. Evidence quality assessment — study hierarchy, replication, and strength of inference.

  5. Logical fallacy identification — the fallacies that recur in scientific argument.

  6. Research design guidance — how to strengthen a design before data collection.

  7. Claim evaluation — separating what was shown from what is being asserted.

Per-topic detail is in references/scientific_method.md, references/common_biases.md, references/statistical_pitfalls.md, references/evidence_hierarchy.md, references/logical_fallacies.md, and references/experimental_design.md.

Application Guidelines

General Approach

  1. Be Constructive

    • Identify strengths as well as weaknesses

    • Suggest improvements rather than just criticizing

    • Distinguish between fatal flaws and minor limitations

    • Recognize that all research has limitations

  2. Be Specific

    • Point to specific instances (e.g., "Table 2 shows..." or "In the Methods section...")

    • Quote problematic statements

    • Provide concrete examples of issues

    • Reference specific principles or standards violated

  3. Be Proportionate

    • Match criticism severity to issue importance

    • Distinguish between major threats to validity and minor concerns

    • Consider whether issues affect primary conclusions

    • Acknowledge uncertainty in your own assessments

  4. Apply Consistent Standards

    • Use same criteria across all studies

    • Don't apply stricter standards to findings you dislike

    • Acknowledge your own potential biases

    • Base judgments on methodology, not results

  5. Consider Context

    • Acknowledge practical and ethical constraints

    • Consider field-specific norms for effect sizes and methods

    • Recognize exploratory vs. confirmatory contexts

    • Account for resource limitations in evaluating studies

Apply risk-of-bias tools to the right unit

For RoB 2, identify the specific result: outcome, time point, intervention comparison, numerical estimate, and effect of assignment versus adherence. Use the variant for individually randomized, cluster, or crossover trials; record signalling answers and justifications rather than assigning one blanket score to the whole paper. Different outcomes in the same trial can have different bias judgments. See the Cochrane RoB 2 guidance.

For non-randomized intervention effects, state whether using ROBINS-I 2016 or the ROBINS-I V2 November 2025 draft for follow-up/cohort studies. Do not mix their domains or algorithms. Diagnostic accuracy appraisal now uses QUADAS-3 (current tool v1.2), at the accuracy-estimate level. See the tool-specific sources before a formal assessment.

When Providing Critique

Structure feedback as:

  1. Summary: Brief overview of what was evaluated

  2. Strengths: What was done well (important for credibility and learning)

  3. Concerns: Issues organized by severity

    • Critical issues (threaten validity of main conclusions)

    • Important issues (affect interpretation but not fatally)

    • Minor issues (worth noting but don't change conclusions)

  4. Specific Recommendations: Actionable suggestions for improvement

  5. Overall Assessment: Balanced conclusion about evidence quality and what can be concluded

Use precise terminology:

  • Name specific biases, fallacies, and methodological issues

  • Reference established standards and guidelines

  • Cite principles from scientific methodology

  • Use technical terms accurately

When Uncertain

  • Acknowledge uncertainty: "This could be X or Y; additional information needed is Z"

  • Ask clarifying questions: "Was [methodological detail] done? This affects interpretation."

  • Provide conditional assessments: "If X was done, then Y follows; if not, then Z is concern"

  • Note what additional information would resolve uncertainty

Reference Materials

This skill includes comprehensive reference materials that provide detailed frameworks for critical evaluation:

  • references/scientific_method.md - Core principles of scientific methodology, the scientific process, critical evaluation criteria, red flags in scientific claims, causal inference standards, peer review, and open science principles

  • references/common_biases.md - Comprehensive taxonomy of cognitive, experimental, methodological, statistical, and analysis biases with detection and mitigation strategies

  • references/statistical_pitfalls.md - Common statistical errors and misinterpretations including p-value misunderstandings, multiple comparisons problems, sample size issues, effect size mistakes, correlation/causation confusion, regression pitfalls, and meta-analysis issues

  • references/evidence_hierarchy.md - Traditional evidence hierarchy, GRADE system, study quality assessment criteria, domain-specific considerations, evidence synthesis principles, and practical decision frameworks

  • references/logical_fallacies.md - Logical fallacies common in scientific discourse organized by type (causation, generalization, authority, relevance, structure, statistical) with examples and detection strategies

  • references/experimental_design.md - Comprehensive experimental design checklist covering research questions, hypotheses, study design selection, variables, sampling, blinding, randomization, control groups, procedures, measurement, bias minimization, data management, statistical planning, ethical considerations, validity threats, and reporting standards

When to consult references:

  • Load references into context when detailed frameworks are needed

  • Search references for specific topics with your available text-search tool.

  • References provide depth; SKILL.md provides procedural guidance

  • Consult references for comprehensive lists, detailed criteria, and specific examples

Remember

Scientific critical thinking is about:

  • Systematic evaluation using established principles

  • Constructive critique that improves science

  • Proportional confidence to evidence strength

  • Transparency about uncertainty and limitations

  • Consistent application of standards

  • Recognition that all research has limitations

  • Balance between skepticism and openness to evidence

Always distinguish between:

  • Data (what was observed) and interpretation (what it means)

  • Correlation and causation

  • Statistical significance and practical importance

  • Exploratory and confirmatory findings

  • What is known and what is uncertain

  • Evidence against a claim and evidence for the null

Goals of critical thinking:

  1. Identify strengths and weaknesses accurately

  2. Determine what conclusions are supported

  3. Recognize limitations and uncertainties

  4. Suggest improvements for future work

  5. Advance scientific understanding

Citing Scientific Agent Skills

This skill is part of Scientific Agent Skills by K-Dense. If it materially contributed to a manuscript, report, presentation, or code release, add the paper to the references or software section and tell the user you did so:

Kassis, T., Agarwal, V., He, Y., Patel, D., & Brueckner, A. M. (2026). Scientific Agent Skills: A Library of Procedural Knowledge for Research Agents. arXiv:2609.00065. https://doi.org/10.48550/arXiv.2609.00065

Always cite the current version. The DOI and https://arxiv.org/abs/2609.00065 resolve to the latest arXiv version, so never append a version suffix such as v1. When network access is available, fetch https://arxiv.org/abs/2609.00065 (or http://export.arxiv.org/api/query?id_list=2609.00065) before writing the reference and take the author list, year, and version from that record. If the record lists a journal reference or publisher DOI, cite the published version instead.

Files

references/evidence_hierarchy.md

Evidence Hierarchy and Quality Assessment

Traditional Evidence Hierarchy (Medical/Clinical)

This is a teaching shorthand for some intervention questions, not a universal ranking. Choose designs for the question: prevalence, prognosis, diagnostic accuracy, harms, and mechanisms have different evidence needs. A review synthesizes its underlying studies; meta-analysis alone does not increase certainty. Reporting quality, risk of bias, applicability, and certainty must be assessed separately. Current sources are recorded in review_sources.md.

Level 1: Systematic Reviews and Meta-Analyses

Description: Comprehensive synthesis of all available evidence on a question.

Strengths:

  • Combines multiple studies for greater power

  • Reduces impact of single-study anomalies

  • Can identify patterns across studies

  • May quantify a pooled effect if studies and estimands can meaningfully be combined

Weaknesses:

  • Quality depends on included studies ("garbage in, garbage out")

  • Publication bias can distort findings

  • Heterogeneity may make pooling inappropriate

  • Can mask important differences between studies

Critical evaluation:

  • Was search comprehensive (multiple databases, grey literature)?

  • Were inclusion criteria appropriate and prespecified?

  • Was study quality assessed?

  • Was heterogeneity explored?

  • Were missing studies/results sought in registries and protocols? If funnel-plot methods were appropriate, were alternative explanations for asymmetry considered? Fail-safe N is not a reliable assessment of missing-evidence bias.

  • Were appropriate statistical methods used?

Level 2: Randomized Controlled Trials (RCTs)

Description: Experimental studies with random assignment to conditions.

Strengths:

  • Strong design for estimating intervention effects when implemented and analysed appropriately

  • Balances prognostic factors in expectation, not necessarily in each realized sample

  • Minimizes selection bias

  • Enables causal inference

Weaknesses:

  • May not be ethical or feasible

  • Artificial settings may limit generalizability

  • Often short-term with selected populations

  • Expensive and time-consuming

Critical evaluation:

  • Was randomization adequate (sequence generation, allocation concealment)?

  • Was blinding implemented (participants, providers, assessors)?

  • Was sample size adequate (power analysis)?

  • Was intention-to-treat analysis used?

  • Was attrition rate acceptable and balanced?

  • Are results generalizable?

Level 3: Cohort Studies

Description: Observational studies following groups over time.

Types:

  • Prospective: Follow forward from exposure to outcome

  • Retrospective: Look backward at existing data

Strengths:

  • Can study multiple outcomes

  • Establishes temporal sequence

  • Can calculate incidence and relative risk

  • More feasible than RCTs for many questions

Weaknesses:

  • Susceptible to confounding

  • Selection bias possible

  • Attrition can bias results

  • Causal interpretation requires explicit identification assumptions and sensitivity analysis

Critical evaluation:

  • Were cohorts comparable at baseline?

  • Was exposure measured reliably?

  • Was follow-up adequate and complete?

  • Were potential confounders measured and controlled?

  • Was outcome assessment blinded to exposure?

Level 4: Case-Control Studies

Description: Compare people with outcome (cases) to those without (controls), looking back at exposures.

Strengths:

  • Efficient for rare outcomes

  • Relatively quick and inexpensive

  • Can study multiple exposures

  • Useful for generating hypotheses

Weaknesses:

  • Usually cannot estimate absolute incidence from sampled case/control counts alone

  • Susceptible to recall bias

  • Selection of controls is challenging

  • Causal interpretation depends on the sampling design and control of relevant biases

Critical evaluation:

  • Were cases and controls defined clearly?

  • Were controls appropriate (same source population)?

  • Was matching appropriate?

  • How was exposure ascertained (records vs. recall)?

  • Were potential confounders controlled?

  • Could recall bias explain findings?

Level 5: Cross-Sectional Studies

Description: Snapshot observation at single point in time.

Strengths:

  • Quick and inexpensive

  • Can assess prevalence

  • Useful for hypothesis generation

  • Can study multiple outcomes and exposures

Weaknesses:

  • Cannot establish temporal sequence

  • Cannot determine causation

  • Prevalence-incidence bias

  • Survival bias

Critical evaluation:

  • Was sample representative?

  • Were measures validated?

  • Could reverse causation explain findings?

  • Are confounders acknowledged?

Level 6: Case Series and Case Reports

Description: Description of observations in clinical practice.

Strengths:

  • Can identify new diseases or effects

  • Hypothesis-generating

  • Details rare phenomena

  • Quick to report

Weaknesses:

  • No control group

  • Descriptive uncertainty may be estimable; uncontrolled counts do not identify a comparative effect

  • Highly susceptible to bias

  • Cannot establish causation or frequency

Use: Primarily for hypothesis generation and clinical description.

Level 7: Expert Opinion

Description: Statements by recognized authorities.

Strengths:

  • Synthesizes experience

  • Useful when no research available

  • May integrate multiple sources

Weaknesses:

  • Subjective and potentially biased

  • May not reflect current evidence

  • Appeal to authority fallacy risk

  • Individual expertise varies

Use: Lowest level of evidence; should be supported by data when possible.

Nuances and Limitations of Traditional Hierarchy

When Lower-Level Evidence Can Be Strong

  1. Well-designed observational studies with:

    • Large effects that plausible bias cannot explain (size alone does not establish this)

    • Dose-response relationships

    • Consistent findings across contexts

    • Biological plausibility

    • Explicitly justified confounding and selection assumptions, with sensitivity analyses

  2. Multiple converging lines of evidence from different study types

  3. Natural experiments approximating randomization

When Higher-Level Evidence Can Be Weak

  1. Poor-quality RCTs with:

    • Inadequate randomization

    • High attrition

    • No blinding when feasible

    • Design, conduct, or reporting problems associated with sponsor influence

  2. Biased meta-analyses:

    • Publication bias

    • Selective inclusion

    • Inappropriate pooling

    • Poor search strategy

  3. Not addressing the right question:

    • Wrong population

    • Wrong comparison

    • Wrong outcome

    • Too artificial to generalize

Alternative: GRADE System

GRADE assesses certainty in a body of evidence for a specified outcome and comparison. Choose the relevant GRADE application, target population and decision threshold; retain effect estimates and uncertainty alongside domain judgments. The current GRADE Book is being released progressively; consult its updated chapters and the handbook only for sections not yet replaced.

For intervention effects, the traditional approach starts randomized evidence at high and non-randomized evidence at low certainty. When using the ROBINS-I-integrated approach, start at high and assess bias from confounding/selection explicitly, usually rating down substantially. Do not penalize lack of randomization twice. Specify the approach before rating. Consider risk of bias, inconsistency, indirectness, imprecision, and missing-evidence bias, and justify any large-effect, dose-response, or opposing-confounding considerations.

These levels are judgments about certainty, not numerical probabilities or recommendation strength:

High Certainty

Definition: High confidence that the effect is on the specified side of a threshold or within the target range.

Characteristics:

  • No important unresolved concerns across the applicable certainty domains

  • Neither an RCT label nor a large effect alone establishes high certainty

Moderate Certainty

Definition: Moderate confidence in that target; the effect could be in another range.

Downgrades from high:

  • Some risk of bias

  • Inconsistency across studies

  • Indirectness (different populations/interventions)

  • Imprecision relative to the specified threshold/range of important effects

  • Publication bias suspected

Low Certainty

Definition: Limited confidence in that target; the effect may be in a different range.

Downgrades:

  • Serious limitations in above factors

  • Observational studies without special strengths

Very Low Certainty

Definition: Very little confidence in that target; substantial uncertainty remains about the effect's range.

Characteristics:

  • Very serious limitations

  • Evidence may be too sparse or indirect to support an effect estimate at all; do not invent a numerical estimate or GRADE rating for an unsupported opinion

  • Multiple serious flaws

Document threshold magnitudes and their rationale. For intervention decisions use absolute effects with an appropriate baseline risk and its uncertainty. Rate down/up only when a concern changes confidence in the target range; do not double-count the same underlying problem across domains. Have independent assessors reconcile formal judgments.

Study Quality Assessment Criteria

Internal Validity (Bias Control)

Questions:

  • Was randomization adequate?

  • Was allocation concealed?

  • Were groups similar at baseline?

  • Was blinding implemented?

  • Could missingness depend on the unobserved outcome? Low or balanced attrition alone does not establish low risk of bias.

  • Was intention-to-treat used?

  • Were all outcomes reported?

External Validity (Generalizability)

Questions:

  • Is sample representative of target population?

  • Are inclusion/exclusion criteria too restrictive?

  • Is setting realistic?

  • Are results applicable to other populations?

  • Are effects consistent across subgroups?

Statistical Conclusion Validity

Questions:

  • Was sample size adequate (power)?

  • Were statistical tests appropriate?

  • Were assumptions checked?

  • Were effect sizes and confidence intervals reported?

  • Were multiple comparisons addressed?

  • Was analysis prespecified?

Construct Validity (Measurement)

Questions:

  • Were measures validated and reliable?

  • Was outcome defined clearly and appropriately?

  • Were assessors blinded?

  • Were exposures measured accurately?

  • Was timing of measurement appropriate?

Critical Appraisal Tools

For Different Study Types

RCTs:

  • Cochrane RoB 2 — recommended for randomized trials in Cochrane intervention reviews; assess a result, use the correct parallel/cluster/crossover variant, and retain reasons

  • PEDro Scale — used for physiotherapy trials; its score is not interchangeable with RoB 2

  • Legacy RoB 1 and Jadad ratings in older reviews should be identified as such

Observational Studies:

  • ROBINS-I applies to non-randomized intervention effects, not every observational question. Specify the 2016 version or November 2025 ROBINS-I V2 draft for follow-up/cohort studies; do not combine version-specific signalling questions or domains.

  • Newcastle-Ottawa Scale has cohort and case-control forms; record the chosen form and item judgments, not an unexplained universal good/poor star cutoff.

Diagnostic Studies:

  • QUADAS-3 (current tool v1.2, published February 2026) assesses bias and applicability for individual accuracy estimates against defined synthesis questions and an ideal test accuracy trial. QUADAS-2 is the previous version.

Systematic Reviews:

  • AMSTAR 2 appraises systematic reviews of healthcare interventions; assess critical weaknesses rather than calculating a total score.

  • PRISMA 2020 assesses reporting completeness; it is not a risk-of-bias instrument.

Design-specific general appraisal:

  • CASP supplies separate checklists for supported study designs; select the matching one. Do not assume every design has the same domains or numerical score.

Domain-Specific Considerations

Basic Science Research

Consider together, without imposing a fixed ranking:

  1. Multiple convergent lines of evidence

  2. Mechanistic understanding

  3. Reproducible experiments

  4. Established theoretical framework

Key considerations:

  • Replication essential

  • Mechanistic plausibility

  • Consistency across model systems

  • Convergence of methods

Psychological Research

Additional concerns:

  • Replication crisis

  • Publication bias particularly problematic

  • Small effect sizes often expected

  • Cultural context matters

  • Measures often indirect (self-report)

Strong evidence includes:

  • Preregistered studies

  • Large samples

  • Multiple measures

  • Behavioral (not just self-report) outcomes

  • Cross-cultural replication

Epidemiology

Causal inference frameworks:

  • Bradford Hill criteria

  • Rothman's causal pies

  • Directed Acyclic Graphs (DAGs)

Strong observational evidence:

  • Dose-response relationships

  • Temporal consistency

  • Biological plausibility

  • Specificity can inform interpretation but is not necessary for causation

  • Consistency across populations

  • Large effects assessed against plausible confounding and selection mechanisms

Social Sciences

Challenges:

  • Complex interventions

  • Context-dependent effects

  • Measurement challenges

  • Ethical constraints on RCTs

Strengthening evidence:

  • Mixed methods

  • Natural experiments

  • Instrumental variables

  • Regression discontinuity designs

  • Multiple operationalizations

Synthesizing Evidence Across Studies

Consistency

Strong evidence:

  • Multiple studies, different investigators

  • Different populations and settings

  • Different research designs converge

  • Different measurement methods

Weak evidence:

  • Single study

  • Only one research group

  • Conflicting results

  • Publication bias evident

Biological/Theoretical Plausibility

Strengthens evidence:

  • Known mechanism

  • Consistent with other knowledge

  • Dose-response relationship

  • Coherent with animal/in vitro data

Weakens evidence:

  • No plausible mechanism

  • Contradicts established knowledge

  • Biological implausibility

Temporality

Essential for causation:

  • Cause must precede effect

  • Cross-sectional studies cannot establish

  • Reverse causation must be ruled out

Specificity

Moderate indicator:

  • Specific cause → specific effect strengthens causation

  • But lack of specificity doesn't rule out causation

  • Most causes have multiple effects

Strength of Association

Strong evidence:

  • Large effects with plausible confounding, selection, and measurement explanations examined

  • Dose-response relationships

  • All-or-none effects

Caution:

  • Small effects may still be real

  • Large effects can still be confounded

Red Flags in Evidence Quality

Study Design Red Flags

  • No control group

  • Self-selected participants

  • No randomization when feasible

  • No blinding when feasible

  • Very small sample

  • Inappropriate statistical tests

Reporting Red Flags

  • Selective outcome reporting

  • No study registration/protocol

  • Missing methodological details

  • No conflicts of interest statement

  • Cherry-picked citations

  • Results don't match methods

Interpretation Red Flags

  • Causal language from correlational data

  • Claiming "proof"

  • Ignoring limitations

  • Overgeneralizing

  • Spinning negative results

  • Post hoc rationalization

Context Red Flags

  • Industry funding without independence

  • Single study in isolation

  • Contradicts preponderance of evidence

  • No replication

  • Published in predatory journal

  • Press release before peer review

Practical Decision Framework

When Evaluating Evidence, Ask:

  1. What type of study is this? (Design)

  2. How well was it conducted? (Quality)

  3. What does it actually show? (Results)

  4. How likely is bias? (Internal validity)

  5. Does it apply to my question? (External validity)

  6. How does it fit with other evidence? (Context)

  7. Are the conclusions justified? (Interpretation)

  8. What are the limitations? (Uncertainty)

Making Decisions with Imperfect Evidence

Certainty does not itself authorize changing practice or policy. A high-certainty estimate may show negligible benefit or important harm. Low-certainty evidence may still inform a decision when delay or alternatives carry costs. A separate evidence-to-decision process considers absolute benefits/harms, values, resources, equity, acceptability, and feasibility. State whose decision and setting are involved; an appraisal is not an individual clinical recommendation. Identify the information most likely to change the decision.

When Evidence is Conflicting

Strategies:

  1. Weight by study quality

  2. Look for systematic differences (population, methods)

  3. Consider publication bias

  4. Update with most recent, rigorous evidence

  5. Conduct/await systematic review

  6. Consider if question is well-formed

Communicating Evidence Strength

Avoid:

  • Absolute certainty ("proves")

  • False balance (equal weight to unequal evidence)

  • Ignoring uncertainty

  • Cherry-picking studies

Better:

  • Quantify uncertainty

  • Describe strength of evidence

  • Acknowledge limitations

  • Present range of evidence

  • Distinguish established from emerging findings

  • Be clear about what is/isn't known

Requirements and compatibility

Works with: Claude Code, Claude apps, OpenAI Codex, GitHub Copilot, Cursor, Gemini CLI, Kiro, OpenCode, Goose, Amp, Roo Code, JetBrains Junie, Factory Droid, Devin (ex-Windsurf), Mistral Vibe, OpenHands, Letta

“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.

Needs: Declares allowed tools, Needs network access, Needs an API key, Needs Python

The author says: Analytical guidance needs no network. Optional figures via the scientific-schematics skill require OPENROUTER_API_KEY and outbound API access to OpenRouter.

Security

No issues found by our static scan· No issues found

A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.

Attribution and licence

Created by
K-Dense Inc.
Publisher
K-Dense (Community)
Source
K-Dense-AI/scientific-agent-skills · skills/scientific-critical-thinking · commit a25cf9e
Licence
MIT License (redistribution allowed)
  • LICENSE (added by AutoGineer from LICENSE.md)
  • ATTRIBUTION.md (written by AutoGineer)
Synced

AutoGineer is not affiliated with or endorsed by K-Dense Inc. or K-Dense.

Download ZIP

The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download scientific-critical-thinking.zip