Neil Lutsky, Department Psychology and the QUIRK Initiative 

I. What do the numbers show?

• What do the numbers mean?
• Where are the numbers?
  • Is there numerical evidence to support a claim?
  • What were the exact figures?
  • How can seeking and analyzing numbers illuminate important phenomena?
• How plausible is a possibility in light of back of the envelope calculations?

II. How representative is that?

• What's the central tendency?
  • "For instance" is not proof; it is an example
  • Mean, Mode, and Median
  • Variance
• Interrogating statistics
  • Are there extreme scores (outliers)?
  • Are there meaningful subgroups (clusters)?
  • What is being measured?
• What are the odds (likelihood) of occurrence?
• What is the base rate?

III. Compared to what?

• What is the implicit or explicit frame of reference?
• What is the unit of measurement?
• Per what?
• What is the order of magnitude?
• Interrogating a graph:
  • What's the Y-axis? Is it zero-based?
  • Does it K.I.S.S. or is it filled with ChartJunk?

IV. Is the outcome statistically significant?

• Is the outcome unlikely to have come about by chance?
  • "Chance is lumpy."
  • Criterion of sufficient rarity due to chance: p < .05
• What does statistical significance mean, and what doesn't it mean?

V. What's the effect size?

• How can we take the measure of how substantial an outcome is?
• How large is the mean difference? How large is the association?
• Standardized mean difference (d): d = (μ1-μ2)/σ

VI. Are the results those of a single study or of a literature?

• What's the source of the numbers: PFA, peer-reviewed, or what?
• Who is sponsoring the research?
• How can we take the measure of what a literature shows?
• The importance of meta-analysis in the contemporary world of QR.

VII. What's the research design (correlational or experimental)?

• Design matters: Experimental vs. correlational design.
• How well does the design support a causal claim?
• Experimental Design:
  • Randomized Controlled Trials (RCT): Research trials in which participants are randomly assigned to the conditions of the study.
  • Double blind trials: RCTs in which neither the researcher nor the patient know the treatment condition.
• Correlational Design: Measuring existing variation and evaluating co-occurrences, possibly controlling for other variables.
  • Interrogating associations (correlations):
    • Are there extreme pairs of scores (outliers)?
    • Are there meaningful subgroups?
    • Is the range of scores in a variable restricted?
    • Is the relationship non-linear?

VIII. How was the variable operationalized?

• What meaning and degree of precision does the measurement procedure justify?
• What elements and procedures result in the assignment of a score to a variable?
• What exactly was asked?
• What's the scale of measurement?
• How might we know if the measurement procedure is a good one?
  • Reliability = Repeated applications of the procedure result in consistent scores.
  • Validity = Evidence supports the use to which the measure is being put.
• Is the measure being manipulated or "gamed"? The iatrogenic effects of measurement.

IX. Who's in the measurement sample?

• What domain is being evaluated? Who's in? Who's not?
• Is the sample from that domain representative, meaningful, and/or sufficient?
• Is the sample random?
• Are two or more samples that are being compared equivalent?

X. Controlling for what?

• What other variables might be influencing the findings?
• Were these assessed or otherwise controlled for in the research design?
• What don't we know, and how can we acknowledge uncertainties?