Inferential statistics as descriptive statistics: there is no replication crisis if we don't expect replication

Amrhein, Valentin and Trafimow, David and Greenland, Sander. (2019) Inferential statistics as descriptive statistics: there is no replication crisis if we don't expect replication. The American Statistician, 73 (S1). pp. 262-270.

[img] PDF - Published Version
Available under License CC BY-NC-ND (Attribution-NonCommercial-NoDerivatives).


Official URL: https://edoc.unibas.ch/71340/

Downloads: Statistics Overview


Statistical inference often fails to replicate. One reason is that many results may be selected for drawing inference because some threshold of a statistic like the P-value was crossed, leading to biased reported effect sizes. Nonetheless, considerable non-replication is to be expected even without selective reporting, and generalizations from single studies are rarely if ever warranted. Honestly reported results must vary from replication to replication because of varying assumption violations and random variation; excessive agreement itself would suggest deeper problems, such as failure to publish results in conflict with group expectations or desires. A general perception of a "replication crisis" may thus reflect failure to recognize that statistical tests not only test hypotheses, but countless assumptions and the entire environment in which research takes place. Because of all the uncertain and unknown assumptions that underpin statistical inferences, we should treat inferential statistics as highly unstable local descriptions of relations between assumptions and data, rather than as providing generalizable inferences about hypotheses or models. And that means we should treat statistical results as being much more incomplete and uncertain than is currently the norm. Acknowledging this uncertainty could help reduce the allure of selective reporting: Since a small P-value could be large in a replication study, and a large P-value could be small, there is simply no need to selectively report studies based on statistical results. Rather than focusing our study reports on uncertain conclusions, we should thus focus on describing accurately how the study was conducted, what problems occurred, what data were obtained, what analysis methods were used and why, and what output those methods produced.
Faculties and Departments:05 Faculty of Science > Departement Umweltwissenschaften > Integrative Biologie > Behavioural Ecology (Amrhein)
UniBasel Contributors:Amrhein, Valentin
Item Type:Article, refereed
Article Subtype:Research Article
Publisher:Taylor & Francis
Note:Publication type according to Uni Basel Research Database: Journal article
Identification Number:
edoc DOI:
Last Modified:06 Jun 2023 13:51
Deposited On:19 Jul 2019 10:04

Repository Staff Only: item control page