Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Ebersole, Charles R. and Mathur, Maya B. and Baranski, Erica and Bart-Plange, Diane-Jo and Buttrick, Nicholas R. and Chartier, Christopher R. and Corker, Katherine S. and Corley, Martin and Hartshorne, Joshua K. and IJzerman, Hans and Lazarević, Ljiljana B. and Rabagliati, Hugh and Ropovik, Ivan and Aczel, Balazs and Aeschbach, Lena F. and Andrighetto, Luca and Arnal, Jack D. and Arrow, Holly and Babincak, Peter and Bakos, Bence E. and Baník, Gabriel and Baskin, Ernest and Belopavlović, Radomir and Bernstein, Michael H. and Białek, Michał and Bloxsom, Nicholas G. and Bodroža, Bojana and Bonfiglio, Diane B. V. and Boucher, Leanne and Brühlmann, Florian and Brumbaugh, Claudia C. and Casini, Erica and Chen, Yiling and Chiorri, Carlo and Chopik, William J. and Christ, Oliver and Ciunci, Antonia M. and Claypool, Heather M. and Coary, Sean and Čolić, Marija V. and Collins, W. Matthew and Curran, Paul G. and Day, Chris R. and Dering, Benjamin and Dreber, Anna and Edlund, John E. and Falcão, Filipe and Fedor, Anna and Feinberg, Lily and Ferguson, Ian R. and Ford, Máire and Frank, Michael C. and Fryberger, Emily and Garinther, Alexander and Gawryluk, Katarzyna and Ashbaugh, Kayla and Giacomantonio, Mauro and Giessner, Steffen R. and Grahe, Jon E. and Guadagno, Rosanna E. and Hałasa, Ewa and Hancock, Peter J. B. and Hilliard, Rias A. and Hüffmeier, Joachim and Hughes, Sean and Idzikowska, Katarzyna and Inzlicht, Michael and Jern, Alan and Jiménez-Leal, William and Johannesson, Magnus and Joy-Gaba, Jennifer A. and Kauff, Mathias and Kellier, Danielle J. and Kessinger, Grecia and Kidwell, Mallory C. and Kimbrough, Amanda M. and King, Josiah P. J. and Kolb, Vanessa S. and Kołodziej, Sabina and Kovacs, Marton and Krasuska, Karolina and Kraus, Sue and Krueger, Lacy E. and Kuchno, Katarzyna and Lage, Caio Ambrosio and Langford, Eleanor V. and Levitan, Carmel A. and de Lima, Tiago Jessé Souza and Lin, Hause and Lins, Samuel and Loy, Jia E. and Manfredi, Dylan and Markiewicz, Łukasz and Menon, Madhavi and Mercier, Brett and Metzger, Mitchell and Meyet, Venus and Millen, Ailsa E. and Miller, Jeremy K. and Montealegre, Andres and Moore, Don A. and Muda, Rafał and Nave, Gideon and Nichols, Austin Lee and Novak, Sarah A. and Nunnally, Christian and Orlić, Ana and Palinkas, Anna and Panno, Angelo and Parks, Kimberly P. and Pedović, Ivana and Pękala, Emilian and Penner, Matthew R. and Pessers, Sebastiaan and Petrović, Boban and Pfeiffer, Thomas and Pieńkosz, Damian and Preti, Emanuele and Purić, Danka and Ramos, Tiago and Ravid, Jonathan and Razza, Timothy S. and Rentzsch, Katrin and Richetin, Juliette and Rife, Sean C. and Rosa, Anna Dalla and Rudy, Kaylis Hase and Salamon, Janos and Saunders, Blair and Sawicki, Przemysław and Schmidt, Kathleen and Schuepfer, Kurt and Schultze, Thomas and Schulz-Hardt, Stefan and Schütz, Astrid and Shabazian, Ani N. and Shubella, Rachel L. and Siegel, Adam and Silva, Rúben and Sioma, Barbara and Skorb, Lauren and de Souza, Luana Elayne Cunha and Steegen, Sara and Stein, L. A. R. and Sternglanz, R. Weylin and Stojilović, Darko and Storage, Daniel and Sullivan, Gavin Brent and Szaszi, Barnabas and Szecsi, Peter and Szöke, Orsolya and Szuts, Attila and Thomae, Manuela and Tidwell, Natasha D. and Tocco, Carly and Torka, Ann-Kathrin and Tuerlinckx, Francis and Vanpaemel, Wolf and Vaughn, Leigh Ann and Vianello, Michelangelo and Viganola, Domenico and Vlachou, Maria and Walker, Ryan J. and Weissgerber, Sophia C. and Wichman, Aaron L. and Wiggins, Bradford J. and Wolf, Daniel and Wood, Michael J. and Zealley, David and Žeželj, Iris and Zrubka, Mark and Nosek, Brian A.. (2020) Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability. Advances in Methods and Practices in Psychological Science, 3 (3). pp. 309-331.

Full text not available from this repository.

Official URL: https://edoc.unibas.ch/84955/

Downloads: Statistics Overview


Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3âEuro"9; median total sample = 1,279.5, range = 276âEuro"3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Î"r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols (r = .05) was similar to that of the RP:P protocols (r = .04) and the original RP:P replications (r = .11), and smaller than that of the original studies (r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00âEuro".15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19âEuro".50).
Faculties and Departments:07 Faculty of Psychology > Departement Psychologie > Society & Choice > Allgemeine Psychologie und Methodologie (Opwis)
UniBasel Contributors:Brühlmann, Florian and Aeschbach, Lena F
Item Type:Article, refereed
Article Subtype:Research Article
Note:Publication type according to Uni Basel Research Database: Journal article
Identification Number:
Last Modified:18 Jan 2022 15:42
Deposited On:01 Nov 2021 17:00

Repository Staff Only: item control page