@article{8637133,
  abstract     = {{We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen's ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.}},
  author       = {{Klein, Richard A. and Vianello, Michelangelo and Hasselman, Fred and Adams, Byron G. and Adams, Reginald B. and Alper, Sinan and Aveyard, Mark and Axt, Jordan R. and Babalola, Mayowa T. and Bahník, Štěpán and Batra, Rishtee and Berkics, Mihály and Bernstein, Michael J. and Berry, Daniel R. and Bialobrzeska, Olga and Binan, Evans Dami and Bocian, Konrad and Brandt, Mark J. and Busching, Robert and Rédei, Anna Cabak and Cai, Huajian and Cambier, Fanny and Cantarero, Katarzyna and Carmichael, Cheryl L. and Ceric, Francisco and Chandler, Jesse and Chang, Jen-Ho and Chatard, Armand and Chen, Eva E. and Cheong, Winnee and Cicero, David C. and Coen, Sharon and Coleman, Jennifer A. and Collisson, Brian and Conway, Morgan A. and Corker, Katherine S. and Curran, Paul G. and Cushman, Fiery and Dagona, Zubairu K. and Dalgar, Ilker and Dalla Rosa, Anna and Davis, William E. and de Bruijn, Maaike and De Schutter, Leander and Devos, Thierry and de Vries, Marieke and Doğulu, Canay and Dozo, Nerisa and Dukes, Kristin Nicole and Dunham, Yarrow and Durrheim, Kevin and Ebersole, Charles R. and Edlund, John E. and Eller, Anja and English, Alexander Scott and Finck, Carolyn and Frankowska, Natalia and Freyre, Miguel-Ángel and Friedman, Mike and Galliani, Elisa Maria and Gandi, Joshua C. and Ghoshal, Tanuka and Giessner, Steffen R. and Gill, Tripat and Gnambs, Timo and Gómez, Ángel and González, Roberto and Graham, Jesse and Grahe, Jon E. and Grahek, Ivan and Green, Eva G. T. and Hai, Kakul and Haigh, Matthew and Haines, Elizabeth L. and Hall, Michael P. and Heffernan, Marie E. and Hicks, Joshua A. and Houdek, Petr and Huntsinger, Jeffrey R. and Huynh, Ho Phi and IJzerman, Hans and Inbar, Yoel and Innes-Ker, Åse H. and Jiménez-Leal, William and John, Melissa-Sue and Joy-Gaba, Jennifer A. and Kamiloğlu, Roza G. and Kappes, Heather Barry and Karabati, Serdar and Karick, Haruna and Keller, Victor N. and Kende, Anna and Kervyn, Nicolas and Knežević, Goran and Kovacs, Carrie and Krueger, Lacy E. and Kurapov, German and Kurtz, Jamie and Lakens, Daniël and Lazarević, Ljiljana B. and Levitan, Carmel A. and Lewis, Neil A. and Lins, Samuel and Lipsey, Nikolette P. and Losee, Joy E. and Maassen, Esther and Maitner, Angela T. and Malingumu, Winfrida and Mallett, Robyn K. and Marotta, Satia A. and Međedović, Janko and Mena-Pacheco, Fernando and Milfont, Taciano L. and Morris, Wendy L. and Murphy, Sean C. and Myachykov, Andriy and Neave, Nick and Neijenhuijs, Koen and Nelson, Anthony J. and Neto, Félix and Lee Nichols, Austin and Ocampo, Aaron and O’Donnell, Susan L. and Oikawa, Haruka and Oikawa, Masanori and Ong, Elsie and Orosz, Gábor and Osowiecka, Malgorzata and Packard, Grant and Pérez-Sánchez, Rolando and Petrović, Boban and Pilati, Ronaldo and Pinter, Brad and Podesta, Lysandra and Pogge, Gabrielle and Pollmann, Monique M. H. and Rutchick, Abraham M. and Saavedra, Patricio and Saeri, Alexander K. and Salomon, Erika and Schmidt, Kathleen and Schönbrodt, Felix D. and Sekerdej, Maciej B. and Sirlopú, David and Skorinko, Jeanine L. M. and Smith, Michael A. and Smith-Castro, Vanessa and Smolders, Karin C. H. J. and Sobkow, Agata and Sowden, Walter and Spachtholz, Philipp and Srivastava, Manini and Steiner, Troy G. and Stouten, Jeroen and Street, Chris N. H. and Sundfelt, Oskar K. and Szeto, Stephanie and Szumowska, Ewa and Tang, Andrew C. W. and Tanzer, Norbert and Tear, Morgan J. and Theriault, Jordan and Thomae, Manuela and Torres, David and Traczyk, Jakub and Tybur, Joshua M. and Ujhelyi, Adrienn and van Aert, Robbie C. M. and van Assen, Marcel A. L. M. and van der Hulst, Marije and van Lange, Paul A. M. and van ’t Veer, Anna Elisabeth and Vásquez- Echeverría, Alejandro and Ann Vaughn, Leigh and Vázquez, Alexandra and Vega, Luis Diego and Verniers, Catherine and Verschoor, Mark and Voermans, Ingrid P. J. and Vranka, Marek A. and Welch, Cheryl and Wichman, Aaron L. and Williams, Lisa A. and Wood, Michael and Woodzicka, Julie A. and Wronska, Marta K. and Young, Liane and Zelenski, John M. and Zhijia, Zeng and Nosek, Brian A.}},
  issn         = {{2515-2459}},
  journal      = {{ADVANCES IN METHODS AND PRACTICES IN PSYCHOLOGICAL SCIENCE}},
  language     = {{eng}},
  number       = {{4}},
  pages        = {{443--490}},
  title        = {{Many labs 2 : investigating variation in replicability across samples and settings}},
  url          = {{http://doi.org/10.1177/2515245918810225}},
  volume       = {{1}},
  year         = {{2018}},
}

Altmetric

View in Altmetric

Web of Science

Times cited: