Redesign 2025
#17 | ResearchBox

ResearchBox #17 - 'DataColada 76 - Heterogeneity is Replicable'


Bingo Table
  Show file names
  Show file IDs
  Show timestamps


  Colada76.I.Maluma.qualtrics.Survey.pdf



  Colada76.I.Maluma.Qualtrics.Survey.qsf


 csv - Study 1

  


  Data 1 - Colada 76 - 2019 04 26 (masked mturk id).csv


  


  Data 2 - Colada 76 - 2019 04 26 (masked mturk id).csv


  R - Maluma Takiti - Colada 76 - Posted - 2019 04 29.R


  Simulation.r



  p-value heterogeneity ebersole.R



  many labs i2 .r


Previewing files
Files can be previewed by clicking on blue font.
Codebooks can be previewed by clicking on

THIS PUBLIC BOX HAS BEEN EDITED

After this Box became public on 2020-08-18
  • 0 files have been 'deleted' (actually just hidden)
  • 2 files have been added
There are three viewing modes.
view only files in box as originally published (including 'deleted' files).
view all files ever available in this box.
view only current files (default).

  

Tell us if something is wrong with this Box

BOX INFORMATION

SUPPLEMENTARY FILES FOR
Uri Simonsohn ; Joe Simmons, '[76] Heterogeneity Is Replicable: Evidence From Maluma, MTurk, and Many Labs', Data Colada
http://datacolada.org/76

CITING THIS RESEARCHBOX
Simonsohn, U., & Simmons, J. (2025). ResearchBox 17, 'DataColada 76 - Heterogeneity is Replicable', https://ResearchBox.org/17. Zenodo. https://doi.org/10.5281/zenodo.15031613

LICENSE FOR USE
All content posted to ResearchBox is under a CC By 4.0 License (all use is allowed as long as authorship of the content is attributed). When using content from ResearchBox please cite the original work, and provide a link to the URL for this box (https://researchbox.org/17).

BOX PUBLIC SINCE
August 18, 2020   

BOX CREATORS
Uri Simonsohn (urisohn@gmail.com)
Joseph Simmons (jsimmo@upenn.edu)

ABSTRACT
A number of authors have recently proposed that (i) psychological research is highly unpredictable, with identical studies obtaining surprisingly different results, (ii) the presence of heterogeneity decreases the replicability of psychological findings. In this post we provide evidence that contradicts both propositions. Consider these quotes: "heterogeneity persists, and to a reasonable degree, even in […] Many Labs projects […] where rigid, vetted protocols with identical study materials are followed […] heterogeneity […] cannot be avoided in psychological research—even if every effort is taken to eliminate it." McShane, Tackett, Böckenholt, and Gelman (American Statistician 2019 .htm) “Heterogeneity […] makes it unlikely that the typical psychological study can be closely replicated” Stanley, Carter, and Doucouliagos (Psychological Bulletin 2018 .htm) "Repeated investigations of the same phenomenon [get] effect sizes that vary more than one would expect […] even in exact replication studies. […] In the presence of heterogeneity, […] even large N studies may find a result in the opposite direction from the original study. This makes us question the wisdom of placing a great deal of faith in a single replication study" Judd and Kenny (Psychological Methods 2019 .htm) This post is not an evaluation of the totality of these three papers, but rather a specific evaluation of the claims in the quoted text. In the case of the paper by Judd and Kenny, the claims quoted above are secondary, used to motivate the remainder of their paper (e.g., in the opening lines of their abstract).