In our previous post, we reported data anomalies in the Study 2 spreadsheet. In this post, we present analyses of the Study 1 spreadsheet. The data are very different, the forensic analyses are very different, but our conclusions are very similar.<\/span><\/p>\n As we indicated in our previous post, to the best of our knowledge, Klaus Wertenbroch never had access to any version of the data for any of the studies. And, we believe it is thanks to him that we do. When Kyle Hyndman reached out to the authors back in 2006, Klaus replied with this email (Klaus gave us permission to include this email in our posts):<\/span><\/p>\n Our ResearchBox<\/a> contains the data and code to reproduce all of the results in both posts.<\/span><\/p>\n Study 1 The authors\u2019 key hypothesis was that evenly spaced deadlines help counter procrastination. In the extreme, if everything is due on the last day, people wait too long to do everything, they have insufficient time to do all three papers at once, and their performance suffers. That\u2019s the idea.<\/span><\/p>\n The authors present three key findings:<\/span><\/p>\n Finding #1<\/strong>: Students who set their own deadlines did not always evenly space them.<\/span><\/p>\n Finding #2<\/strong>: Grades in the Set Your Own Deadlines section were significantly lower than grades in the Evenly Spaced Deadlines section.<\/span><\/p>\n Finding #3<\/strong>: Within the Set Your Own Deadlines section, students who chose to spread out their deadlines performed better than those who did not. Indeed, the grade gap between those students and the Evenly Spaced Deadlines section was reportedly \u201creduced by 59%\u201d (p. 222) [1<\/a>].<\/span><\/p>\n Finding #3 is especially critical to this post.<\/span><\/p>\n The Original Data Columns B-D show us the deadlines that participants set for themselves in the Set your Own Deadline condition, measured in days until the last class. For instance, “0, 0, 0” means that they made all papers due on the last day, and \u201c12, 12, 12\u201d means they made all papers due 12 days before the last day of class. \u00a0<\/span><\/p>\n Column F is of critical importance to this post. This variable, which was included in the original spreadsheet, indicates whether the deadlines were spread out over different days (\u201cyes\u201d) or whether they were all set on the same day (\u201cno\u201d). In the Evenly Spaced Deadlines section, all deadlines were forcibly spaced by the professor, and so all rows are a \u201cyes.\u201d In the Set Your Own Deadlines section, some are “yes” and some are “no” (This variable is incorrectly described in the paper; see footnote [2<\/a>]).<\/span><\/p>\n The next five columns, G-K, are the component grades on which the final grade, Column L, is based. Those graded components include (1) the three short papers, (2) whatever \u201cB-Points\u201d were, (3) participation, (4) a final project, and (5) the exam.<\/span><\/p>\n Using that \u201cSpreadsheet Final Grade\u201d column (L), we successfully reproduce all the Study 1 results reported in the published paper, confirming to our satisfaction that we have the original data. See Appendix 1 (.pdf<\/a>).<\/span><\/p>\n How Was \u201cFinal Grade\u201d Computed? We expected this to be straightforward, but simple intuition and trial-and-error wasn\u2019t giving us the answer. And so we resorted to an old-school AI method: linear regression. We set the dataset\u2019s final grade as the dependent variable and the five component grades as the predictors. The coefficients tell us the average effect of each component on the final grade. Because final grades should be perfectly<\/u> <\/em>predicted by the grade components, we should get an R2<\/sup> of 100%. Instead we got an R2<\/sup> of 56.5%:<\/span><\/p>\n Huh.<\/span><\/p>\n So only about half the variance in final grades comes from the five grade components. Where does the other half come from? There are some benign possibilities. For example, perhaps the instructor exercised discretion when setting final grades, discretion not documented in the spreadsheet. Or perhaps final grade was computed with a non<\/em>linear combination of component grades. Or perhaps the data file got partially scrambled somehow (an innocent error that has twice happened to one of the authors of this post). By itself, an R2 <\/sup>below 100% is a red flag, but it does not necessarily imply anything sinister.<\/span><\/p>\n An additional possibility, also potentially benign (though odd), is that perhaps the two sections used different rules<\/em> to compute final grades (e.g., maybe final project mattered 35% in one section and only 25% in the other). That is easy to test, as you just need to run the same regression separately for each section. So we did that, thinking that maybe now we\u2019d see some perfect 100% R2<\/sup>s. Well, we did, but only for one of the two sections.<\/span><\/p>\n In the Evenly Spaced Deadlines section, we see this:<\/span><\/p>\n These results make perfect sense. Four of the coefficients are .2, meaning that those four components were equally weighted. The fifth component, \u201cparticipation\u201d, had a weight of 0, implying that though this element was recorded, it did not contribute at all to the final grade. Furthermore, because the intercept is 20, and all grades are on a 0 to 100 scale, all students were graded as if<\/em> they had 100% in participation. So, in this section, the final grade is the simple average of the five individual component grades, once you give everyone a 100% in participation. This is simple and sensible.<\/span><\/p>\n Results in the Set Your Own Deadlines section are less simple. In fact, they are a bit of a nightmare:<\/span><\/p>\n Not only is the R2 <\/sup>way lower than 100%, but three of the coefficients are negative<\/em>. The higher the student’s exam grade, the lower<\/em> their final grade? As folks on the internet might say, WTAF?<\/span><\/p>\n We think the answer to that question is: data fraud.<\/span><\/p>\n Evidence of Data Alterations in the Set Your Own Deadlines Condition So we now have for each student a “spreadsheet final grade” and a “re-computed final grade”. We calculated the difference between them and plotted it. We had absolutely no idea what that would look like. It looked like this:<\/span><\/p>\n This figure tells us two important things.<\/span><\/p>\n First, for most students (36 out of 49), the spreadsheet final grade in the Set Your Own Deadlines section was exactly the same as the re-computed final grade (i.e., the difference score was zero). This strongly suggests that the same simple grading rule was originally used in both<\/em> sections.<\/span><\/p>\n Second, for the remaining 13 students, all deviations from this simple grading rule were integers (i.e., exactly -20, -10, -6, +4, and +10). This suggests that final grades weren\u2019t different because of how the component grades were weighted. It suggests that the final grades were edited manually.<\/span><\/p>\n Evidence That The Final Grade Alterations Were Intentional Finding #3, remember, was that students who chose to spread out their deadlines performed better than those who didn’t, and similarly to those in the Evenly Spaced Deadlines section.<\/span><\/p>\n To understand the evidence we are about to show, let’s imagine that Finding #3 were not supported in the original data: that students who spread their deadlines did just as well as, or worse than, students who did not spread their deadlines. And imagine you were tasked with tampering with the data to produce Finding #3. What would you have to do? You would need to increase the grades of students who did spread out their deadlines and\/or decrease the grades of those who did not. Figure 6 shows that that’s exactly how the grades were altered:<\/span><\/p>\n Fig 6.<\/strong> Re-computed vs. Spreadsheet Final Grades<\/em><\/span><\/p>\n The x-axis shows the \u201cre-computed final grade\u201d \u2013 the simple average of the five component grades \u2013 and the y-axis shows the “spreadsheet final grade\u201d. If the spreadsheet grade was computed using the simple average rule, the data would fall on the diagonal (45-degree) line.<\/span><\/p>\n The left panel shows that this is the case for every student in the Evenly Spaced Deadlines section. In the right panel we see the 13 observations with altered grades. And, critically, 12 of these were in the direction of the authors\u2019 hypothesis that spreading deadlines improves grades<\/span><\/p>\n All of the grades that were re-computed to be lower were from students who did not spread their deadlines (red dots), and 5 of the 6 that were re-computed to be higher were from students who did spread their deadlines (blue dots).<\/span><\/p>\n As mentioned, without these 13 alterations Finding #3 disappears [3<\/a>].<\/span><\/p>\n An interesting fact is that these data alterations, all in the Set Your Own Deadline section, somehow did not at all influence the overall<\/em> average grade of that section. The average final grade in the spreadsheet is 85.76 and the average recomputed final grade is 85.76. So the data tamperer altered the data in a way that got the new result they wanted while preserving the original mean. Why would someone do that?<\/span><\/p>\n Mr. Green In The Ballroom With The Candlestick This led us to suspect that Finding #3 had been added to the paper in response to reviewers. And that made us wish that we could see an earlier version of the paper, one that was written before the authors responded to reviewers.<\/span><\/p>\n To our surprise, we found an earlier version of the paper posted to the INSEAD Working Paper Series back in 2001 (.htm<\/a>) with a title page indicating that it was \u201cUnder review at Psychological Science”<\/em>.<\/span><\/p>\n Interestingly, the earlier paper differed from the published paper in two important ways [4<\/a>].\u00a0<\/span><\/p>\n First, as we had suspected, the earlier paper did not include Finding #3.<\/span><\/p>\n Second, although the condition means reported in the earlier vs. published manuscripts were, to two decimal places, exactly the same (i.e., in both versions it was 85.76 in the Set Your Own Deadlines condition), the t-tests comparing those means generated different results<\/em>:<\/span><\/p>\n In the published paper, the t-test was reported as: \u00a0\u00a0\u00a0\u00a0\u00a0t<\/em>(97) = 3.03. Why should you care about these small differences in t-values between two versions of a 25-year-old manuscript? Because they have big forensic value. And that\u2019s because we can reproduce both<\/em> results. <\/span>Specifically:<\/span><\/p>\n When we run the t-test with the spreadsheet<\/em> final grades, we get the published paper’s result: t<\/em>(97) = 3.03. Our ability to reproduce the t(95) = 2.92 with the re-computed grades is critical. It shows that when we reverse the grades back to what we think they were prior to the falsification, we get the exact same results that the authors reported in the earlier version of their manuscript. This proves that:<\/span><\/p>\n So Here’s What We Think Happened But when that analysis was run, it didn\u2019t work. To make it work, some of the final grades had to be altered. The person doing the altering wanted to achieve two things at once: (1) generate the desired result, while (2) preserving the condition means, so that those means did not differ between the two versions of the manuscript. And so that is what they did. They altered the data while preserving the means. Now, that is not an easy thing to do, and so perhaps one of the 13 alterations had to go in the wrong direction, simply to get the means right. The authors now had reportable evidence that the differences in performance were due to the different deadlines rather than to something else.<\/span><\/p>\n The result? The publication of a paper that has had widespread (and continued) influence within the behavioral sciences.<\/span><\/p>\n (Kind of Amazing) Post-script: Before reporting on what they said, it is worth absorbing how remarkable that is. This was a time when reviewing was done by mail (e.g., the editor\u2019s letter ends with a reminder to send four copies of the manuscript), and there are very few floppy disks, hard drives, or servers from 2001 that have survived to the present. Opening these reviews was like opening a time capsule to a different era.<\/span><\/p>\n So what was in that time capsule? Well, “Reviewer B” says, \u201cyou need to\u2026 show that the people who imposed deadlines similar to those imposed by the experimenter did about as well as the people who had deadlines imposed by the experimenter<\/em>.\u201d That request was echoed by the editor. The response to that request was the introduction of Finding #3, the source of the data alterations detailed in this post. And so we have direct evidence that, yes, in fact, this falsified analysis was conducted and included as a response to a request from reviewers.<\/span><\/p>\n We reprint here the author feedback that we showed in our previous post Klaus Wertenbroch <\/strong>sent us a response (PDF<\/a>) in which he begins by thanking Hyndman and Bisin for having done the replication and us for having written these posts. He restates that he never had access to the data for any of the studies. He distinguishes between demand <\/em>for precommitment,<\/em> a finding that was replicated by Hyndman and Bisin and which is consistent with earlier work by him and others, and the effectiveness of such precommitments in these specific studies, which did not replicate. And he indicated that he has asked the editor to retract the paper.<\/span><\/p>\n You can read his response in full (PDF<\/a>).\u00a0<\/span><\/p>\n Dan Ariely <\/strong>did not reply to any of the three emails we sent him. But on August 7th, he wrote on LinkedIn (htm)<\/a> and on his personal website (htm<\/a>): “. . . Recently, I was made aware that data underlying a 2002 paper about deadlines and procrastination that I co-authored contained serious anomalies. <\/em>The documentary record I have at my disposal today about those experiments isn\u2019t sufficient to answer the questions that have been raised, and more than two decades, and hundreds of experiments later, my memory is similarly insufficient. <\/em>Moving forward, my responsibility lies in ensuring accuracy \u2013 in updating the record on these experiments and, along with my co-author, cooperating with the journal that first published our paper to support their reviews and retraction processes.”<\/em><\/span><\/p>\n![\"\"](\"https:\/\/datacolada.org\/wp-content\/uploads\/screenshot-klaus-email-1024x552.png\")
<\/span><\/p>\n![\"\"](\"https:\/\/datacolada.org\/wp-content\/uploads\/Scatter-Plot-Actual-vs-Expected-Both-Conditions.png\")
<\/p>\n
\n<\/u><\/strong><\/span>In Study 1, the authors compared students\u2019 performance in two sections of the same MIT executive education course. In both sections, students were required to write three short papers over the course of the semester. In the Evenly Spaced Deadlines section, the papers were due every four weeks, whereas in the Set Your Own Deadlines section, each student set their own due dates at the beginning of the semester and then were expected to meet those deadlines as the semester progressed. In both sections, submissions incurred a 1% penalty for every day they were turned in after the deadline.<\/span><\/p>\n
\n<\/u><\/strong><\/span>The first step to understanding what happened in this study is to orient to the data. Here is a screenshot of the spreadsheet, formatted to make it easier to digest:<\/span><\/p>\n![\"\"](\"https:\/\/datacolada.org\/wp-content\/uploads\/Annotated-Spreadsheet-Screenshot-07-12-26-scaled.png\")
Fig 1<\/strong>. Screenshot of the Study 1 Spreadsheet<\/em>
\n<\/em><\/span>Note: For clarity, we show only a subset of rows, with renamed and reordered columns, rounded numbers, and added color.<\/em><\/span><\/p>\n
\n<\/u><\/strong><\/span>Because the failed replication focused on Study 2 (see our previous post), we weren\u2019t originally planning to closely examine the Study 1 data. But we decided to take a look, and, in an effort to understand the data file, we tried to figure out how the final grades were computed. Specifically, how were the five grade components (e.g., participation, final exam) combined to generate the final grade?<\/span><\/p>\n![\"\"](\"https:\/\/datacolada.org\/wp-content\/uploads\/Reg-1.png\")
Fig 2.<\/strong> <\/span>Regression predicting final grades with component grades<\/span><\/em><\/p>\n![\"\"](\"https:\/\/datacolada.org\/wp-content\/uploads\/reg-2-1.png\")
Fig 3.<\/strong> Regression predicting final grades with component grades (<\/em>Evenly Spaced Deadlines section).<\/em><\/span><\/p>\n![\"\"](\"https:\/\/datacolada.org\/wp-content\/uploads\/reg-3.png\")
Fig 4.<\/strong> Regression predicting final grades with component grades (<\/em>Set Your Own Deadlines section).<\/em><\/span><\/p>\n
\n<\/u><\/strong><\/span>OK, so in one section grades are a simple average and in the other we don’t know what’s going on. To try to figure it out, we went back and regraded every student in the Set Your Own Deadlines section, relying on the exact same rule from the Evenly Spaced Deadlines section: first giving everyone a 100% participation grade, and then averaging all five component grades. We call this the \u201cre-computed<\/strong> Final Grade\u201d.<\/span><\/p>\n![\"\"](\"https:\/\/datacolada.org\/wp-content\/uploads\/Frequency-of-Grade-Alterations-e1783897255743.png\")
Fig 5.<\/strong> Difference between final grade in the spreadsheet and the re-computed final grade generated by averaging the five component grades.<\/em><\/span><\/p>\n
\n<\/u><\/strong><\/span>If final grades were intentionally<\/em> altered, then they were presumably altered with a purpose, likely to provide (false) evidence for one of the authors\u2019 hypotheses. And, indeed, without the 13 grade alterations, there is no Finding #3.<\/span><\/p>\n<\/p>\n
\n<\/u><\/strong><\/span>When reading the paper closely, we got the sense that Finding #3 was being presented without full conviction, the way authors reluctantly report results reviewers requested. For example, the authors motivate Finding #3 by writing, \u201cit is possible that students in the [Set Your Own Deadlines] condition treated their deadlines as less binding” (p. 222), while saying in Footnote 3 why they don’t think that’s likely: \u201cEvidence against this argument is that all students (in both sections) handed in their papers on or before the deadlines\u201d (p. 222).<\/span><\/p>\n
\n<\/span>In the earlier version of the paper, it was reported as: t<\/em>(95) = 2.92.
\n<\/span>(In about 30 seconds, we will explain why the degrees of freedom – 95 vs. 97 \u2013 are different.)<\/span><\/p>\n
\n<\/span>When we run the t-test with the re-computed<\/em> final grades, we get the earlier paper’s result:\u00a0 \u00a0 \u00a0t<\/em>(95) = 2.92.
\n(As promised 30 seconds ago, this footnote explains why the degrees of freedom are not identical: [5<\/a>]).<\/span><\/span><\/p>\n\n
![\"\"](\"https:\/\/datacolada.org\/wp-content\/uploads\/Clue-1024x683.png\")
<\/p>\n
\n<\/strong><\/span>In the original manuscript, the authors reported that the Evenly Spaced Deadlines section did better than the Set Your Own Deadlines section. Because this could occur for many reasons, someone on the review team asked the authors to demonstrate that the deadlines were the driving force. One way to do that is to show that students who set<\/em> evenly-spaced deadlines did as well as those assigned<\/em> evenly-spaced deadlines.<\/span><\/p>\n
\n<\/strong><\/span>After drafting this post, we reached out to the original authors, first to Wertenbroch and then to Ariely. Their responses appear in the blue text at the end of this post, but we want to highlight something separate and kind of astonishing. Wertenbroch asked if he could provide us with anything useful, and we asked whether he had his reviews from 2001. To our surprise he did. And he immediately sent them to us [6<\/a>]. (We did some fact-checking of the reviews’ authenticity; see Appendix 2 pdf<\/a>).<\/span><\/p>\n![\"Wide](\"https:\/\/datacolada.org\/wp-content\/uploads\/2014\/02\/Wide-logo-300x145.jpg\"){kind=logo}
<\/span><\/p>\n
\n
\n<\/strong>About 6 weeks ago, on July 20th, 2026, we shared drafts of our posts with the original authors (Dan Ariely and Klaus Wertenbroch), the replication authors (Kyle Hyndman and Alberto Bisin), and the editor-in-chief of Psychological Science <\/em>(Simine Vazire).<\/span><\/p>\n
\n
\n