RERUN
Claim 30

How much psychology research holds up when it's repeated?

Independent rebuild

The study repeated about 100 psychology studies, and said only 36% got a significant result again. The Lab retraced it: ✓ same result.

Whose code is this?

Our own small program, run on the project's public results table. It isn't the authors' code, and it doesn't redo the experiments. It counted 35 of 97, which is 36.08%, the same count the authors' own script prints. It's a settled showcase, so there was no forecasting.

See the code

This is the study that made the replication crisis front-page news, and it's why a site like this exists.

Cited by 8,484 studies

What this result means

We test one thing: does this code and data give this number? That's not the same as the science being right. The forecast price never changes the verdict.

See the proof
36% significant again, the paper says. Counts if between 35.5 and 36.5.
Where the number comes fromThirty-six percent of replications had statistically significant resultsAbstract of the paper. The authors' own script prints 35/97

The exact claim as posted

When psychology studies were repeated, how many held up? The Reproducibility Project said 36%

The Open Science Collaboration (Science, 2015) repeated 100 published psychology experiments. Of the 97 originals with significant results, 36% of the repeats were significant too. This recounts that share with our own small program on the project's public results table. It doesn't redo the experiments, and a mismatch ends as inconclusive.

RERUNCLAIM #0030

How much psychology research holds up when it's repeated?

Measured in
% significant again
Paper says
36
Counts if
35.5 to 36.5
Lab got
36.0825
VerdictNumbers check out
90a002bd7f0b85c8e044Nov 06, 2026

This result is final

Paper said 36. We got 36.0825.

The code and data produced the same result. That doesn't prove the study's method is right.

This slip is the Lab's .

✓ Verified on NEAR. See the contract's history

See the full receipt

  1. People forecast
  2. Forecasting ends
  3. The Lab retraces it
  4. Verdict posted, open to challenge
  5. Winners collect

What we didn't check

  • Whether the sample was fair or large enough.
  • Whether test data leaked into training or tuning.
  • Whether the result holds on new data.
  • Whether the study's method was the right one for its question.
  • The 100 experiments themselves. We only recount the project's own results table.
  • Whether statistical significance is the right test of a repeat. The paper gives other measures too.

No forecasting on this claim. It was posted with the result already known.

The evidence

See the full receipt

How much evidence is behind this?

Each step is a tougher test. Only the first one decides who wins the market.

  1. 1. ✓

    The Lab's retrace of the code and data named in the claim

    The Lab retraced the code and data and got a number inside the claim's range. YES wins.

  2. 2. –

    Stress test: does the number depend on luck?

    Fetching the extra runs…

  3. 3. –

    Rebuilt from the description (experimental)

    Fetching the AI's rebuild…

Coming next: a data-leakage check, for the same rows in training and test data, and a maths check on the statistics a paper reports. Neither has run on this claim.

Even all three steps can't prove the science is true. They show how well this result stands up to checking.

Verified

Your browser just re-checked these itself, so you don't have to trust this site.

Checking…

What the Lab retraces

Source
https://doi.org/10.1126/science.aac4716
Open Science Collaboration; rebuild by the RERUN project, Rebuild: MIT. Data: CC0, fetched from the project's OSF page, not redistributed.
Code, at this exact version
https://github.com/Maveri-Q/rerun-rebuilds
commit 175eddea88cc8adab8dd20f2100b5b453d359244
Commands
python3.11
$ python reproducibility-project-2015/rebuild.py rpp_data.csv > rerun_output.txt && cat rerun_output.txt
Where the result is read
rerun_output.txt using regex percent_significant_again (-?[0-9.]+)
Data files
rpp_data.csv
fingerprint 4d87a2636181d8e31230422854cf0fabeaecd2f1b47a59627482c05a2b56d54d
Limits
2 CPU, 4 GB, 15 minutes
90a002bd7f0b85c8e04421cea8fca462082b99eba380cb2ce19807bcaed030fb
Show the claim exactly as stored on the blockchain
{"artifact":{"commit":"175eddea88cc8adab8dd20f2100b5b453d359244","data":[{"path":"rpp_data.csv","sha256":"4d87a2636181d8e31230422854cf0fabeaecd2f1b47a59627482c05a2b56d54d","url":"https://osf.io/download/fgjvw/"}],"entrypoint":"python reproducibility-project-2015/rebuild.py rpp_data.csv > rerun_output.txt && cat rerun_output.txt","output":{"extractor":{"expr":"percent_significant_again (-?[0-9.]+)","type":"regex"},"path":"rerun_output.txt"},"repo_url":"https://github.com/Maveri-Q/rerun-rebuilds","runtime":"python3.11","setup":[]},"metric":{"comparison":"within_tolerance","name":"percent of repeated studies significant again","reported_value":"36","tolerance":{"type":"abs","value":"0.5"}},"private_data":null,"resources":{"cpu":2,"max_minutes":15,"mem_gb":4},"source":{"authors":"Open Science Collaboration; rebuild by the RERUN project","doi":"10.1126/science.aac4716","license":"Rebuild: MIT. Data: CC0, fetched from the project's OSF page, not redistributed.","type":"paper","url":"https://doi.org/10.1126/science.aac4716"},"statement":"The Open Science Collaboration (Science, 2015) repeated 100 published psychology experiments. Of the 97 originals with significant results, 36% of the repeats were significant too. This recounts that share with our own small program on the project's public results table. It doesn't redo the experiments, and a mismatch ends as inconclusive.","title":"When psychology studies were repeated, how many held up? The Reproducibility Project said 36%","version":1}

Community notes

Notes are people's own words, kept on the blockchain. They never change a verdict, and the Lab's AI never reads them. Posting costs a little NEAR for storage, and what isn't used comes back.