Does the name on a CV change who gets a callback?
Independent rebuild
The study said CVs with white-sounding names got 50% more callbacks than similar CVs with Black-sounding names. The Lab retraced it: ✓ same result.
Whose code is this?
Our own small program, run on a public copy of the experiment's data from OpenIntro. It isn't the authors' code. It got a ratio of 1.4968 against the paper's 1.50. The working paper's table has 2,445 CVs in each group and this data has 2,435, so the two percentages differ a little (9.65% and 6.45% here). It's a settled showcase, so there was no forecasting.
See the codeIt's one of the best-known experiments on hiring, and it's still cited whenever people argue about anonymous CVs.
Cited by 5,025 studies
We test one thing: does this code and data give this number? That's not the same as the science being right. The forecast price never changes the verdict.
See the proof
Counts as holding up
Where the number comes fromAll sent resumes: 10.06%, 6.70%, ratio 1.50Table 1, first row, of the working paper (NBER 9873)The exact claim as posted
Do CVs with white-sounding names get more callbacks? Bertrand and Mullainathan found 1.5 times as many
Bertrand and Mullainathan (American Economic Review, 2004) sent about 5,000 made-up CVs to real job ads in Boston and Chicago, giving them either white-sounding or Black-sounding names at random. CVs with white-sounding names got 1.50 times as many callbacks. This runs our own small program on a public copy of the experiment's data. The working paper's table has slightly more CVs than this data. The rebuild isn't the authors' code, so a mismatch ends as inconclusive.
Does the name on a CV change who gets a callback?
- Measured in
- times the callbacks
- Paper says
- 1.50
- Counts if
- 1.49 to 1.51
- Lab got
- 1.4968
This result is final
Paper said 1.50. We got 1.4968.
The code and data produced the same result. That doesn't prove the study's method is right.
This slip is the Lab's .
✓ Verified on NEAR. See the contract's history
- People forecast
- Forecasting ends
- The Lab retraces it
- Verdict posted, open to challenge
- Winners collect
What we didn't check
- Whether the sample was fair or large enough.
- Whether test data leaked into training or tuning.
- Whether the result holds on new data.
- Whether the study's method was the right one for its question.
- Whether the same gap exists today, or outside Boston and Chicago in 2001 and 2002.
- Whether the names signal only race, and not also class.
No forecasting on this claim. It was posted with the result already known.
The evidence
See the full receiptHow much evidence is behind this?
Each step is a tougher test. Only the first one decides who wins the market.
- 1. ✓
The Lab's retrace of the code and data named in the claim
The Lab retraced the code and data and got a number inside the claim's range. YES wins.
- 2. –
Stress test: does the number depend on luck?
Fetching the extra runs…
- 3. –
Rebuilt from the description (experimental)
Fetching the AI's rebuild…
Coming next: a data-leakage check, for the same rows in training and test data, and a maths check on the statistics a paper reports. Neither has run on this claim.
Even all three steps can't prove the science is true. They show how well this result stands up to checking.
Verified
Your browser just re-checked these itself, so you don't have to trust this site.
Checking…
What the Lab retraces
- Source
- https://doi.org/10.1257/0002828042002561
M. Bertrand, S. Mullainathan; rebuild by the RERUN project, Rebuild: MIT. Data: GPL-3 (OpenIntro), fetched from OpenIntro, not redistributed. - Code, at this exact version
- https://github.com/Maveri-Q/rerun-rebuilds
commit 175eddea88cc8adab8dd20f2100b5b453d359244 - Commands
- python3.11
$ python names-on-cvs-2004/rebuild.py resume.csv > rerun_output.txt && cat rerun_output.txt - Where the result is read
- rerun_output.txt using regex callback_ratio (-?[0-9.]+)
- Data files
- resume.csv
fingerprint 3085bba2a81ac05dde852e697fdc35aa7f4333e0b9a28f535ab6353c7af5d438 - Limits
- 2 CPU, 4 GB, 15 minutes
- 2a325a5acb04febe5efd5b805632cf8c44cab27f04b1a5cad7497defedaaf8cf
Show the claim exactly as stored on the blockchain
{"artifact":{"commit":"175eddea88cc8adab8dd20f2100b5b453d359244","data":[{"path":"resume.csv","sha256":"3085bba2a81ac05dde852e697fdc35aa7f4333e0b9a28f535ab6353c7af5d438","url":"https://www.openintro.org/data/csv/resume.csv"}],"entrypoint":"python names-on-cvs-2004/rebuild.py resume.csv > rerun_output.txt && cat rerun_output.txt","output":{"extractor":{"expr":"callback_ratio (-?[0-9.]+)","type":"regex"},"path":"rerun_output.txt"},"repo_url":"https://github.com/Maveri-Q/rerun-rebuilds","runtime":"python3.11","setup":[]},"metric":{"comparison":"within_tolerance","name":"callbacks for white-sounding names per callback for Black-sounding names","reported_value":"1.50","tolerance":{"type":"abs","value":"0.01"}},"private_data":null,"resources":{"cpu":2,"max_minutes":15,"mem_gb":4},"source":{"authors":"M. Bertrand, S. Mullainathan; rebuild by the RERUN project","doi":"10.1257/0002828042002561","license":"Rebuild: MIT. Data: GPL-3 (OpenIntro), fetched from OpenIntro, not redistributed.","type":"paper","url":"https://doi.org/10.1257/0002828042002561"},"statement":"Bertrand and Mullainathan (American Economic Review, 2004) sent about 5,000 made-up CVs to real job ads in Boston and Chicago, giving them either white-sounding or Black-sounding names at random. CVs with white-sounding names got 1.50 times as many callbacks. This runs our own small program on a public copy of the experiment's data. The working paper's table has slightly more CVs than this data. The rebuild isn't the authors' code, so a mismatch ends as inconclusive.","title":"Do CVs with white-sounding names get more callbacks? Bertrand and Mullainathan found 1.5 times as many","version":1}Community notes
Notes are people's own words, kept on the blockchain. They never change a verdict, and the Lab's AI never reads them. Posting costs a little NEAR for storage, and what isn't used comes back.