Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty
Research output: Contribution to journal › Journal article › Research › peer-review
Standard
Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty. / Breznau, Nate; Rinke, Eike Mark; Wuttke, Alexander; Nguyen, Hung H. V.; Adem, Muna; Adriaans, Jule; Alvarez-Benjumea, Amalia; Andersen, Henrik K.; Auer, Daniel; Azevedo, Flavio; Bahnsen, Oke; Balzer, Dave; Bauer, Gerrit; Bauer, Paul C.; Baumann, Markus; Baute, Sharon; Benoit, Verena; Bernauer, Julian; Berning, Carl; Berthold, Anna; Bethke, Felix S.; Biegert, Thomas; Blinzler, Katharina; Blumenberg, Johannes N.; Bobzien, Licia; Bohman, Andrea; Bol, Thijs; Bostic, Amie; Brzozowska, Zuzanna; Burgdorf, Katharina; Burger, Kaspar; Busch, Kathrin B.; Carlos-Castillo, Juan; Chan, Nathan; Christmann, Pablo; Connelly, Roxanne; Czymara, Christian S.; Damian, Elena; Ecker, Alejandro; Edelmann, Achim; Eger, Maureen A.; Ellerbrock, Simon; Forke, Anna; Forster, Andrea; Gaasendam, Chris; Gavras, Konstantin; Gayle, Vernon; Gessler, Theresa; Merhout, Friedolin; Schaeffer, Merlin; The Crowdsourced Replication Inititative.
In: Proceedings of the National Academy of Sciences of the United States of America, Vol. 119, No. 44, 2203150119, 2022.Research output: Contribution to journal › Journal article › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - JOUR
T1 - Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty
AU - Breznau, Nate
AU - Rinke, Eike Mark
AU - Wuttke, Alexander
AU - Nguyen, Hung H. V.
AU - Adem, Muna
AU - Adriaans, Jule
AU - Alvarez-Benjumea, Amalia
AU - Andersen, Henrik K.
AU - Auer, Daniel
AU - Azevedo, Flavio
AU - Bahnsen, Oke
AU - Balzer, Dave
AU - Bauer, Gerrit
AU - Bauer, Paul C.
AU - Baumann, Markus
AU - Baute, Sharon
AU - Benoit, Verena
AU - Bernauer, Julian
AU - Berning, Carl
AU - Berthold, Anna
AU - Bethke, Felix S.
AU - Biegert, Thomas
AU - Blinzler, Katharina
AU - Blumenberg, Johannes N.
AU - Bobzien, Licia
AU - Bohman, Andrea
AU - Bol, Thijs
AU - Bostic, Amie
AU - Brzozowska, Zuzanna
AU - Burgdorf, Katharina
AU - Burger, Kaspar
AU - Busch, Kathrin B.
AU - Carlos-Castillo, Juan
AU - Chan, Nathan
AU - Christmann, Pablo
AU - Connelly, Roxanne
AU - Czymara, Christian S.
AU - Damian, Elena
AU - Ecker, Alejandro
AU - Edelmann, Achim
AU - Eger, Maureen A.
AU - Ellerbrock, Simon
AU - Forke, Anna
AU - Forster, Andrea
AU - Gaasendam, Chris
AU - Gavras, Konstantin
AU - Gayle, Vernon
AU - Gessler, Theresa
AU - Merhout, Friedolin
AU - Schaeffer, Merlin
AU - The Crowdsourced Replication Inititative
PY - 2022
Y1 - 2022
N2 - This study explores how researchers' analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers' expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team's workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers' results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.
AB - This study explores how researchers' analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers' expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team's workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers' results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.
KW - metascience
KW - many analysts
KW - researcher degrees of freedom
KW - analytical flexibility
KW - immigration and policy preferences
KW - WELFARE-STATE
KW - IMMIGRATION
KW - SUPPORT
KW - REDISTRIBUTION
KW - PREFERENCES
KW - ANALYSTS
KW - IDEAS
U2 - 10.1073/pnas.2203150119
DO - 10.1073/pnas.2203150119
M3 - Journal article
C2 - 36306328
VL - 119
JO - Proceedings of the National Academy of Sciences of the United States of America
JF - Proceedings of the National Academy of Sciences of the United States of America
SN - 0027-8424
IS - 44
M1 - 2203150119
ER -
ID: 332564184