TY - RPRT

T1 - Observing Many Researchers Using the Same Data and Hypothesis Reveals a Hidden Universe of Uncertainty

AU - Merhout, Friedolin

AU - Breznau, Nate

PY - 2021

Y1 - 2021

N2 - This study explores how analytical choices of researchers affect the reliability of scientificfindings. Current lack-of-reliability discussions focus on systematic biases. We broaden the lensto include idiosyncratic decisions in data analysis that lead researchers to diverging results andconclusions. We coordinated and observed decisions among 73 research-teams as theyindependently tested the same hypothesis using the same data. Results show that in this typicalsecondary data research situation, the universe of pathways from data to results is so vast that eachanalysis was unique in some way. Teams reported divergent findings with contradictorysubstantive implications that could not be explained by differences in researchers’ expertise, priorbeliefs, and expectations. This calls for greater humility and clarity in presentation of scientificfindings. Idiosyncratic variation may also be a cause for why many hypotheses remain highlycontested, particularly in large-scale social and behavioral research

AB - This study explores how analytical choices of researchers affect the reliability of scientificfindings. Current lack-of-reliability discussions focus on systematic biases. We broaden the lensto include idiosyncratic decisions in data analysis that lead researchers to diverging results andconclusions. We coordinated and observed decisions among 73 research-teams as theyindependently tested the same hypothesis using the same data. Results show that in this typicalsecondary data research situation, the universe of pathways from data to results is so vast that eachanalysis was unique in some way. Teams reported divergent findings with contradictorysubstantive implications that could not be explained by differences in researchers’ expertise, priorbeliefs, and expectations. This calls for greater humility and clarity in presentation of scientificfindings. Idiosyncratic variation may also be a cause for why many hypotheses remain highlycontested, particularly in large-scale social and behavioral research

M3 - Report

BT - Observing Many Researchers Using the Same Data and Hypothesis Reveals a Hidden Universe of Uncertainty

ER -