Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

Nate Breznau; Eike Mark Rinke; Alexander Wuttke; Hung H. V. Nguyen; Muna Adem; Jule Adriaans; Amalia Alvarez-Benjumea; Henrik K. Andersen; Daniel Auer; Flavio Azevedo; Oke Bahnsen; Dave Balzer; Gerrit Bauer; Paul C. Bauer; Markus Baumann; Sharon Baute; Verena Benoit; Julian Bernauer; Carl Berning; Anna Berthold; Felix S. Bethke; Thomas Biegert; Katharina Blinzler; Johannes N. Blumenberg; Licia Bobzien; Andrea Bohman; Thijs Bol; Amie Bostic; Zuzanna Brzozowska; Katharina Burgdorf; Kaspar Burger; Kathrin B. Busch; Juan Carlos-Castillo; Nathan Chan; Pablo Christmann; Roxanne Connelly; Christian S. Czymara; Elena Damian; Alejandro Ecker; Achim Edelmann; Maureen A. Eger; Simon Ellerbrock; Anna Forke; Andrea Forster; Chris Gaasendam; Konstantin Gavras; Vernon Gayle; Theresa Gessler; Timo Gnambs; Amélie Godefroidt; Max Grömping; Martin Groß; Stefan Gruber; Tobias Gummer; Andreas Hadjar; Jan Paul Heisig; Sebastian Hellmeier; Stefanie Heyne; Magdalena Hirsch; Mikael Hjerm; Oshrat Hochman; Andreas Hövermann; Sophia Hunger; Christian Hunkler; Nora Huth; Zsófia S. Ignácz; Laura Jacobs; Jannes Jacobsen; Bastian Jaeger; Sebastian Jungkunz; Nils Jungmann; Mathias Kauff; Manuel Kleinert; Julia Klinger; Jan-philipp Kolb; Marta Kołczyńska; John Kuk; Katharina Kunißen; Dafina Kurti sinatra; Alexander Langenkamp; Philipp M. Lersch; Lea-maria Löbel; Philipp Lutscher; Matthias Mader; Joan E. Madia; Natalia Malancu; Luis Maldonado; Helge Marahrens; Nicole Martin; Paul Martinez; Jochen Mayerl; Oscar J. Mayorga; Patricia Mcmanus; Kyle Mcwagner; Cecil Meeusen; Daniel Meierrieks; Jonathan Mellon; Friedolin Merhout; Samuel Merk; Daniel Meyer; Leticia Micheli; Jonathan Mijs; Cristóbal Moya; Marcel Neunhoeffer; Daniel Nüst; Olav Nygård; Fabian Ochsenfeld; Gunnar Otte; Anna O. Pechenkina; Christopher Prosser; Louis Raes; Kevin Ralston; Miguel R. Ramos; Arne Roets; Jonathan Rogers; Guido Ropers; Robin Samuel; Gregor Sand; Ariela Schachter; Merlin Schaeffer; David Schieferdecker; Elmar Schlueter; Regine Schmidt; Katja M. Schmidt; Alexander Schmidt-Catran; Claudia Schmiedeberg; Jürgen Schneider; Martijn Schoonvelde; Julia Schulte-Cloos; Sandy Schumann; Reinhard Schunck; Jürgen Schupp; Julian Seuring; Henning Silber; Willem Sleegers; Nico Sonntag; Alexander Staudt; Nadia Steiber; Nils Steiner; Sebastian Sternberg; Dieter Stiers; Dragana Stojmenovska; Nora Storz; Erich Striessnig; Anne-kathrin Stroppe; Janna Teltemann; Andrey Tibajev; Brian Tung; Giacomo Vagni; Jasper Van assche; Meta Van der linden; Jolanda Van der noll; Arno Van hootegem; Stefan Vogtenhuber; Bogdan Voicu; Fieke Wagemans; Nadja Wehl; Hannah Werner; Brenton M. Wiernik; Fabian Winter; Christof Wolf; Yuki Yamada; Nan Zhang; Conrad Ziller; Stefan Zins; Tomasz Żółtak

doi:10.1073/pnas.2203150119

Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

Nate Breznau^*, Eike Mark Rinke, Alexander Wuttke, Hung H. V. Nguyen, Muna Adem, Jule Adriaans, Amalia Alvarez-Benjumea, Henrik K. Andersen, Daniel Auer, Flavio Azevedo, Oke Bahnsen, Dave Balzer, Gerrit Bauer, Paul C. Bauer, Markus Baumann, Sharon Baute, Verena Benoit, Julian Bernauer, Carl Berning, Anna BertholdFelix S. Bethke, Thomas Biegert, Katharina Blinzler, Johannes N. Blumenberg, Licia Bobzien, Andrea Bohman, Thijs Bol, Amie Bostic, Zuzanna Brzozowska, Katharina Burgdorf, Kaspar Burger, Kathrin B. Busch, Juan Carlos-Castillo, Nathan Chan, Pablo Christmann, Roxanne Connelly, Christian S. Czymara, Elena Damian, Alejandro Ecker, Achim Edelmann, Maureen A. Eger, Simon Ellerbrock, Anna Forke, Andrea Forster, Chris Gaasendam, Konstantin Gavras, Vernon Gayle, Theresa Gessler, Timo Gnambs, Amélie Godefroidt, Max Grömping, Martin Groß, Stefan Gruber, Tobias Gummer, Andreas Hadjar, Jan Paul Heisig, Sebastian Hellmeier, Stefanie Heyne, Magdalena Hirsch, Mikael Hjerm, Oshrat Hochman, Andreas Hövermann, Sophia Hunger, Christian Hunkler, Nora Huth, Zsófia S. Ignácz, Laura Jacobs, Jannes Jacobsen, Bastian Jaeger, Sebastian Jungkunz, Nils Jungmann, Mathias Kauff, Manuel Kleinert, Julia Klinger, Jan-philipp Kolb, Marta Kołczyńska, John Kuk, Katharina Kunißen, Dafina Kurti sinatra, Alexander Langenkamp, Philipp M. Lersch, Lea-maria Löbel, Philipp Lutscher, Matthias Mader, Joan E. Madia, Natalia Malancu, Luis Maldonado, Helge Marahrens, Nicole Martin, Paul Martinez, Jochen Mayerl, Oscar J. Mayorga, Patricia Mcmanus, Kyle Mcwagner, Cecil Meeusen, Daniel Meierrieks, Jonathan Mellon, Friedolin Merhout, Samuel Merk, Daniel Meyer, Leticia Micheli, Jonathan Mijs, Cristóbal Moya, Marcel Neunhoeffer, Daniel Nüst, Olav Nygård, Fabian Ochsenfeld, Gunnar Otte, Anna O. Pechenkina, Christopher Prosser, Louis Raes, Kevin Ralston, Miguel R. Ramos, Arne Roets, Jonathan Rogers, Guido Ropers, Robin Samuel, Gregor Sand, Ariela Schachter, Merlin Schaeffer, David Schieferdecker, Elmar Schlueter, Regine Schmidt, Katja M. Schmidt, Alexander Schmidt-Catran, Claudia Schmiedeberg, Jürgen Schneider, Martijn Schoonvelde, Julia Schulte-Cloos, Sandy Schumann, Reinhard Schunck, Jürgen Schupp, Julian Seuring, Henning Silber, Willem Sleegers, Nico Sonntag, Alexander Staudt, Nadia Steiber, Nils Steiner, Sebastian Sternberg, Dieter Stiers, Dragana Stojmenovska, Nora Storz, Erich Striessnig, Anne-kathrin Stroppe, Janna Teltemann, Andrey Tibajev, Brian Tung, Giacomo Vagni, Jasper Van assche, Meta Van der linden, Jolanda Van der noll, Arno Van hootegem, Stefan Vogtenhuber, Bogdan Voicu, Fieke Wagemans, Nadja Wehl, Hannah Werner, Brenton M. Wiernik, Fabian Winter, Christof Wolf, Yuki Yamada, Nan Zhang, Conrad Ziller, Stefan Zins, Tomasz Żółtak

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

4 Downloads (Pure)

Abstract

This study explores how researchers’ analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.

Original language	English
Article number	e2203150119
Number of pages	8
Journal	Proceedings of the National Academy of Sciences
Volume	119
Issue number	44
DOIs	https://doi.org/10.1073/pnas.2203150119
Publication status	Published - 28 Oct 2022

Bibliographical note

Acknowledgments:
We thank the Mannheim Centre for European Social Research at the University of Mannheim (Mannheim, Germany) for providing extra institutional support for this large-scale project and Joachim Gassen at the Humboldt University (Berlin, Germany) for inspiration and feedback for our interactive app. Additionally helpful were comments provided at the Stanford Meta Research Innovation Center at Stanford (METRICS) International Forum, the Ludwig Maximilian University of Munich Sociology Research Colloquium, and the Technical University Chemnitz “Open Science” Conference and in personal communications with Steve Velsko. A portion of the late phases of coding and shiny app development was supported by a Freies Wissen (“Open Knowledge”) Fellowship of Wikimedia. The views expressed herein are those of the authors and do not reflect the position of the US Military Academy, the Department of the Army, or the Department of Defense.

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1073/pnas.2203150119Licence: Creative Commons: Attribution (CC BY)

BreznauN2022ObservingFinal published version, 3.36 MBLicence: Creative Commons: Attribution (CC BY)

Cite this

Breznau, N., Rinke, E. M., Wuttke, A., Nguyen, H. H. V., Adem, M., Adriaans, J., Alvarez-Benjumea, A., Andersen, H. K., Auer, D., Azevedo, F., Bahnsen, O., Balzer, D., Bauer, G., Bauer, P. C., Baumann, M., Baute, S., Benoit, V., Bernauer, J., Berning, C., ... Żółtak, T. (2022). Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty. Proceedings of the National Academy of Sciences, 119(44), Article e2203150119. https://doi.org/10.1073/pnas.2203150119

@article{1950433ab7674dca9a3f8930a11c43ae,

title = "Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty",

abstract = "This study explores how researchers{\textquoteright} analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers{\textquoteright} expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team{\textquoteright}s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers{\textquoteright} results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.",

author = "Nate Breznau and Rinke, {Eike Mark} and Alexander Wuttke and Nguyen, {Hung H. V.} and Muna Adem and Jule Adriaans and Amalia Alvarez-Benjumea and Andersen, {Henrik K.} and Daniel Auer and Flavio Azevedo and Oke Bahnsen and Dave Balzer and Gerrit Bauer and Bauer, {Paul C.} and Markus Baumann and Sharon Baute and Verena Benoit and Julian Bernauer and Carl Berning and Anna Berthold and Bethke, {Felix S.} and Thomas Biegert and Katharina Blinzler and Blumenberg, {Johannes N.} and Licia Bobzien and Andrea Bohman and Thijs Bol and Amie Bostic and Zuzanna Brzozowska and Katharina Burgdorf and Kaspar Burger and Busch, {Kathrin B.} and Juan Carlos-Castillo and Nathan Chan and Pablo Christmann and Roxanne Connelly and Czymara, {Christian S.} and Elena Damian and Alejandro Ecker and Achim Edelmann and Eger, {Maureen A.} and Simon Ellerbrock and Anna Forke and Andrea Forster and Chris Gaasendam and Konstantin Gavras and Vernon Gayle and Theresa Gessler and Timo Gnambs and Am{\'e}lie Godefroidt and Max Gr{\"o}mping and Martin Gro{\ss} and Stefan Gruber and Tobias Gummer and Andreas Hadjar and Heisig, {Jan Paul} and Sebastian Hellmeier and Stefanie Heyne and Magdalena Hirsch and Mikael Hjerm and Oshrat Hochman and Andreas H{\"o}vermann and Sophia Hunger and Christian Hunkler and Nora Huth and Ign{\'a}cz, {Zs{\'o}fia S.} and Laura Jacobs and Jannes Jacobsen and Bastian Jaeger and Sebastian Jungkunz and Nils Jungmann and Mathias Kauff and Manuel Kleinert and Julia Klinger and Jan-philipp Kolb and Marta Ko{\l}czy{\'n}ska and John Kuk and Katharina Kuni{\ss}en and {Kurti sinatra}, Dafina and Alexander Langenkamp and Lersch, {Philipp M.} and Lea-maria L{\"o}bel and Philipp Lutscher and Matthias Mader and Madia, {Joan E.} and Natalia Malancu and Luis Maldonado and Helge Marahrens and Nicole Martin and Paul Martinez and Jochen Mayerl and Mayorga, {Oscar J.} and Patricia Mcmanus and Kyle Mcwagner and Cecil Meeusen and Daniel Meierrieks and Jonathan Mellon and Friedolin Merhout and Samuel Merk and Daniel Meyer and Leticia Micheli and Jonathan Mijs and Crist{\'o}bal Moya and Marcel Neunhoeffer and Daniel N{\"u}st and Olav Nyg{\aa}rd and Fabian Ochsenfeld and Gunnar Otte and Pechenkina, {Anna O.} and Christopher Prosser and Louis Raes and Kevin Ralston and Ramos, {Miguel R.} and Arne Roets and Jonathan Rogers and Guido Ropers and Robin Samuel and Gregor Sand and Ariela Schachter and Merlin Schaeffer and David Schieferdecker and Elmar Schlueter and Regine Schmidt and Schmidt, {Katja M.} and Alexander Schmidt-Catran and Claudia Schmiedeberg and J{\"u}rgen Schneider and Martijn Schoonvelde and Julia Schulte-Cloos and Sandy Schumann and Reinhard Schunck and J{\"u}rgen Schupp and Julian Seuring and Henning Silber and Willem Sleegers and Nico Sonntag and Alexander Staudt and Nadia Steiber and Nils Steiner and Sebastian Sternberg and Dieter Stiers and Dragana Stojmenovska and Nora Storz and Erich Striessnig and Anne-kathrin Stroppe and Janna Teltemann and Andrey Tibajev and Brian Tung and Giacomo Vagni and {Van assche}, Jasper and {Van der linden}, Meta and {Van der noll}, Jolanda and {Van hootegem}, Arno and Stefan Vogtenhuber and Bogdan Voicu and Fieke Wagemans and Nadja Wehl and Hannah Werner and Wiernik, {Brenton M.} and Fabian Winter and Christof Wolf and Yuki Yamada and Nan Zhang and Conrad Ziller and Stefan Zins and Tomasz {\.Z}{\'o}{\l}tak",

note = "Acknowledgments: We thank the Mannheim Centre for European Social Research at the University of Mannheim (Mannheim, Germany) for providing extra institutional support for this large-scale project and Joachim Gassen at the Humboldt University (Berlin, Germany) for inspiration and feedback for our interactive app. Additionally helpful were comments provided at the Stanford Meta Research Innovation Center at Stanford (METRICS) International Forum, the Ludwig Maximilian University of Munich Sociology Research Colloquium, and the Technical University Chemnitz “Open Science” Conference and in personal communications with Steve Velsko. A portion of the late phases of coding and shiny app development was supported by a Freies Wissen (“Open Knowledge”) Fellowship of Wikimedia. The views expressed herein are those of the authors and do not reflect the position of the US Military Academy, the Department of the Army, or the Department of Defense.",

year = "2022",

month = oct,

day = "28",

doi = "10.1073/pnas.2203150119",

language = "English",

volume = "119",

journal = "Proceedings of the National Academy of Sciences",

issn = "0027-8424",

publisher = "National Academy of Sciences",

number = "44",

}

Breznau, N, Rinke, EM, Wuttke, A, Nguyen, HHV, Adem, M, Adriaans, J, Alvarez-Benjumea, A, Andersen, HK, Auer, D, Azevedo, F, Bahnsen, O, Balzer, D, Bauer, G, Bauer, PC, Baumann, M, Baute, S, Benoit, V, Bernauer, J, Berning, C, Berthold, A, Bethke, FS, Biegert, T, Blinzler, K, Blumenberg, JN, Bobzien, L, Bohman, A, Bol, T, Bostic, A, Brzozowska, Z, Burgdorf, K, Burger, K, Busch, KB, Carlos-Castillo, J, Chan, N, Christmann, P, Connelly, R, Czymara, CS, Damian, E, Ecker, A, Edelmann, A, Eger, MA, Ellerbrock, S, Forke, A, Forster, A, Gaasendam, C, Gavras, K, Gayle, V, Gessler, T, Gnambs, T, Godefroidt, A, Grömping, M, Groß, M, Gruber, S, Gummer, T, Hadjar, A, Heisig, JP, Hellmeier, S, Heyne, S, Hirsch, M, Hjerm, M, Hochman, O, Hövermann, A, Hunger, S, Hunkler, C, Huth, N, Ignácz, ZS, Jacobs, L, Jacobsen, J, Jaeger, B, Jungkunz, S, Jungmann, N, Kauff, M, Kleinert, M, Klinger, J, Kolb, J, Kołczyńska, M, Kuk, J, Kunißen, K, Kurti sinatra, D, Langenkamp, A, Lersch, PM, Löbel, L, Lutscher, P, Mader, M, Madia, JE, Malancu, N, Maldonado, L, Marahrens, H, Martin, N, Martinez, P, Mayerl, J, Mayorga, OJ, Mcmanus, P, Mcwagner, K, Meeusen, C, Meierrieks, D, Mellon, J, Merhout, F, Merk, S, Meyer, D, Micheli, L, Mijs, J, Moya, C, Neunhoeffer, M, Nüst, D, Nygård, O, Ochsenfeld, F, Otte, G, Pechenkina, AO, Prosser, C, Raes, L, Ralston, K, Ramos, MR, Roets, A, Rogers, J, Ropers, G, Samuel, R, Sand, G, Schachter, A, Schaeffer, M, Schieferdecker, D, Schlueter, E, Schmidt, R, Schmidt, KM, Schmidt-Catran, A, Schmiedeberg, C, Schneider, J, Schoonvelde, M, Schulte-Cloos, J, Schumann, S, Schunck, R, Schupp, J, Seuring, J, Silber, H, Sleegers, W, Sonntag, N, Staudt, A, Steiber, N, Steiner, N, Sternberg, S, Stiers, D, Stojmenovska, D, Storz, N, Striessnig, E, Stroppe, A, Teltemann, J, Tibajev, A, Tung, B, Vagni, G, Van assche, J, Van der linden, M, Van der noll, J, Van hootegem, A, Vogtenhuber, S, Voicu, B, Wagemans, F, Wehl, N, Werner, H, Wiernik, BM, Winter, F, Wolf, C, Yamada, Y, Zhang, N, Ziller, C, Zins, S & Żółtak, T 2022, 'Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty', Proceedings of the National Academy of Sciences, vol. 119, no. 44, e2203150119. https://doi.org/10.1073/pnas.2203150119

TY - JOUR

T1 - Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

AU - Breznau, Nate

AU - Rinke, Eike Mark

AU - Wuttke, Alexander

AU - Nguyen, Hung H. V.

AU - Adem, Muna

AU - Adriaans, Jule

AU - Alvarez-Benjumea, Amalia

AU - Andersen, Henrik K.

AU - Auer, Daniel

AU - Azevedo, Flavio

AU - Bahnsen, Oke

AU - Balzer, Dave

AU - Bauer, Gerrit

AU - Bauer, Paul C.

AU - Baumann, Markus

AU - Baute, Sharon

AU - Benoit, Verena

AU - Bernauer, Julian

AU - Berning, Carl

AU - Berthold, Anna

AU - Bethke, Felix S.

AU - Biegert, Thomas

AU - Blinzler, Katharina

AU - Blumenberg, Johannes N.

AU - Bobzien, Licia

AU - Bohman, Andrea

AU - Bol, Thijs

AU - Bostic, Amie

AU - Brzozowska, Zuzanna

AU - Burgdorf, Katharina

AU - Burger, Kaspar

AU - Busch, Kathrin B.

AU - Carlos-Castillo, Juan

AU - Chan, Nathan

AU - Christmann, Pablo

AU - Connelly, Roxanne

AU - Czymara, Christian S.

AU - Damian, Elena

AU - Ecker, Alejandro

AU - Edelmann, Achim

AU - Eger, Maureen A.

AU - Ellerbrock, Simon

AU - Forke, Anna

AU - Forster, Andrea

AU - Gaasendam, Chris

AU - Gavras, Konstantin

AU - Gayle, Vernon

AU - Gessler, Theresa

AU - Gnambs, Timo

AU - Godefroidt, Amélie

AU - Grömping, Max

AU - Groß, Martin

AU - Gruber, Stefan

AU - Gummer, Tobias

AU - Hadjar, Andreas

AU - Heisig, Jan Paul

AU - Hellmeier, Sebastian

AU - Heyne, Stefanie

AU - Hirsch, Magdalena

AU - Hjerm, Mikael

AU - Hochman, Oshrat

AU - Hövermann, Andreas

AU - Hunger, Sophia

AU - Hunkler, Christian

AU - Huth, Nora

AU - Ignácz, Zsófia S.

AU - Jacobs, Laura

AU - Jacobsen, Jannes

AU - Jaeger, Bastian

AU - Jungkunz, Sebastian

AU - Jungmann, Nils

AU - Kauff, Mathias

AU - Kleinert, Manuel

AU - Klinger, Julia

AU - Kolb, Jan-philipp

AU - Kołczyńska, Marta

AU - Kuk, John

AU - Kunißen, Katharina

AU - Kurti sinatra, Dafina

AU - Langenkamp, Alexander

AU - Lersch, Philipp M.

AU - Löbel, Lea-maria

AU - Lutscher, Philipp

AU - Mader, Matthias

AU - Madia, Joan E.

AU - Malancu, Natalia

AU - Maldonado, Luis

AU - Marahrens, Helge

AU - Martin, Nicole

AU - Martinez, Paul

AU - Mayerl, Jochen

AU - Mayorga, Oscar J.

AU - Mcmanus, Patricia

AU - Mcwagner, Kyle

AU - Meeusen, Cecil

AU - Meierrieks, Daniel

AU - Mellon, Jonathan

AU - Merhout, Friedolin

AU - Merk, Samuel

AU - Meyer, Daniel

AU - Micheli, Leticia

AU - Mijs, Jonathan

AU - Moya, Cristóbal

AU - Neunhoeffer, Marcel

AU - Nüst, Daniel

AU - Nygård, Olav

AU - Ochsenfeld, Fabian

AU - Otte, Gunnar

AU - Pechenkina, Anna O.

AU - Prosser, Christopher

AU - Raes, Louis

AU - Ralston, Kevin

AU - Ramos, Miguel R.

AU - Roets, Arne

AU - Rogers, Jonathan

AU - Ropers, Guido

AU - Samuel, Robin

AU - Sand, Gregor

AU - Schachter, Ariela

AU - Schaeffer, Merlin

AU - Schieferdecker, David

AU - Schlueter, Elmar

AU - Schmidt, Regine

AU - Schmidt, Katja M.

AU - Schmidt-Catran, Alexander

AU - Schmiedeberg, Claudia

AU - Schneider, Jürgen

AU - Schoonvelde, Martijn

AU - Schulte-Cloos, Julia

AU - Schumann, Sandy

AU - Schunck, Reinhard

AU - Schupp, Jürgen

AU - Seuring, Julian

AU - Silber, Henning

AU - Sleegers, Willem

AU - Sonntag, Nico

AU - Staudt, Alexander

AU - Steiber, Nadia

AU - Steiner, Nils

AU - Sternberg, Sebastian

AU - Stiers, Dieter

AU - Stojmenovska, Dragana

AU - Storz, Nora

AU - Striessnig, Erich

AU - Stroppe, Anne-kathrin

AU - Teltemann, Janna

AU - Tibajev, Andrey

AU - Tung, Brian

AU - Vagni, Giacomo

AU - Van assche, Jasper

AU - Van der linden, Meta

AU - Van der noll, Jolanda

AU - Van hootegem, Arno

AU - Vogtenhuber, Stefan

AU - Voicu, Bogdan

AU - Wagemans, Fieke

AU - Wehl, Nadja

AU - Werner, Hannah

AU - Wiernik, Brenton M.

AU - Winter, Fabian

AU - Wolf, Christof

AU - Yamada, Yuki

AU - Zhang, Nan

AU - Ziller, Conrad

AU - Zins, Stefan

AU - Żółtak, Tomasz

N1 - Acknowledgments: We thank the Mannheim Centre for European Social Research at the University of Mannheim (Mannheim, Germany) for providing extra institutional support for this large-scale project and Joachim Gassen at the Humboldt University (Berlin, Germany) for inspiration and feedback for our interactive app. Additionally helpful were comments provided at the Stanford Meta Research Innovation Center at Stanford (METRICS) International Forum, the Ludwig Maximilian University of Munich Sociology Research Colloquium, and the Technical University Chemnitz “Open Science” Conference and in personal communications with Steve Velsko. A portion of the late phases of coding and shiny app development was supported by a Freies Wissen (“Open Knowledge”) Fellowship of Wikimedia. The views expressed herein are those of the authors and do not reflect the position of the US Military Academy, the Department of the Army, or the Department of Defense.

PY - 2022/10/28

Y1 - 2022/10/28

N2 - This study explores how researchers’ analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.

AB - This study explores how researchers’ analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.

U2 - 10.1073/pnas.2203150119

DO - 10.1073/pnas.2203150119

M3 - Article

SN - 0027-8424

VL - 119

JO - Proceedings of the National Academy of Sciences

JF - Proceedings of the National Academy of Sciences

IS - 44

M1 - e2203150119

ER -

Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

Abstract

Bibliographical note

UN SDGs

Access to Document

Fingerprint

Cite this