Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left uncons...

Full description

Bibliographic Details
Main Authors:	Schweinsberg, M., Feldman, M., Staub, N., van den Akker, O.R., van Aert, R.C.M., van Assen, M.A.L.M., Liu, Y., Althoff, T., Heer, J., Kale, A., Mohamed, Z., Amireh, H., Venkatesh Prasad, V., Bernstein, A., Robinson, E., Snellman, K., Amy Sommer, S., Otner, S.M.G., Robinson, D., Madan, N., Silberzahn, R., Goldstein, P., Tierney, W., Murase, T., Mandl, B., Viganola, D., Strobl, C., Schaumans, C.B.C., Kelchtermans, S., Naseeb, C., Mason Garrison, S., Yarkoni, T., Richard Chan, C.S., Adie, P., Alaburda, P., Albers, C., Alspaugh, S., Alstott, J., Nelson, A.A., Ariño de la Rubia, E., Arzi, A., Bahník, Š., Baik, J., Winther Balling, L., Banker, S., AA Baranger, D., Barr, D.J., Barros-Rivera, B., Bauer, M., Blaise, E., Boelen, L., Bohle Carbonell, K., Briers, R.A., Burkhard, O., Canela, M.-A., Castrillo, L., Catlett, T., Chen, O., Clark, M., Cohn, B., Coppock, A., Cugueró-Escofet, N., Curran, P.G., Cyrus-Lai, W., Dai, D., Valentino Dalla Riva, G., Danielsson, H., Russo, R.D.F.S.M., de Silva, N., Derungs, C., Dondelinger, F., Duarte de Souza, C., Tyson Dube, B., Dubova, M., Mark Dunn, B., Adriaan Edelsbrunner, P., Finley, S., Fox, N., Gnambs, T., Gong, Y., Grand, E., Greenawalt, B., Han, D., Hanel, P.H.P., Hong, A.B., Hood, D., Hsueh, J., Huang, L., Hui, K.N., Hultman, K.A., Javaid, A., Ji Jiang, L., Jong, J., Kamdar, J., Kane, D., Kappler, G., Kaszubowski, E., Kavanagh, C.M., Khabsa, M., Kleinberg, B., Kouros, J., Krause, H., Krypotos, A.-M., Lavbič, D., Ling Lee, R., Leffel, T., Yang Lim, W., Liverani, S., Loh, B., Lønsmann, D., Wei Low, J., Lu, A., MacDonald, K., Madan, C.R., Hjorth Madsen, L., Maimone, C., Mangold, A., Marshall, A., Ester Matskewich, H., Mavon, K., McLain, K.L., McNamara, A.A., McNeill, M., Mertens, U., Miller, D., Moore, B., Moore, A., Nantz, E., Nasrullah, Z., Nejkovic, V., Nell, C.S., Arthur Nelson, A., Nilsonne, G., Nolan, R., O Brien, C.E., O Neill, P., O Shea, K., Olita, Toto, Otterbacher, J., Palsetia, D., Pereira, B., Pozdniakov, I., Protzko, J., Reyt, J.-N., Riddle, T., Akmal Ridhwan Omar Ali, A., Ropovik, I., Rosenberg, J.M., Rothen, S., Schulte-Mecklenbeck, M., Sharma, N., Shotwell, G., Skarzynski, M., Stedden, W., Stodden, V., Stoffel, M.A., Stoltzman, S., Subbaiah, S., Tatman, R., Thibodeau, P.H., Tomkins, S., Valdivia, A., Druijff-van de Woestijne, G.B., Viana, L., Villesèche, F., Duncan Wadsworth, W., Wanders, F., Watts, K., Wells, J.D., Whelpley, C.E., Won, A., Wu, L., Yip, A., Youngflesh, C., Yu, J.-C., Zandian, A., Zhang, L., Zibman, C., Luis Uhlmann, E.
Format:	Journal Article
Published:	2021
Online Access:	http://hdl.handle.net/20.500.11937/84806

_version_	1848764692554579968
author	Schweinsberg, M. Feldman, M. Staub, N. van den Akker, O.R. van Aert, R.C.M. van Assen, M.A.L.M. Liu, Y. Althoff, T. Heer, J. Kale, A. Mohamed, Z. Amireh, H. Venkatesh Prasad, V. Bernstein, A. Robinson, E. Snellman, K. Amy Sommer, S. Otner, S.M.G. Robinson, D. Madan, N. Silberzahn, R. Goldstein, P. Tierney, W. Murase, T. Mandl, B. Viganola, D. Strobl, C. Schaumans, C.B.C. Kelchtermans, S. Naseeb, C. Mason Garrison, S. Yarkoni, T. Richard Chan, C.S. Adie, P. Alaburda, P. Albers, C. Alspaugh, S. Alstott, J. Nelson, A.A. Ariño de la Rubia, E. Arzi, A. Bahník, Š. Baik, J. Winther Balling, L. Banker, S. AA Baranger, D. Barr, D.J. Barros-Rivera, B. Bauer, M. Blaise, E. Boelen, L. Bohle Carbonell, K. Briers, R.A. Burkhard, O. Canela, M.-A. Castrillo, L. Catlett, T. Chen, O. Clark, M. Cohn, B. Coppock, A. Cugueró-Escofet, N. Curran, P.G. Cyrus-Lai, W. Dai, D. Valentino Dalla Riva, G. Danielsson, H. Russo, R.D.F.S.M. de Silva, N. Derungs, C. Dondelinger, F. Duarte de Souza, C. Tyson Dube, B. Dubova, M. Mark Dunn, B. Adriaan Edelsbrunner, P. Finley, S. Fox, N. Gnambs, T. Gong, Y. Grand, E. Greenawalt, B. Han, D. Hanel, P.H.P. Hong, A.B. Hood, D. Hsueh, J. Huang, L. Hui, K.N. Hultman, K.A. Javaid, A. Ji Jiang, L. Jong, J. Kamdar, J. Kane, D. Kappler, G. Kaszubowski, E. Kavanagh, C.M. Khabsa, M. Kleinberg, B. Kouros, J. Krause, H. Krypotos, A.-M. Lavbič, D. Ling Lee, R. Leffel, T. Yang Lim, W. Liverani, S. Loh, B. Lønsmann, D. Wei Low, J. Lu, A. MacDonald, K. Madan, C.R. Hjorth Madsen, L. Maimone, C. Mangold, A. Marshall, A. Ester Matskewich, H. Mavon, K. McLain, K.L. McNamara, A.A. McNeill, M. Mertens, U. Miller, D. Moore, B. Moore, A. Nantz, E. Nasrullah, Z. Nejkovic, V. Nell, C.S. Arthur Nelson, A. Nilsonne, G. Nolan, R. O Brien, C.E. O Neill, P. O Shea, K. Olita, Toto Otterbacher, J. Palsetia, D. Pereira, B. Pozdniakov, I. Protzko, J. Reyt, J.-N. Riddle, T. Akmal Ridhwan Omar Ali, A. Ropovik, I. Rosenberg, J.M. Rothen, S. Schulte-Mecklenbeck, M. Sharma, N. Shotwell, G. Skarzynski, M. Stedden, W. Stodden, V. Stoffel, M.A. Stoltzman, S. Subbaiah, S. Tatman, R. Thibodeau, P.H. Tomkins, S. Valdivia, A. Druijff-van de Woestijne, G.B. Viana, L. Villesèche, F. Duncan Wadsworth, W. Wanders, F. Watts, K. Wells, J.D. Whelpley, C.E. Won, A. Wu, L. Yip, A. Youngflesh, C. Yu, J.-C. Zandian, A. Zhang, L. Zibman, C. Luis Uhlmann, E.
author_facet	Schweinsberg, M. Feldman, M. Staub, N. van den Akker, O.R. van Aert, R.C.M. van Assen, M.A.L.M. Liu, Y. Althoff, T. Heer, J. Kale, A. Mohamed, Z. Amireh, H. Venkatesh Prasad, V. Bernstein, A. Robinson, E. Snellman, K. Amy Sommer, S. Otner, S.M.G. Robinson, D. Madan, N. Silberzahn, R. Goldstein, P. Tierney, W. Murase, T. Mandl, B. Viganola, D. Strobl, C. Schaumans, C.B.C. Kelchtermans, S. Naseeb, C. Mason Garrison, S. Yarkoni, T. Richard Chan, C.S. Adie, P. Alaburda, P. Albers, C. Alspaugh, S. Alstott, J. Nelson, A.A. Ariño de la Rubia, E. Arzi, A. Bahník, Š. Baik, J. Winther Balling, L. Banker, S. AA Baranger, D. Barr, D.J. Barros-Rivera, B. Bauer, M. Blaise, E. Boelen, L. Bohle Carbonell, K. Briers, R.A. Burkhard, O. Canela, M.-A. Castrillo, L. Catlett, T. Chen, O. Clark, M. Cohn, B. Coppock, A. Cugueró-Escofet, N. Curran, P.G. Cyrus-Lai, W. Dai, D. Valentino Dalla Riva, G. Danielsson, H. Russo, R.D.F.S.M. de Silva, N. Derungs, C. Dondelinger, F. Duarte de Souza, C. Tyson Dube, B. Dubova, M. Mark Dunn, B. Adriaan Edelsbrunner, P. Finley, S. Fox, N. Gnambs, T. Gong, Y. Grand, E. Greenawalt, B. Han, D. Hanel, P.H.P. Hong, A.B. Hood, D. Hsueh, J. Huang, L. Hui, K.N. Hultman, K.A. Javaid, A. Ji Jiang, L. Jong, J. Kamdar, J. Kane, D. Kappler, G. Kaszubowski, E. Kavanagh, C.M. Khabsa, M. Kleinberg, B. Kouros, J. Krause, H. Krypotos, A.-M. Lavbič, D. Ling Lee, R. Leffel, T. Yang Lim, W. Liverani, S. Loh, B. Lønsmann, D. Wei Low, J. Lu, A. MacDonald, K. Madan, C.R. Hjorth Madsen, L. Maimone, C. Mangold, A. Marshall, A. Ester Matskewich, H. Mavon, K. McLain, K.L. McNamara, A.A. McNeill, M. Mertens, U. Miller, D. Moore, B. Moore, A. Nantz, E. Nasrullah, Z. Nejkovic, V. Nell, C.S. Arthur Nelson, A. Nilsonne, G. Nolan, R. O Brien, C.E. O Neill, P. O Shea, K. Olita, Toto Otterbacher, J. Palsetia, D. Pereira, B. Pozdniakov, I. Protzko, J. Reyt, J.-N. Riddle, T. Akmal Ridhwan Omar Ali, A. Ropovik, I. Rosenberg, J.M. Rothen, S. Schulte-Mecklenbeck, M. Sharma, N. Shotwell, G. Skarzynski, M. Stedden, W. Stodden, V. Stoffel, M.A. Stoltzman, S. Subbaiah, S. Tatman, R. Thibodeau, P.H. Tomkins, S. Valdivia, A. Druijff-van de Woestijne, G.B. Viana, L. Villesèche, F. Duncan Wadsworth, W. Wanders, F. Watts, K. Wells, J.D. Whelpley, C.E. Won, A. Wu, L. Yip, A. Youngflesh, C. Yu, J.-C. Zandian, A. Zhang, L. Zibman, C. Luis Uhlmann, E.
author_sort	Schweinsberg, M.
building	Curtin Institutional Repository
collection	Online Access
description	In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for organizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed.
first_indexed	2025-11-14T11:23:24Z
format	Journal Article
id	curtin-20.500.11937-84806
institution	Curtin University Malaysia
institution_category	Local University
last_indexed	2025-11-14T11:23:24Z
publishDate	2021
recordtype	eprints
repository_type	Digital Repository
spelling	curtin-20.500.11937-848062021-08-12T05:31:23Z Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis Schweinsberg, M. Feldman, M. Staub, N. van den Akker, O.R. van Aert, R.C.M. van Assen, M.A.L.M. Liu, Y. Althoff, T. Heer, J. Kale, A. Mohamed, Z. Amireh, H. Venkatesh Prasad, V. Bernstein, A. Robinson, E. Snellman, K. Amy Sommer, S. Otner, S.M.G. Robinson, D. Madan, N. Silberzahn, R. Goldstein, P. Tierney, W. Murase, T. Mandl, B. Viganola, D. Strobl, C. Schaumans, C.B.C. Kelchtermans, S. Naseeb, C. Mason Garrison, S. Yarkoni, T. Richard Chan, C.S. Adie, P. Alaburda, P. Albers, C. Alspaugh, S. Alstott, J. Nelson, A.A. Ariño de la Rubia, E. Arzi, A. Bahník, Š. Baik, J. Winther Balling, L. Banker, S. AA Baranger, D. Barr, D.J. Barros-Rivera, B. Bauer, M. Blaise, E. Boelen, L. Bohle Carbonell, K. Briers, R.A. Burkhard, O. Canela, M.-A. Castrillo, L. Catlett, T. Chen, O. Clark, M. Cohn, B. Coppock, A. Cugueró-Escofet, N. Curran, P.G. Cyrus-Lai, W. Dai, D. Valentino Dalla Riva, G. Danielsson, H. Russo, R.D.F.S.M. de Silva, N. Derungs, C. Dondelinger, F. Duarte de Souza, C. Tyson Dube, B. Dubova, M. Mark Dunn, B. Adriaan Edelsbrunner, P. Finley, S. Fox, N. Gnambs, T. Gong, Y. Grand, E. Greenawalt, B. Han, D. Hanel, P.H.P. Hong, A.B. Hood, D. Hsueh, J. Huang, L. Hui, K.N. Hultman, K.A. Javaid, A. Ji Jiang, L. Jong, J. Kamdar, J. Kane, D. Kappler, G. Kaszubowski, E. Kavanagh, C.M. Khabsa, M. Kleinberg, B. Kouros, J. Krause, H. Krypotos, A.-M. Lavbič, D. Ling Lee, R. Leffel, T. Yang Lim, W. Liverani, S. Loh, B. Lønsmann, D. Wei Low, J. Lu, A. MacDonald, K. Madan, C.R. Hjorth Madsen, L. Maimone, C. Mangold, A. Marshall, A. Ester Matskewich, H. Mavon, K. McLain, K.L. McNamara, A.A. McNeill, M. Mertens, U. Miller, D. Moore, B. Moore, A. Nantz, E. Nasrullah, Z. Nejkovic, V. Nell, C.S. Arthur Nelson, A. Nilsonne, G. Nolan, R. O Brien, C.E. O Neill, P. O Shea, K. Olita, Toto Otterbacher, J. Palsetia, D. Pereira, B. Pozdniakov, I. Protzko, J. Reyt, J.-N. Riddle, T. Akmal Ridhwan Omar Ali, A. Ropovik, I. Rosenberg, J.M. Rothen, S. Schulte-Mecklenbeck, M. Sharma, N. Shotwell, G. Skarzynski, M. Stedden, W. Stodden, V. Stoffel, M.A. Stoltzman, S. Subbaiah, S. Tatman, R. Thibodeau, P.H. Tomkins, S. Valdivia, A. Druijff-van de Woestijne, G.B. Viana, L. Villesèche, F. Duncan Wadsworth, W. Wanders, F. Watts, K. Wells, J.D. Whelpley, C.E. Won, A. Wu, L. Yip, A. Youngflesh, C. Yu, J.-C. Zandian, A. Zhang, L. Zibman, C. Luis Uhlmann, E. In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for organizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed. 2021 Journal Article http://hdl.handle.net/20.500.11937/84806 10.1016/j.obhdp.2021.02.003 http://creativecommons.org/licenses/by/4.0/ fulltext
spellingShingle	Schweinsberg, M. Feldman, M. Staub, N. van den Akker, O.R. van Aert, R.C.M. van Assen, M.A.L.M. Liu, Y. Althoff, T. Heer, J. Kale, A. Mohamed, Z. Amireh, H. Venkatesh Prasad, V. Bernstein, A. Robinson, E. Snellman, K. Amy Sommer, S. Otner, S.M.G. Robinson, D. Madan, N. Silberzahn, R. Goldstein, P. Tierney, W. Murase, T. Mandl, B. Viganola, D. Strobl, C. Schaumans, C.B.C. Kelchtermans, S. Naseeb, C. Mason Garrison, S. Yarkoni, T. Richard Chan, C.S. Adie, P. Alaburda, P. Albers, C. Alspaugh, S. Alstott, J. Nelson, A.A. Ariño de la Rubia, E. Arzi, A. Bahník, Š. Baik, J. Winther Balling, L. Banker, S. AA Baranger, D. Barr, D.J. Barros-Rivera, B. Bauer, M. Blaise, E. Boelen, L. Bohle Carbonell, K. Briers, R.A. Burkhard, O. Canela, M.-A. Castrillo, L. Catlett, T. Chen, O. Clark, M. Cohn, B. Coppock, A. Cugueró-Escofet, N. Curran, P.G. Cyrus-Lai, W. Dai, D. Valentino Dalla Riva, G. Danielsson, H. Russo, R.D.F.S.M. de Silva, N. Derungs, C. Dondelinger, F. Duarte de Souza, C. Tyson Dube, B. Dubova, M. Mark Dunn, B. Adriaan Edelsbrunner, P. Finley, S. Fox, N. Gnambs, T. Gong, Y. Grand, E. Greenawalt, B. Han, D. Hanel, P.H.P. Hong, A.B. Hood, D. Hsueh, J. Huang, L. Hui, K.N. Hultman, K.A. Javaid, A. Ji Jiang, L. Jong, J. Kamdar, J. Kane, D. Kappler, G. Kaszubowski, E. Kavanagh, C.M. Khabsa, M. Kleinberg, B. Kouros, J. Krause, H. Krypotos, A.-M. Lavbič, D. Ling Lee, R. Leffel, T. Yang Lim, W. Liverani, S. Loh, B. Lønsmann, D. Wei Low, J. Lu, A. MacDonald, K. Madan, C.R. Hjorth Madsen, L. Maimone, C. Mangold, A. Marshall, A. Ester Matskewich, H. Mavon, K. McLain, K.L. McNamara, A.A. McNeill, M. Mertens, U. Miller, D. Moore, B. Moore, A. Nantz, E. Nasrullah, Z. Nejkovic, V. Nell, C.S. Arthur Nelson, A. Nilsonne, G. Nolan, R. O Brien, C.E. O Neill, P. O Shea, K. Olita, Toto Otterbacher, J. Palsetia, D. Pereira, B. Pozdniakov, I. Protzko, J. Reyt, J.-N. Riddle, T. Akmal Ridhwan Omar Ali, A. Ropovik, I. Rosenberg, J.M. Rothen, S. Schulte-Mecklenbeck, M. Sharma, N. Shotwell, G. Skarzynski, M. Stedden, W. Stodden, V. Stoffel, M.A. Stoltzman, S. Subbaiah, S. Tatman, R. Thibodeau, P.H. Tomkins, S. Valdivia, A. Druijff-van de Woestijne, G.B. Viana, L. Villesèche, F. Duncan Wadsworth, W. Wanders, F. Watts, K. Wells, J.D. Whelpley, C.E. Won, A. Wu, L. Yip, A. Youngflesh, C. Yu, J.-C. Zandian, A. Zhang, L. Zibman, C. Luis Uhlmann, E. Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis
title	Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis
title_full	Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis
title_fullStr	Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis
title_full_unstemmed	Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis
title_short	Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis
title_sort	same data, different conclusions: radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis
url	http://hdl.handle.net/20.500.11937/84806

Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

Similar Items