Many analysts, one dataset: making transparent how variations in analytical choices affect results

Twenty-nine teams involving 61 analysts used the same dataset to address the same research question: whether soccer referees are more likely to give red cards to dark skin toned players than light skin toned players. Analytic approaches varied widely across teams, and estimated effect sizes ranged f...

Full description

Bibliographic Details
Main Authors:	Silberzahn, R., Uhlmann, E.L., Martin, D.P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M.A., Dalla Rosa, A., Dam, L., Evans, M.H., Flores Cervantes, I., Fong, N., Gamez-Djokic, M., Glenz, A., Gordon-McKeon, S., Heaton, T.J., Hederos, K., Heene, M., Hofelich Mohr, A.J., Högden, F., Hui, K., Johannesson, M., Kalodimos, J., Kaszubowski, E., Kennedy, D.M., Lei, R., Lindsay, T.A., Liverani, S., Madan, C.R., Molden, D., Molleman, E., Morey, R.D., Mulder, L.B., Nijstad, B.R., Pope, N.G., Pope, B., Prenoveau, J.M., Rink, F., Robusto, E., Roderique, H., Sandberg, A., Schlüter, E., Schönbrodt, F.D., Sherman, M.F., Sommer, S.A., Sotak, K., Spain, S., Spörlein, C., Stafford, T., Stefanutti, L., Tauber, S., Ullrich, J., Vianello, M., Wagenmakers, E.-J., Witkowiak, M., Yoon, S., Nosek, B.A.
Format:	Article
Published:	SAGE Publications 2018
Subjects:	Crowdsourcing science; Data analysis; Scientific transparency; Open data; Open materials
Online Access:	https://eprints.nottingham.ac.uk/48166/

_version_	1848797707302338560
author	Silberzahn, R. Uhlmann, E.L. Martin, D.P. Anselmi, P. Aust, F. Awtrey, E. Bahník, Š. Bai, F. Bannard, C. Bonnier, E. Carlsson, R. Cheung, F. Christensen, G. Clay, R. Craig, M.A. Dalla Rosa, A. Dam, L. Evans, M.H. Flores Cervantes, I. Fong, N. Gamez-Djokic, M. Glenz, A. Gordon-McKeon, S. Heaton, T.J. Hederos, K. Heene, M. Hofelich Mohr, A.J. Högden, F. Hui, K. Johannesson, M. Kalodimos, J. Kaszubowski, E. Kennedy, D.M. Lei, R. Lindsay, T.A. Liverani, S. Madan, C.R. Molden, D. Molleman, E. Morey, R.D. Mulder, L.B. Nijstad, B.R. Pope, N.G. Pope, B. Prenoveau, J.M. Rink, F. Robusto, E. Roderique, H. Sandberg, A. Schlüter, E. Schönbrodt, F.D. Sherman, M.F. Sommer, S.A. Sotak, K. Spain, S. Spörlein, C. Stafford, T. Stefanutti, L. Tauber, S. Ullrich, J. Vianello, M. Wagenmakers, E.-J. Witkowiak, M. Yoon, S. Nosek, B.A.
author_facet	Silberzahn, R. Uhlmann, E.L. Martin, D.P. Anselmi, P. Aust, F. Awtrey, E. Bahník, Š. Bai, F. Bannard, C. Bonnier, E. Carlsson, R. Cheung, F. Christensen, G. Clay, R. Craig, M.A. Dalla Rosa, A. Dam, L. Evans, M.H. Flores Cervantes, I. Fong, N. Gamez-Djokic, M. Glenz, A. Gordon-McKeon, S. Heaton, T.J. Hederos, K. Heene, M. Hofelich Mohr, A.J. Högden, F. Hui, K. Johannesson, M. Kalodimos, J. Kaszubowski, E. Kennedy, D.M. Lei, R. Lindsay, T.A. Liverani, S. Madan, C.R. Molden, D. Molleman, E. Morey, R.D. Mulder, L.B. Nijstad, B.R. Pope, N.G. Pope, B. Prenoveau, J.M. Rink, F. Robusto, E. Roderique, H. Sandberg, A. Schlüter, E. Schönbrodt, F.D. Sherman, M.F. Sommer, S.A. Sotak, K. Spain, S. Spörlein, C. Stafford, T. Stefanutti, L. Tauber, S. Ullrich, J. Vianello, M. Wagenmakers, E.-J. Witkowiak, M. Yoon, S. Nosek, B.A.
author_sort	Silberzahn, R.
building	Nottingham Research Data Repository
collection	Online Access
description	Twenty-nine teams involving 61 analysts used the same dataset to address the same research question: whether soccer referees are more likely to give red cards to dark skin toned players than light skin toned players. Analytic approaches varied widely across teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median of 1.31. Twenty teams (69%) found a statistically significant positive effect and nine teams (31%) observed a nonsignificant relationship. Overall 29 different analyses used 21 unique combinations of covariates. We found that neither analysts' prior beliefs about the effect, nor their level of expertise, nor peer-reviewed quality of analysis readily explained variation in analysis outcomes. This suggests that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy by which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective analytic choices influence research results.
first_indexed	2025-11-14T20:08:09Z
format	Article
id	nottingham-48166
institution	University of Nottingham Malaysia Campus
institution_category	Local University
last_indexed	2025-11-14T20:08:09Z
publishDate	2018
publisher	SAGE Publications
recordtype	eprints
repository_type	Digital Repository
spelling	nottingham-481662020-05-04T19:08:33Z https://eprints.nottingham.ac.uk/48166/ Many analysts, one dataset: making transparent how variations in analytical choices affect results Silberzahn, R. Uhlmann, E.L. Martin, D.P. Anselmi, P. Aust, F. Awtrey, E. Bahník, Š. Bai, F. Bannard, C. Bonnier, E. Carlsson, R. Cheung, F. Christensen, G. Clay, R. Craig, M.A. Dalla Rosa, A. Dam, L. Evans, M.H. Flores Cervantes, I. Fong, N. Gamez-Djokic, M. Glenz, A. Gordon-McKeon, S. Heaton, T.J. Hederos, K. Heene, M. Hofelich Mohr, A.J. Högden, F. Hui, K. Johannesson, M. Kalodimos, J. Kaszubowski, E. Kennedy, D.M. Lei, R. Lindsay, T.A. Liverani, S. Madan, C.R. Molden, D. Molleman, E. Morey, R.D. Mulder, L.B. Nijstad, B.R. Pope, N.G. Pope, B. Prenoveau, J.M. Rink, F. Robusto, E. Roderique, H. Sandberg, A. Schlüter, E. Schönbrodt, F.D. Sherman, M.F. Sommer, S.A. Sotak, K. Spain, S. Spörlein, C. Stafford, T. Stefanutti, L. Tauber, S. Ullrich, J. Vianello, M. Wagenmakers, E.-J. Witkowiak, M. Yoon, S. Nosek, B.A. Twenty-nine teams involving 61 analysts used the same dataset to address the same research question: whether soccer referees are more likely to give red cards to dark skin toned players than light skin toned players. Analytic approaches varied widely across teams, and estimated effect sizes ranged from 0.89 to 2.93 in odds ratio units, with a median of 1.31. Twenty teams (69%) found a statistically significant positive effect and nine teams (31%) observed a nonsignificant relationship. Overall 29 different analyses used 21 unique combinations of covariates. We found that neither analysts' prior beliefs about the effect, nor their level of expertise, nor peer-reviewed quality of analysis readily explained variation in analysis outcomes. This suggests that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy by which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective analytic choices influence research results. SAGE Publications 2018-08-23 Article PeerReviewed Silberzahn, R., Uhlmann, E.L., Martin, D.P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M.A., Dalla Rosa, A., Dam, L., Evans, M.H., Flores Cervantes, I., Fong, N., Gamez-Djokic, M., Glenz, A., Gordon-McKeon, S., Heaton, T.J., Hederos, K., Heene, M., Hofelich Mohr, A.J., Högden, F., Hui, K., Johannesson, M., Kalodimos, J., Kaszubowski, E., Kennedy, D.M., Lei, R., Lindsay, T.A., Liverani, S., Madan, C.R., Molden, D., Molleman, E., Morey, R.D., Mulder, L.B., Nijstad, B.R., Pope, N.G., Pope, B., Prenoveau, J.M., Rink, F., Robusto, E., Roderique, H., Sandberg, A., Schlüter, E., Schönbrodt, F.D., Sherman, M.F., Sommer, S.A., Sotak, K., Spain, S., Spörlein, C., Stafford, T., Stefanutti, L., Tauber, S., Ullrich, J., Vianello, M., Wagenmakers, E.-J., Witkowiak, M., Yoon, S. and Nosek, B.A. (2018) Many analysts, one dataset: making transparent how variations in analytical choices affect results. Advances in Methods and Practices in Psychological Science . ISSN 2515-2459 Crowdsourcing science; Data analysis; Scientific transparency; Open data; Open materials http://journals.sagepub.com/doi/10.1177/2515245917747646 doi:10.1177/2515245917747646 doi:10.1177/2515245917747646
spellingShingle	Crowdsourcing science; Data analysis; Scientific transparency; Open data; Open materials Silberzahn, R. Uhlmann, E.L. Martin, D.P. Anselmi, P. Aust, F. Awtrey, E. Bahník, Š. Bai, F. Bannard, C. Bonnier, E. Carlsson, R. Cheung, F. Christensen, G. Clay, R. Craig, M.A. Dalla Rosa, A. Dam, L. Evans, M.H. Flores Cervantes, I. Fong, N. Gamez-Djokic, M. Glenz, A. Gordon-McKeon, S. Heaton, T.J. Hederos, K. Heene, M. Hofelich Mohr, A.J. Högden, F. Hui, K. Johannesson, M. Kalodimos, J. Kaszubowski, E. Kennedy, D.M. Lei, R. Lindsay, T.A. Liverani, S. Madan, C.R. Molden, D. Molleman, E. Morey, R.D. Mulder, L.B. Nijstad, B.R. Pope, N.G. Pope, B. Prenoveau, J.M. Rink, F. Robusto, E. Roderique, H. Sandberg, A. Schlüter, E. Schönbrodt, F.D. Sherman, M.F. Sommer, S.A. Sotak, K. Spain, S. Spörlein, C. Stafford, T. Stefanutti, L. Tauber, S. Ullrich, J. Vianello, M. Wagenmakers, E.-J. Witkowiak, M. Yoon, S. Nosek, B.A. Many analysts, one dataset: making transparent how variations in analytical choices affect results
title	Many analysts, one dataset: making transparent how variations in analytical choices affect results
title_full	Many analysts, one dataset: making transparent how variations in analytical choices affect results
title_fullStr	Many analysts, one dataset: making transparent how variations in analytical choices affect results
title_full_unstemmed	Many analysts, one dataset: making transparent how variations in analytical choices affect results
title_short	Many analysts, one dataset: making transparent how variations in analytical choices affect results
title_sort	many analysts, one dataset: making transparent how variations in analytical choices affect results
topic	Crowdsourcing science; Data analysis; Scientific transparency; Open data; Open materials
url	https://eprints.nottingham.ac.uk/48166/ https://eprints.nottingham.ac.uk/48166/ https://eprints.nottingham.ac.uk/48166/

Many analysts, one dataset: making transparent how variations in analytical choices affect results

Similar Items