Advanced search
1 file | 3.85 MB Add to list

Variability in the analysis of a single neuroimaging dataset by many teams

(2020) NATURE. 582(7810). p.84-88
Author
Organization
Abstract
Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses(1). The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset(2-5). Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed. The results obtained by seventy different teams analysing the same functional magnetic resonance imaging dataset show substantial variation, highlighting the influence of analytical choices and the importance of sharing workflows publicly and performing multiple analyses.
Keywords
PREDICTION MARKETS, NEURAL BASIS, REPLICABILITY

Downloads

  • (...).pdf
    • full text (Published version)
    • |
    • UGent only
    • |
    • PDF
    • |
    • 3.85 MB

Citation

Please use this url to cite or link to this publication:

MLA
Botvinik-Nezer, Rotem, et al. “Variability in the Analysis of a Single Neuroimaging Dataset by Many Teams.” NATURE, vol. 582, no. 7810, 2020, pp. 84–88, doi:10.1038/s41586-020-2314-9.
APA
Botvinik-Nezer, R., Holzmeister, F., Camerer, C. F., Dreber, A., Huber, J., Johannesson, M., … Schonberg, T. (2020). Variability in the analysis of a single neuroimaging dataset by many teams. NATURE, 582(7810), 84–88. https://doi.org/10.1038/s41586-020-2314-9
Chicago author-date
Botvinik-Nezer, Rotem, Felix Holzmeister, Colin F. Camerer, Anna Dreber, Juergen Huber, Magnus Johannesson, Michael Kirchler, et al. 2020. “Variability in the Analysis of a Single Neuroimaging Dataset by Many Teams.” NATURE 582 (7810): 84–88. https://doi.org/10.1038/s41586-020-2314-9.
Chicago author-date (all authors)
Botvinik-Nezer, Rotem, Felix Holzmeister, Colin F. Camerer, Anna Dreber, Juergen Huber, Magnus Johannesson, Michael Kirchler, Roni Iwanir, Jeanette A. Mumford, R. Alison Adcock, Paolo Avesani, Blazej M. Baczkowski, Aahana Bajracharya, Leah Bakst, Sheryl Ball, Marco Barilari, Nadege Bault, Derek Beaton, Julia Beitner, Roland G. Benoit, Ruud M. W. J. Berkers, Jamil P. Bhanji, Bharat B. Biswal, Sebastian Bobadilla-Suarez, Tiago Bortolini, Katherine L. Bottenhorn, Alexander Bowring, Senne Braem, Hayley R. Brooks, Emily G. Brudner, Cristian Buc Calderon, Julia A. Camilleri, Jaime J. Castrellon, Luca Cecchetti, Edna C. Cieslik, Zachary J. Cole, Olivier Collignon, Robert W. Cox, William A. Cunningham, Stefan Czoschke, Kamalaker Dadi, Charles P. Davis, Alberto De Luca, Mauricio R. Delgado, Lysia Demetriou, Jeffrey B. Dennison, Xin Di, Erin W. Dickie, Ekaterina Dobryakova, Claire L. Donnat, Juergen Dukart, Niall W. Duncan, Joke Durnez, Amr Eed, Simon B. Eickhoff, Andrew Erhart, Laura Fontanesi, G. Matthew Fricke, Shiguang Fu, Adriana Galvan, Remi Gau, Sarah Genon, Tristan Glatard, Enrico Glerean, Jelle J. Goeman, Sergej A. E. Golowin, Carlos Gonzalez-Garcia, Krzysztof J. Gorgolewski, Cheryl L. Grady, Mikella A. Green, Joao F. Guassi Moreira, Olivia Guest, Shabnam Hakimi, J. Paul Hamilton, Roeland Hancock, Giacomo Handjaras, Bronson B. Harry, Colin Hawco, Peer Herholz, Gabrielle Herman, Stephan Heunis, Felix Hoffstaedter, Jeremy Hogeveen, Susan Holmes, Chuan-Peng Hu, Scott A. Huettel, Matthew E. Hughes, Vittorio Iacovella, Alexandru D. Iordan, Peder M. Isager, Ayse I. Isik, Andrew Jahn, Matthew R. Johnson, Tom Johnstone, Michael J. E. Joseph, Anthony C. Juliano, Joseph W. Kable, Michalis Kassinopoulos, Cemal Koba, Xiang-Zhen Kong, Timothy R. Koscik, Nuri Erkut Kucukboyaci, Brice A. Kuhl, Sebastian Kupek, Angela R. Laird, Claus Lamm, Robert Langner, Nina Lauharatanahirun, Hongmi Lee, Sangil Lee, Alexander Leemans, Andrea Leo, Elise Lesage, Flora Li, Monica Y. C. Li, Phui Cheng Lim, Evan N. Lintz, Schuyler W. Liphardt, Annabel B. Losecaat Vermeer, Bradley C. Love, Michael L. Mack, Norberto Malpica, Theo Marins, Camille Maumet, Kelsey McDonald, Joseph T. McGuire, Helena Melero, Adriana S. Mendez Leal, Benjamin Meyer, Kristin N. Meyer, Glad Mihai, Georgios D. Mitsis, Jorge Moll, Dylan M. Nielson, Gustav Nilsonne, Michael P. Notter, Emanuele Olivetti, Adrian I. Onicas, Paolo Papale, Kaustubh R. Patil, Jonathan E. Peelle, Alexandre Perez, Doris Pischedda, Jean-Baptiste Poline, Yanina Prystauka, Shruti Ray, Patricia A. Reuter-Lorenz, Richard C. Reynolds, Emiliano Ricciardi, Jenny R. Rieck, Anais M. Rodriguez-Thompson, Anthony Romyn, Taylor Salo, Gregory R. Samanez-Larkin, Emilio Sanz-Morales, Margaret L. Schlichting, Douglas H. Schultz, Qiang Shen, Margaret A. Sheridan, Jennifer A. Silvers, Kenny Skagerlund, Alec Smith, David V. Smith, Peter Sokol-Hessner, Simon R. Steinkamp, Sarah M. Tashjian, Bertrand Thirion, John N. Thorp, Gustav Tinghog, Loreen Tisdall, Steven H. Tompson, Claudio Toro-Serey, Juan Jesus Torre Tresols, Leonardo Tozzi, Vuong Truong, Luca Turella, Anna E. van ’t Veer, Tom Verguts, Jean M. Vettel, Sagana Vijayarajah, Khoi Vo, Matthew B. Wall, Wouter D. Weeda, Susanne Weis, David J. White, David Wisniewski, Alba Xifra-Porxas, Emily A. Yearling, Sangsuk Yoon, Rui Yuan, Kenneth S. L. Yuen, Lei Zhang, Xu Zhang, Joshua E. Zosky, Thomas E. Nichols, Russell A. Poldrack, and Tom Schonberg. 2020. “Variability in the Analysis of a Single Neuroimaging Dataset by Many Teams.” NATURE 582 (7810): 84–88. doi:10.1038/s41586-020-2314-9.
Vancouver
1.
Botvinik-Nezer R, Holzmeister F, Camerer CF, Dreber A, Huber J, Johannesson M, et al. Variability in the analysis of a single neuroimaging dataset by many teams. NATURE. 2020;582(7810):84–8.
IEEE
[1]
R. Botvinik-Nezer et al., “Variability in the analysis of a single neuroimaging dataset by many teams,” NATURE, vol. 582, no. 7810, pp. 84–88, 2020.
@article{8673623,
  abstract     = {Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses(1). The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset(2-5). Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed. The results obtained by seventy different teams analysing the same functional magnetic resonance imaging dataset show substantial variation, highlighting the influence of analytical choices and the importance of sharing workflows publicly and performing multiple analyses.},
  author       = {Botvinik-Nezer, Rotem and Holzmeister, Felix and Camerer, Colin F. and Dreber, Anna and Huber, Juergen and Johannesson, Magnus and Kirchler, Michael and Iwanir, Roni and Mumford, Jeanette A. and Adcock, R. Alison and Avesani, Paolo and Baczkowski, Blazej M. and Bajracharya, Aahana and Bakst, Leah and Ball, Sheryl and Barilari, Marco and Bault, Nadege and Beaton, Derek and Beitner, Julia and Benoit, Roland G. and Berkers, Ruud M. W. J. and Bhanji, Jamil P. and Biswal, Bharat B. and Bobadilla-Suarez, Sebastian and Bortolini, Tiago and Bottenhorn, Katherine L. and Bowring, Alexander and Braem, Senne and Brooks, Hayley R. and Brudner, Emily G. and Buc Calderon, Cristian and Camilleri, Julia A. and Castrellon, Jaime J. and Cecchetti, Luca and Cieslik, Edna C. and Cole, Zachary J. and Collignon, Olivier and Cox, Robert W. and Cunningham, William A. and Czoschke, Stefan and Dadi, Kamalaker and Davis, Charles P. and Luca, Alberto De and Delgado, Mauricio R. and Demetriou, Lysia and Dennison, Jeffrey B. and Di, Xin and Dickie, Erin W. and Dobryakova, Ekaterina and Donnat, Claire L. and Dukart, Juergen and Duncan, Niall W. and Durnez, Joke and Eed, Amr and Eickhoff, Simon B. and Erhart, Andrew and Fontanesi, Laura and Fricke, G. Matthew and Fu, Shiguang and Galvan, Adriana and Gau, Remi and Genon, Sarah and Glatard, Tristan and Glerean, Enrico and Goeman, Jelle J. and Golowin, Sergej A. E. and Gonzalez-Garcia, Carlos and Gorgolewski, Krzysztof J. and Grady, Cheryl L. and Green, Mikella A. and Guassi Moreira, Joao F. and Guest, Olivia and Hakimi, Shabnam and Hamilton, J. Paul and Hancock, Roeland and Handjaras, Giacomo and Harry, Bronson B. and Hawco, Colin and Herholz, Peer and Herman, Gabrielle and Heunis, Stephan and Hoffstaedter, Felix and Hogeveen, Jeremy and Holmes, Susan and Hu, Chuan-Peng and Huettel, Scott A. and Hughes, Matthew E. and Iacovella, Vittorio and Iordan, Alexandru D. and Isager, Peder M. and Isik, Ayse I. and Jahn, Andrew and Johnson, Matthew R. and Johnstone, Tom and Joseph, Michael J. E. and Juliano, Anthony C. and Kable, Joseph W. and Kassinopoulos, Michalis and Koba, Cemal and Kong, Xiang-Zhen and Koscik, Timothy R. and Kucukboyaci, Nuri Erkut and Kuhl, Brice A. and Kupek, Sebastian and Laird, Angela R. and Lamm, Claus and Langner, Robert and Lauharatanahirun, Nina and Lee, Hongmi and Lee, Sangil and Leemans, Alexander and Leo, Andrea and Lesage, Elise and Li, Flora and Li, Monica Y. C. and Lim, Phui Cheng and Lintz, Evan N. and Liphardt, Schuyler W. and Losecaat Vermeer, Annabel B. and Love, Bradley C. and Mack, Michael L. and Malpica, Norberto and Marins, Theo and Maumet, Camille and McDonald, Kelsey and McGuire, Joseph T. and Melero, Helena and Mendez Leal, Adriana S. and Meyer, Benjamin and Meyer, Kristin N. and Mihai, Glad and Mitsis, Georgios D. and Moll, Jorge and Nielson, Dylan M. and Nilsonne, Gustav and Notter, Michael P. and Olivetti, Emanuele and Onicas, Adrian I. and Papale, Paolo and Patil, Kaustubh R. and Peelle, Jonathan E. and Perez, Alexandre and Pischedda, Doris and Poline, Jean-Baptiste and Prystauka, Yanina and Ray, Shruti and Reuter-Lorenz, Patricia A. and Reynolds, Richard C. and Ricciardi, Emiliano and Rieck, Jenny R. and Rodriguez-Thompson, Anais M. and Romyn, Anthony and Salo, Taylor and Samanez-Larkin, Gregory R. and Sanz-Morales, Emilio and Schlichting, Margaret L. and Schultz, Douglas H. and Shen, Qiang and Sheridan, Margaret A. and Silvers, Jennifer A. and Skagerlund, Kenny and Smith, Alec and Smith, David V. and Sokol-Hessner, Peter and Steinkamp, Simon R. and Tashjian, Sarah M. and Thirion, Bertrand and Thorp, John N. and Tinghog, Gustav and Tisdall, Loreen and Tompson, Steven H. and Toro-Serey, Claudio and Torre Tresols, Juan Jesus and Tozzi, Leonardo and Truong, Vuong and Turella, Luca and van 't Veer, Anna E. and Verguts, Tom and Vettel, Jean M. and Vijayarajah, Sagana and Vo, Khoi and Wall, Matthew B. and Weeda, Wouter D. and Weis, Susanne and White, David J. and Wisniewski, David and Xifra-Porxas, Alba and Yearling, Emily A. and Yoon, Sangsuk and Yuan, Rui and Yuen, Kenneth S. L. and Zhang, Lei and Zhang, Xu and Zosky, Joshua E. and Nichols, Thomas E. and Poldrack, Russell A. and Schonberg, Tom},
  issn         = {0028-0836},
  journal      = {NATURE},
  keywords     = {PREDICTION MARKETS,NEURAL BASIS,REPLICABILITY},
  language     = {eng},
  number       = {7810},
  pages        = {84--88},
  title        = {Variability in the analysis of a single neuroimaging dataset by many teams},
  url          = {http://dx.doi.org/10.1038/s41586-020-2314-9},
  volume       = {582},
  year         = {2020},
}

Altmetric
View in Altmetric
Web of Science
Times cited: