Advanced search
1 file | 1.30 MB Add to list
Author
Organization
Abstract
Data analysis workflows in many scientific domains have become increasingly complex and flexible. To assess the impact of this flexibility on functional magnetic resonance imaging (fMRI) results, the same dataset was independently analyzed by 70 teams, testing nine ex-ante hypotheses. The flexibility of analytic approaches is exemplified by the fact that no two teams chose identical workflows to analyze the data. This flexibility resulted in sizeable variation in hypothesis test results, even for teams whose statistical maps were highly correlated at intermediate stages of their analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Importantly, meta-analytic approaches that aggregated information across teams yielded significant consensus in activated regions across teams. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that analytic flexibility can have substantial effects on scientific conclusions, and demonstrate factors related to variability in fMRI. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for multiple analyses of the same data. Potential approaches to mitigate issues related to analytical variability are discussed.

Downloads

  • 843193v1.full.pdf
    • full text (Author's original)
    • |
    • open access
    • |
    • PDF
    • |
    • 1.30 MB

Citation

Please use this url to cite or link to this publication:

MLA
Botvinik-Nezer, Rotem, et al. “Variability in the Analysis of a Single Neuroimaging Dataset by Many Teams.” BioRxiv, 2019.
APA
Botvinik-Nezer, R., Holzmeister, F., Camerer, C. F., Dreber, A., Huber, J., Johannesson, M., … Schonberg, T. (2019). Variability in the analysis of a single neuroimaging dataset by many teams. bioRxiv.
Chicago author-date
Botvinik-Nezer, Rotem, Felix Holzmeister, Colin F. Camerer, Anna Dreber, Juergen Huber, Magnus Johannesson, Michael Kirchler, et al. 2019. “Variability in the Analysis of a Single Neuroimaging Dataset by Many Teams.” BioRxiv.
Chicago author-date (all authors)
Botvinik-Nezer, Rotem, Felix Holzmeister, Colin F. Camerer, Anna Dreber, Juergen Huber, Magnus Johannesson, Michael Kirchler, Roni Iwanir, Jeanette A. Mumford, Alison Adcock, Paolo Avesani, Blazej Baczkowski, Aahana Bajracharya, Leah Bakst, Sheryl Ball, Marco Barilari, Nadège Bault, Derek Beaton, Julia Beitner, Roland Benoit, Ruud Berkers, Jamil Bhanji, Bharat Biswal, Sebastian Bobadilla-Suarez, Tiago Bortolini, Katherine Bottenhorn, Alexander Bowring, Senne Braem, Hayley Brooks, Emily Brudner, Cristian Buc Calderon, Julia Camilleri, Jaime Castrellon, Luca Cecchetti, Edna Cieslik, Zachary Cole, Olivier Collignon, Robert Cox, William Cunningham, Stefan Czoschke, Kamalaker Dadi, Charles Davis, Alberto De Luca, Mauricio Delgado, Lysia Demetriou, Jeffrey Dennison, Xin Di, Erin Dickie, Ekaterina Dobryakova, Claire Donnat, Juergen Dukart, Niall W. Duncan, Joke Durnez, Amr Eed, Simon Eickhoff, Andrew Erhart, Laura Fontanesi, G. Matthew Fricke, Adriana Galvan, Remi Gau, Sarah Genon, Tristan Glatard, Enrico Glerean, Jelle Goeman, Sergej Golowin, Carlos Gonzalez-Garcia, Krzysztof Gorgolewski, Cheryl Grady, Mikella Green, João Guassi Moreira, Olivia Guest, Shabnam Hakimi, J. Paul Hamilton, Roeland Hancock, Giacomo Handjaras, Bronson Harry, Colin Hawco, Peer Herholz, Gabrielle Herman, Stephan Heunis, Felix Hoffstaedter, Jeremy Hogeveen, Susan Holmes, Chuan-Peng Hu, Scott Huettel, Matthew Hughes, Vittorio Iacovella, Alexandru Iordan, Peder Isager, Ayse Ilkay Isik, Andrew Jahn, Matthew Johnson, Tom Johnstone, Michael Joseph, Anthony Juliano, Joseph Kable, Michalis Kassinopoulos, Cemal Koba, Xiang-Zhen Kong, Timothy Koscik, Nuri Erkut Kucukboyaci, Brice Kuhl, Sebastian Kupek, Angela Laird, Claus Lamm, Robert Langner, Nina Lauharatanahirun, Hongmi Lee, Sangil Lee, Alexander Leemans, Andrea Leo, Elise Lesage, Flora Li, Monica Li, Phui Cheng Lim, Evan Lintz, Schuyler Liphardt, Annabel Losecaat Vermeer, Bradley Love, Michael Mack, Norberto Malpica, Theo Marins, Camille Maumet, Kelsey McDonald, Joseph McGuire, Helena Melero, Adriana Méndez Leal, Benjamin Meyer, Kristin Meyer, Paul Mihai, Georgios Mitsis, Jorge Moll, Dylan Nielson, Gustav Nilsonne, Michael Notter, Emanuele Olivetti, Adrian Onicas, Paolo Papale, Kaustubh Patil, Jonathan E. Peelle, Alexandre Pérez, Doris Pischedda, Jean-Baptiste Poline, Yanina Prystauka, Shruti Ray, Patricia Reuter-Lorenz, Richard Reynolds, Emiliano Ricciardi, Jenny Rieck, Anais Rodriguez-Thompson, Anthony Romyn, Taylor Salo, Gregory Samanez-Larkin, Emilio Sanz-Morales, Margaret Schlichting, Douglas Schultz, Qiang Shen, Margaret Sheridan, Fu Shiguang, Jennifer Silvers, Kenny Skagerlund, Alec Smith, David Smith, Peter Sokol-Hessner, Simon Steinkamp, Sarah Tashjian, Bertrand Thirion, John Thorp, Gustav Tinghög, Loreen Tisdall, Steven Tompson, Claudio Toro-Serey, Juan Torre, Leonardo Tozzi, Vuong Truong, Luca Turella, Anna E. van’t Veer, Tom Verguts, Jean Vettel, Sagana Vijayarajah, Khoi Vo, Matthew Wall, Wouter D. Weeda, Susanne Weis, David White, David Wisniewski, Alba Xifra-Porxas, Emily Yearling, Sangsuk Yoon, Rui Yuan, Kenneth Yuen, Lei Zhang, Xu Zhang, Joshua Zosky, Thomas E. Nichols, Russell A. Poldrack, and Tom Schonberg. 2019. “Variability in the Analysis of a Single Neuroimaging Dataset by Many Teams.” BioRxiv.
Vancouver
1.
Botvinik-Nezer R, Holzmeister F, Camerer CF, Dreber A, Huber J, Johannesson M, et al. Variability in the analysis of a single neuroimaging dataset by many teams. bioRxiv. 2019.
IEEE
[1]
R. Botvinik-Nezer et al., “Variability in the analysis of a single neuroimaging dataset by many teams,” bioRxiv. 2019.
@misc{8641719,
  abstract     = {{Data analysis workflows in many scientific domains have become increasingly complex and flexible. To assess the impact of this flexibility on functional magnetic resonance imaging (fMRI) results, the same dataset was independently analyzed by 70 teams, testing nine ex-ante hypotheses. The flexibility of analytic approaches is exemplified by the fact that no two teams chose identical workflows to analyze the data. This flexibility resulted in sizeable variation in hypothesis test results, even for teams whose statistical maps were highly correlated at intermediate stages of their analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Importantly, meta-analytic approaches that aggregated information across teams yielded significant consensus in activated regions across teams. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that analytic flexibility can have substantial effects on scientific conclusions, and demonstrate factors related to variability in fMRI. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for multiple analyses of the same data. Potential approaches to mitigate issues related to analytical variability are discussed.}},
  author       = {{Botvinik-Nezer, Rotem and Holzmeister, Felix and Camerer, Colin F. and Dreber, Anna and Huber, Juergen and Johannesson, Magnus and Kirchler, Michael and Iwanir, Roni and Mumford, Jeanette A. and Adcock, Alison and Avesani, Paolo and Baczkowski, Blazej and Bajracharya, Aahana and Bakst, Leah and Ball, Sheryl and Barilari, Marco and Bault, Nadège and Beaton, Derek and Beitner, Julia and Benoit, Roland and Berkers, Ruud and Bhanji, Jamil and Biswal, Bharat and Bobadilla-Suarez, Sebastian and Bortolini, Tiago and Bottenhorn, Katherine and Bowring, Alexander and Braem, Senne and Brooks, Hayley and Brudner, Emily and Buc Calderon, Cristian and Camilleri, Julia and Castrellon, Jaime and Cecchetti, Luca and Cieslik, Edna and Cole, Zachary and Collignon, Olivier and Cox, Robert and Cunningham, William and Czoschke, Stefan and Dadi, Kamalaker and Davis, Charles and De Luca, Alberto and Delgado, Mauricio and Demetriou, Lysia and Dennison, Jeffrey and Di, Xin and Dickie, Erin and Dobryakova, Ekaterina and Donnat, Claire and Dukart, Juergen and Duncan, Niall W. and Durnez, Joke and Eed, Amr and Eickhoff, Simon and Erhart, Andrew and Fontanesi, Laura and Fricke, G. Matthew and Galvan, Adriana and Gau, Remi and Genon, Sarah and Glatard, Tristan and Glerean, Enrico and Goeman, Jelle and Golowin, Sergej and Gonzalez-Garcia, Carlos and Gorgolewski, Krzysztof and Grady, Cheryl and Green, Mikella and Guassi Moreira, João and Guest, Olivia and Hakimi, Shabnam and Hamilton, J. Paul and Hancock, Roeland and Handjaras, Giacomo and Harry, Bronson and Hawco, Colin and Herholz, Peer and Herman, Gabrielle and Heunis, Stephan and Hoffstaedter, Felix and Hogeveen, Jeremy and Holmes, Susan and Hu, Chuan-Peng and Huettel, Scott and Hughes, Matthew and Iacovella, Vittorio and Iordan, Alexandru and Isager, Peder and Isik, Ayse Ilkay and Jahn, Andrew and Johnson, Matthew and Johnstone, Tom and Joseph, Michael and Juliano, Anthony and Kable, Joseph and Kassinopoulos, Michalis and Koba, Cemal and Kong, Xiang-Zhen and Koscik, Timothy and Kucukboyaci, Nuri Erkut and Kuhl, Brice and Kupek, Sebastian and Laird, Angela and Lamm, Claus and Langner, Robert and Lauharatanahirun, Nina and Lee, Hongmi and Lee, Sangil and Leemans, Alexander and Leo, Andrea and Lesage, Elise and Li, Flora and Li, Monica and Lim, Phui Cheng and Lintz, Evan and Liphardt, Schuyler and Losecaat Vermeer, Annabel and Love, Bradley and Mack, Michael and Malpica, Norberto and Marins, Theo and Maumet, Camille and McDonald, Kelsey and McGuire, Joseph and Melero, Helena and Méndez Leal, Adriana and Meyer, Benjamin and Meyer, Kristin and Mihai, Paul and Mitsis, Georgios and Moll, Jorge and Nielson, Dylan and Nilsonne, Gustav and Notter, Michael and Olivetti, Emanuele and Onicas, Adrian and Papale, Paolo and Patil, Kaustubh and Peelle, Jonathan E. and Pérez, Alexandre and Pischedda, Doris and Poline, Jean-Baptiste and Prystauka, Yanina and Ray, Shruti and Reuter-Lorenz, Patricia and Reynolds, Richard and Ricciardi, Emiliano and Rieck, Jenny and Rodriguez-Thompson, Anais and Romyn, Anthony and Salo, Taylor and Samanez-Larkin, Gregory and Sanz-Morales, Emilio and Schlichting, Margaret and Schultz, Douglas and Shen, Qiang and Sheridan, Margaret and Shiguang, Fu and Silvers, Jennifer and Skagerlund, Kenny and Smith, Alec and Smith, David and Sokol-Hessner, Peter and Steinkamp, Simon and Tashjian, Sarah and Thirion, Bertrand and Thorp, John and Tinghög, Gustav and Tisdall, Loreen and Tompson, Steven and Toro-Serey, Claudio and Torre, Juan and Tozzi, Leonardo and Truong, Vuong and Turella, Luca and van’t Veer, Anna E. and Verguts, Tom and Vettel, Jean and Vijayarajah, Sagana and Vo, Khoi and Wall, Matthew and Weeda, Wouter D. and Weis, Susanne and White, David and Wisniewski, David and Xifra-Porxas, Alba and Yearling, Emily and Yoon, Sangsuk and Yuan, Rui and Yuen, Kenneth and Zhang, Lei and Zhang, Xu and Zosky, Joshua and Nichols, Thomas E. and Poldrack, Russell A. and Schonberg, Tom}},
  language     = {{eng}},
  pages        = {{31}},
  series       = {{bioRxiv}},
  title        = {{Variability in the analysis of a single neuroimaging dataset by many teams}},
  url          = {{https://www.biorxiv.org/content/10.1101/843193v1}},
  year         = {{2019}},
}