Consensus-based guidance for conducting and reporting multi-analyst studies

Balazs Aczel; Barnabas Szaszi; Gustav Nilsonne; Olmo R van den Akker; Casper J Albers; Marcel Alm van Assen; Jojanneke A Bastiaansen; Daniel Benjamin; Udo Boehm; Rotem Botvinik-Nezer; Laura F Bringmann; Niko A Busch; Emmanuel Caruyer; Andrea M Cataldo; Nelson Cowan; Andrew Delios; Noah N N van Dongen; Chris Donkin; Johnny B van Doorn; Ann Dreber; Gilles Dutilh; Gary F Egan; Morton Ann Gernsbacher; Rink Hoekstra; Sabine Hoffmann; Felix Holzmeister; Juergen Huber; Magnus Johannesson; Kai J Jonas; Alexander T Kindel; Michael Kirchler; Yoram K Kunkels; D Stephen Lindsay; Jean-Francois Mangin; Dora Matzke; Marcus R Munafò; Ben R Newell; Brian A Nosek; Russell A Poldrack; Don van Ravenzwaaij; Jörg Rieskamp; Matthew J Salganik; Alexandra Sarafoglou; Tom Schonberg; Martin Schweinsberg; David Shanks; Raphael Silberzahn; Daniel J Simons; Barbara A Spellman; Samuel St-Jean; Jeffrey J Starns; Eric Luis Uhlmann; Jelte Wicherts; Eric-Jan Wagenmakers

doi:10.7554/eLife.72185

Consensus-based guidance for conducting and reporting multi-analyst studies

Balazs Aczel, Barnabas Szaszi, Gustav Nilsonne, Olmo R van den Akker, Casper J Albers, Marcel Alm van Assen, Jojanneke A Bastiaansen, Daniel Benjamin, Udo Boehm, Rotem Botvinik-Nezer, Laura F Bringmann, Niko A Busch, Emmanuel Caruyer, Andrea M Cataldo, Nelson Cowan, Andrew Delios, Noah N N van Dongen, Chris Donkin, Johnny B van Doorn, Ann DreberGilles Dutilh, Gary F Egan, Morton Ann Gernsbacher, Rink Hoekstra, Sabine Hoffmann, Felix Holzmeister, Juergen Huber, Magnus Johannesson, Kai J Jonas, Alexander T Kindel, Michael Kirchler, Yoram K Kunkels, D Stephen Lindsay, Jean-Francois Mangin, Dora Matzke, Marcus R Munafò, Ben R Newell, Brian A Nosek, Russell A Poldrack, Don van Ravenzwaaij, Jörg Rieskamp, Matthew J Salganik, Alexandra Sarafoglou, Tom Schonberg, Martin Schweinsberg, David Shanks, Raphael Silberzahn, Daniel J Simons, Barbara A Spellman, Samuel St-Jean, Jeffrey J Starns, Eric Luis Uhlmann, Jelte Wicherts, Eric-Jan Wagenmakers

Research output: Contribution to journal › Article (Academic Journal) › peer-review

30 Citations (Scopus)

57 Downloads (Pure)

Abstract

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

Original language	English
Article number	e72185
Journal	eLife
Volume	10
Early online date	9 Nov 2021
DOIs	https://doi.org/10.7554/eLife.72185
Publication status	Published - 9 Nov 2021

Bibliographical note

Funding Information:
VENI fellowship grant

Publisher Copyright:
© Aczel et al.

Access to Document

10.7554/eLife.72185Licence: CC BY

Full-text PDF (final published version)
This is the final published version of the article (version of record). It first appeared online via eLife at https://doi.org/10.7554/eLife.72185 . Please refer to any applicable terms of use of the publisher.
Final published version, 468 KBLicence: CC BY

Handle.net

Persistent link

Cite this

Aczel, B., Szaszi, B., Nilsonne, G., van den Akker, O. R., Albers, C. J., van Assen, M. A., Bastiaansen, J. A., Benjamin, D., Boehm, U., Botvinik-Nezer, R., Bringmann, L. F., Busch, N. A., Caruyer, E., Cataldo, A. M., Cowan, N., Delios, A., van Dongen, N. N. N., Donkin, C., van Doorn, J. B., ... Wagenmakers, E.-J. (2021). Consensus-based guidance for conducting and reporting multi-analyst studies. eLife, 10, Article e72185. https://doi.org/10.7554/eLife.72185

@article{372132ba0af249b6b7a77f5ea42166c6,

title = "Consensus-based guidance for conducting and reporting multi-analyst studies",

abstract = "Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.",

author = "Balazs Aczel and Barnabas Szaszi and Gustav Nilsonne and {van den Akker}, {Olmo R} and Albers, {Casper J} and {van Assen}, {Marcel Alm} and Bastiaansen, {Jojanneke A} and Daniel Benjamin and Udo Boehm and Rotem Botvinik-Nezer and Bringmann, {Laura F} and Busch, {Niko A} and Emmanuel Caruyer and Cataldo, {Andrea M} and Nelson Cowan and Andrew Delios and {van Dongen}, {Noah N N} and Chris Donkin and {van Doorn}, {Johnny B} and Ann Dreber and Gilles Dutilh and Egan, {Gary F} and Gernsbacher, {Morton Ann} and Rink Hoekstra and Sabine Hoffmann and Felix Holzmeister and Juergen Huber and Magnus Johannesson and Jonas, {Kai J} and Kindel, {Alexander T} and Michael Kirchler and Kunkels, {Yoram K} and Lindsay, {D Stephen} and Jean-Francois Mangin and Dora Matzke and Munaf{\`o}, {Marcus R} and Newell, {Ben R} and Nosek, {Brian A} and Poldrack, {Russell A} and {van Ravenzwaaij}, Don and J{\"o}rg Rieskamp and Salganik, {Matthew J} and Alexandra Sarafoglou and Tom Schonberg and Martin Schweinsberg and David Shanks and Raphael Silberzahn and Simons, {Daniel J} and Spellman, {Barbara A} and Samuel St-Jean and Starns, {Jeffrey J} and Uhlmann, {Eric Luis} and Jelte Wicherts and Eric-Jan Wagenmakers",

note = "Funding Information: VENI fellowship grant Publisher Copyright: {\textcopyright} Aczel et al.",

year = "2021",

month = nov,

day = "9",

doi = "10.7554/eLife.72185",

language = "English",

volume = "10",

journal = "eLife",

issn = "2050-084X",

publisher = "eLife Sciences Publications",

}

Aczel, B, Szaszi, B, Nilsonne, G, van den Akker, OR, Albers, CJ, van Assen, MA, Bastiaansen, JA, Benjamin, D, Boehm, U, Botvinik-Nezer, R, Bringmann, LF, Busch, NA, Caruyer, E, Cataldo, AM, Cowan, N, Delios, A, van Dongen, NNN, Donkin, C, van Doorn, JB, Dreber, A, Dutilh, G, Egan, GF, Gernsbacher, MA, Hoekstra, R, Hoffmann, S, Holzmeister, F, Huber, J, Johannesson, M, Jonas, KJ, Kindel, AT, Kirchler, M, Kunkels, YK, Lindsay, DS, Mangin, J-F, Matzke, D, Munafò, MR, Newell, BR, Nosek, BA, Poldrack, RA, van Ravenzwaaij, D, Rieskamp, J, Salganik, MJ, Sarafoglou, A, Schonberg, T, Schweinsberg, M, Shanks, D, Silberzahn, R, Simons, DJ, Spellman, BA, St-Jean, S, Starns, JJ, Uhlmann, EL, Wicherts, J & Wagenmakers, E-J 2021, 'Consensus-based guidance for conducting and reporting multi-analyst studies', eLife, vol. 10, e72185. https://doi.org/10.7554/eLife.72185

TY - JOUR

T1 - Consensus-based guidance for conducting and reporting multi-analyst studies

AU - Aczel, Balazs

AU - Szaszi, Barnabas

AU - Nilsonne, Gustav

AU - van den Akker, Olmo R

AU - Albers, Casper J

AU - van Assen, Marcel Alm

AU - Bastiaansen, Jojanneke A

AU - Benjamin, Daniel

AU - Boehm, Udo

AU - Botvinik-Nezer, Rotem

AU - Bringmann, Laura F

AU - Busch, Niko A

AU - Caruyer, Emmanuel

AU - Cataldo, Andrea M

AU - Cowan, Nelson

AU - Delios, Andrew

AU - van Dongen, Noah N N

AU - Donkin, Chris

AU - van Doorn, Johnny B

AU - Dreber, Ann

AU - Dutilh, Gilles

AU - Egan, Gary F

AU - Gernsbacher, Morton Ann

AU - Hoekstra, Rink

AU - Hoffmann, Sabine

AU - Holzmeister, Felix

AU - Huber, Juergen

AU - Johannesson, Magnus

AU - Jonas, Kai J

AU - Kindel, Alexander T

AU - Kirchler, Michael

AU - Kunkels, Yoram K

AU - Lindsay, D Stephen

AU - Mangin, Jean-Francois

AU - Matzke, Dora

AU - Munafò, Marcus R

AU - Newell, Ben R

AU - Nosek, Brian A

AU - Poldrack, Russell A

AU - van Ravenzwaaij, Don

AU - Rieskamp, Jörg

AU - Salganik, Matthew J

AU - Sarafoglou, Alexandra

AU - Schonberg, Tom

AU - Schweinsberg, Martin

AU - Shanks, David

AU - Silberzahn, Raphael

AU - Simons, Daniel J

AU - Spellman, Barbara A

AU - St-Jean, Samuel

AU - Starns, Jeffrey J

AU - Uhlmann, Eric Luis

AU - Wicherts, Jelte

AU - Wagenmakers, Eric-Jan

PY - 2021/11/9

Y1 - 2021/11/9

N2 - Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

AB - Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

U2 - 10.7554/eLife.72185

DO - 10.7554/eLife.72185

M3 - Article (Academic Journal)

C2 - 34751133

SN - 2050-084X

VL - 10

JO - eLife

JF - eLife

M1 - e72185

ER -

Consensus-based guidance for conducting and reporting multi-analyst studies

Abstract

Bibliographical note

Access to Document

Handle.net

Fingerprint

Cite this