Abstract
Reproducibility is an essential feature of all scientific outcomes. Scientific evidence can only reach its true status as reliable if replicated, but the results of well-conducted replication studies face an uphill battle to be performed, and little attention and dedication have been put into publishing the results of replication attempts. Therefore, we asked a small cohort of researchers about their attempts to replicate results from other groups, as well as from their own laboratories, and their general perception of the issues concerning reproducibility in their field. We also asked how they perceive the venues, i.e. journals, to communicate and discuss the results of these attempts. To this aim we pre-registered and shared a questionnaire among scientists at diverse levels. The results indicate that, in general, replication attempts of their own protocols are quite successful (with over 80% reporting not or rarely having problems with their own protocols). Although the majority of respondents tried to replicate a study or experiment from other labs (75.4%), the median successful rate was scored at 3 (in a 1-5 scale), while the median for the general estimation of replication success in their field was found to be 5 (in a 1-10 scale). The majority of respondents (70.2%) also perceive journals as unwelcoming of replication studies.
References
Barba LA. Terminologies for Reproducible Research. arXiv arXiv:1802.03311 [Preprint]. 2018. OCLC: 1106287615.
Schmidt S. Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences. Review of General Psychology. 2009;13(2):90-100. doi: 10.1037/a0015108.
Nosek BA, Errington TM. What is replication?. PLOS Biology. 2020;18(3):e3000691. doi: 10.1371/journal.pbio.3000691.
Open Science Collaboration. Estimating the reproducibility of psychological science. Science. 2015;349(6251):aac4716. doi: 10.1126/science.aac4716.
Kaiser J. Plan to replicate 50 high-impact cancer papers shrinks to just 18. Science. 2018;. doi: 10.1126/science.aau9619.
Errington TM, Iorns E, Gunn W, Tan FE, Lomax J, Nosek BA. An open investigation of the reproducibility of cancer biology research. eLife. 2014;3. doi: 10.7554/elife.04333.
Poldrack RA, Baker CI, Durnez J, Gorgolewski KJ, Matthews PM, Munafò MR et al. Scanning the horizon: towards transparent and reproducible neuroimaging research. Nature Reviews Neuroscience. 2017;18(2):115-126. doi: 10.1038/nrn.2016.167.
Border R, Johnson EC, Evans LM, Smolen A, Berley N, Sullivan PF et al. No Support for Historical Candidate Gene or Candidate Gene-by-Interaction Hypotheses for Major Depression Across Multiple Large Samples. American Journal of Psychiatry. 2019;176(5):376-387. doi: 10.1176/appi.ajp.2018.18070881.
Yeung AWK. Do Neuroscience Journals Accept Replications? A Survey of Literature. Frontiers in Human Neuroscience. 2017;11. doi: 10.3389/fnhum.2017.00468.
Martin GN, Clarke RM. Are Psychology Journals Anti-replication? A Snapshot of Editorial Practices. Frontiers in Psychology. 2017;8. doi: 10.3389/fpsyg.2017.00523.
Collins K, Shiffman D, Rock J. How Are Scientists Using Social Media in the Workplace?. PLOS ONE. 2016;11(10):e0162680. doi: 10.1371/journal.pone.0162680.
Wallach JD, Boyack KW, Ioannidis JPA. Reproducible research practices, transparency, and open access data in the biomedical literature, 2015–2017. PLOS Biology. 2018;16(11):e2006930. doi: 10.1371/journal.pbio.2006930.
Teytelman L, Stoliartchouk A, Kindler L, Hurwitz BL. Protocols.io: Virtual Communities for Protocol Development and Discussion. PLOS Biology. 2016;14(8):e1002538. doi: 10.1371/journal.pbio.1002538.
Institute of Medicine, Board on Health Sciences Policy, Committee on Strategies for Responsible Sharing of Clinical Trial Data. Sharing Clinical Trial Data: Maximizing Benefits, Minimizing Risk. Washington, DC: National Aacademies Press. doi: 10.17226/18998.
Savage CJ, Vickers AJ. Empirical Study of Data Sharing by Authors Publishing in PLoS Journals. PLoS ONE. 2009;4(9):e7078. doi: 10.1371/journal.pone.0007078.
Tenopir C, Rice NM, Allard S, Baird L, Borycz J, Christian L et al. Data sharing, management, use, and reuse: Practices and perceptions of scientists worldwide. PLOS ONE. 2020;15(3):e0229003. doi: 10.1371/journal.pone.0229003.
Longo DL, Drazen JM. Data Sharing. New England Journal of Medicine. 2016;374(3):276-277. doi: 10.1056/nejme1516564.
Puebla I. Preprints: a tool and a vehicle towards greater reproducibility in the life sciences. Journal for Reproducibility in Neuroscience. 2020;2. doi: 10.31885/jrn.2.2021.1465.
Berg JM, Bhalla N, Bourne PE, Chalfie M, Drubin DG, Fraser JS et al. Preprints for the life sciences. Science. 2016;352(6288):899-901. doi: 10.1126/science.aaf9133.
Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proceedings of the National Academy of Sciences. 2018;115(11):2600-2606. doi: 10.1073/pnas.1708274114.
Kiyonaga A, Scimeca JM. Practical Considerations for Navigating Registered Reports. Trends in Neurosciences. 2019;42(9):568-572. doi: 10.1016/j.tins.2019.07.003.
Center for Open Science. Registered Reports; [cited 9 May 2021]. Available from: https://www.cos.io/initiatives/registered-reports. Archived: https://web.archive.org/web/20210521015044/https://www.cos.io/initiatives/registered-reports.
Casarotto P, Brembs B. A platform for reproducibility. Journal for Reproducibility in Neuroscience. 2020;1. doi: 10.31885/jrn.1.2020.303.
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright (c) 2021 Giuliano Didio, Plinio Casarotto