Konkol Markus, Nüst Daniel, Goulier Laura
Institute for Geoinformatics, University of Münster, Münster, Germany.
Res Integr Peer Rev. 2020 Jul 14;5:10. doi: 10.1186/s41073-020-00095-y. eCollection 2020.
The trend toward open science increases the pressure on authors to provide access to the source code and data they used to compute the results reported in their scientific papers. Since sharing materials reproducibly is challenging, several projects have developed solutions to support the release of executable analyses alongside articles.
We reviewed 11 applications that can assist researchers in adhering to reproducibility principles. The applications were found through a literature search and interactions with the reproducible research community. An application was included in our analysis if it was actively maintained at the time the data for this paper was collected, supports the publication of executable code and data, is connected to the scholarly publication process. By investigating the software documentation and published articles, we compared the applications across 19 criteria, such as deployment options and features that support authors in creating and readers in studying executable papers.
From the 11 applications, eight allow publishers to self-host the system for free, whereas three provide paid services. Authors can submit an executable analysis using Jupyter Notebooks or R Markdown documents (10 applications support these formats). All approaches provide features to assist readers in studying the materials, e.g., one-click reproducible results or tools for manipulating the analysis parameters. Six applications allow for modifying materials after publication.
The applications support authors to publish reproducible research predominantly with literate programming. Concerning readers, most applications provide user interfaces to inspect and manipulate the computational analysis. The next step is to investigate the gaps identified in this review, such as the costs publishers have to expect when hosting an application, the consideration of sensitive data, and impacts on the review process.
开放科学的趋势增加了作者提供其用于计算科学论文中所报告结果的源代码和数据访问权限的压力。由于可重复地共享材料具有挑战性，一些项目已经开发出解决方案来支持在发表文章的同时发布可执行分析。
我们审查了11个可协助研究人员遵循可重复性原则的应用程序。这些应用程序是通过文献检索以及与可重复研究社区的互动找到的。如果一个应用程序在收集本文数据时仍在积极维护、支持可执行代码和数据的发布并且与学术出版过程相关联，那么它就被纳入我们的分析。通过研究软件文档和已发表的文章，我们在19个标准上对这些应用程序进行了比较，例如部署选项以及支持作者创建和读者研究可执行论文的功能。
在这11个应用程序中，有8个允许出版商免费自行托管系统，而3个提供付费服务。作者可以使用Jupyter Notebook或R Markdown文档提交可执行分析（10个应用程序支持这些格式）。所有方法都提供了帮助读者研究材料的功能，例如一键重现结果或用于操作分析参数的工具。6个应用程序允许在发表后修改材料。
这些应用程序主要通过文学编程支持作者发表可重复的研究。对于读者而言，大多数应用程序提供了用于检查和操作计算分析的用户界面。下一步是研究本次审查中发现的差距，例如出版商托管应用程序时可能预期的成本、对敏感数据的考虑以及对评审过程的影响。
