Ioanas Horea-Ioan, Macdonald Austin, Halchenko Yaroslav O
Center for Open Neuroscience, Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, United States.
Front Neuroinform. 2024 Jul 22;18:1376022. doi: 10.3389/fninf.2024.1376022. eCollection 2024.
The value of research articles is increasingly contingent on complex data analysis results which substantiate their claims. Compared to data production, data analysis more readily lends itself to a higher standard of transparency and repeated operator-independent execution. This higher standard can be approached via fully reexecutable research outputs, which contain the entire instruction set for automatic end-to-end generation of an entire article from the earliest feasible provenance point. In this study, we make use of a peer-reviewed neuroimaging article which provides complete but fragile reexecution instructions, as a starting point to draft a new reexecution system which is both robust and portable. We render this system modular as a core design aspect, so that reexecutable article code, data, and environment specifications could potentially be substituted or adapted. In conjunction with this system, which forms the demonstrative product of this study, we detail the core challenges with full article reexecution and specify a number of best practices which permitted us to mitigate them. We further show how the capabilities of our system can subsequently be used to provide reproducibility assessments, both via simple statistical metrics and by visually highlighting divergent elements for human inspection. We argue that fully reexecutable articles are thus a feasible best practice, which can greatly enhance the understanding of data analysis variability and the trust in results. Lastly, we comment at length on the outlook for reexecutable research outputs and encourage re-use and derivation of the system produced herein.
研究文章的价值越来越取决于复杂的数据分析结果，这些结果证实了文章的观点。与数据生成相比，数据分析更容易达到更高的透明度标准，并且可以由独立于操作人员的方式重复执行。通过完全可重新执行的研究输出可以达到这一更高标准，这种输出包含从最早可行的来源点自动端到端生成整篇文章的完整指令集。在本研究中，我们以一篇经过同行评审的神经影像学文章为起点，该文章提供了完整但脆弱的重新执行指令，以此来起草一个既强大又便携的新重新执行系统。我们将此系统设计为模块化的核心方面，以便可重新执行的文章代码、数据和环境规范有可能被替换或调整。结合这个构成本研究示范产品的系统，我们详细阐述了全文重新执行的核心挑战，并指定了一些最佳实践，使我们能够减轻这些挑战。我们还展示了我们系统的功能随后如何用于提供可重复性评估，既通过简单的统计指标，也通过直观地突出不同元素以供人工检查。我们认为，完全可重新执行的文章因此是一种可行的最佳实践，它可以极大地增进对数据分析变异性的理解和对结果的信任。最后，我们详细评论了可重新执行研究输出的前景，并鼓励对本文所产生的系统进行再利用和衍生。
