Pokutnaya Darya, Van Panhuis Willem G, Childers Bruce, Hawkins Marquis S, Arcury-Quandt Alice E, Matlack Meghan, Carpio Kharlya, Hochheiser Harry
University of Pittsburgh, Department of Epidemiology; Pittsburgh, Pennsylvania, United States of America.
Office of Data Science and Emerging Technologies, National Institute of Allergy and Infectious Diseases; Rockville, Maryland, United States of America [note that Dr. Van Panhuis completed the research described in this paper during his time at the University of Pittsburgh, before starting his position at NIAID].
medRxiv. 2023 Mar 22:2023.03.21.23287529. doi: 10.1101/2023.03.21.23287529.
Infectious disease computational modeling studies have been widely published during the coronavirus disease 2019 (COVID-19) pandemic, yet they have limited reproducibility. Developed through an iterative testing process with multiple reviewers, the Infectious Disease Modeling Reproducibility Checklist (IDMRC) enumerates the minimal elements necessary to support reproducible infectious disease computational modeling publications. The primary objective of this study was to assess the reliability of the IDMRC and to identify which reproducibility elements were unreported in a sample of COVID-19 computational modeling publications.
Four reviewers used the IDMRC to assess 46 preprint and peer reviewed COVID-19 modeling studies published between March 13, 2020, and July 31, 2020. The inter-rater reliability was evaluated by mean percent agreement and Fleiss' kappa coefficients (κ). Papers were ranked based on the average number of reported reproducibility elements, and average proportion of papers that reported each checklist item were tabulated.
Questions related to the computational environment (mean κ = 0.90, range = 0.90-0.90), analytical software (mean κ = 0.74, range = 0.68-0.82), model description (mean κ = 0.71, range = 0.58-0.84), model implementation (mean κ = 0.68, range = 0.39-0.86), and experimental protocol (mean κ = 0.63, range = 0.58-0.69) had moderate or greater (κ > 0.41) inter-rater reliability. Questions related to data had the lowest values (mean κ = 0.37, range = 0.23-0.59). Reviewers ranked similar papers in the upper and lower quartiles based on the proportion of reproducibility elements each paper reported. While over 70% of the publications provided data used in their models, less than 30% provided the model implementation.
The IDMRC is the first comprehensive, quality-assessed tool for guiding researchers in reporting reproducible infectious disease computational modeling studies. The inter-rater reliability assessment found that most scores were characterized by moderate or greater agreement. These results suggests that the IDMRC might be used to provide reliable assessments of the potential for reproducibility of published infectious disease modeling publications. Results of this evaluation identified opportunities for improvement to the model implementation and data questions that can further improve the reliability of the checklist.
在2019冠状病毒病（COVID-19）大流行期间，传染病计算建模研究已广泛发表，但它们的可重复性有限。通过与多位审稿人进行的迭代测试过程开发的《传染病建模可重复性清单》（IDMRC）列举了支持可重复的传染病计算建模出版物所需的最少要素。本研究的主要目的是评估IDMRC的可靠性，并确定在COVID-19计算建模出版物样本中哪些可重复性要素未被报告。
四位审稿人使用IDMRC评估了2020年3月13日至2020年7月31日期间发表的46篇预印本和同行评审的COVID-19建模研究。通过平均百分比一致性和Fleiss卡方系数（κ）评估评分者间的可靠性。根据报告的可重复性要素的平均数量对论文进行排名，并列出报告每个清单项目的论文的平均比例。
与计算环境相关的问题（平均κ = 0.90，范围 = 0.90 - 0.90）、分析软件（平均κ = 0.74，范围 = 0.68 - 0.82）、模型描述（平均κ = 0.71，范围 = 0.58 - 0.84）、模型实施（平均κ = 0.68，范围 = 0.39 - 0.86）和实验方案（平均κ = 0.63，范围 = 0.58 - 0.69）具有中等或更高（κ > 0.41）的评分者间可靠性。与数据相关的问题得分最低（平均κ = 0.37，范围 = 0.23 - 0.59）。审稿人根据每篇论文报告的可重复性要素的比例将类似的论文排在上下四分位数中。虽然超过70%的出版物提供了其模型中使用的数据，但不到30%的出版物提供了模型实施情况。
IDMRC是第一个用于指导研究人员报告可重复的传染病计算建模研究的全面的、经过质量评估的工具。评分者间可靠性评估发现，大多数分数的特征是中等或更高的一致性。这些结果表明，IDMRC可用于对已发表的传染病建模出版物的可重复性潜力进行可靠评估。本次评估结果确定了模型实施和数据问题的改进机会，可进一步提高清单的可靠性。
