Alpers Rieke, Kühne Lisa, Truong Hong-Phuc, Zeeb Hajo, Westphal Max, Jäckle Sonja
Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany.
Department of Prevention and Evaluation, Leibniz Institute for Prevention Research and Epidemiology - BIPS, Bremen, Germany.
JMIR Form Res. 2023 Jun 27;7:e44549. doi: 10.2196/44549.
During the COVID-19 pandemic, local health authorities were responsible for managing and reporting current cases in Germany. Since March 2020, employees had to contain the spread of COVID-19 by monitoring and contacting infected persons as well as tracing their contacts. In the EsteR project, we implemented existing and newly developed statistical models as decision support tools to assist in the work of the local health authorities.
The main goal of this study was to validate the EsteR toolkit in two complementary ways: first, investigating the stability of the answers provided by our statistical tools regarding model parameters in the back end and, second, evaluating the usability and applicability of our web application in the front end by test users.
For model stability assessment, a sensitivity analysis was carried out for all 5 developed statistical models. The default parameters of our models as well as the test ranges of the model parameters were based on a previous literature review on COVID-19 properties. The obtained answers resulting from different parameters were compared using dissimilarity metrics and visualized using contour plots. In addition, the parameter ranges of general model stability were identified. For the usability evaluation of the web application, cognitive walk-throughs and focus group interviews were conducted with 6 containment scouts located at 2 different local health authorities. They were first asked to complete small tasks with the tools and then express their general impressions of the web application.
The simulation results showed that some statistical models were more sensitive to changes in their parameters than others. For each of the single-person use cases, we determined an area where the respective model could be rated as stable. In contrast, the results of the group use cases highly depended on the user inputs, and thus, no area of parameters with general model stability could be identified. We have also provided a detailed simulation report of the sensitivity analysis. In the user evaluation, the cognitive walk-throughs and focus group interviews revealed that the user interface needed to be simplified and more information was necessary as guidance. In general, the testers rated the web application as helpful, especially for new employees.
This evaluation study allowed us to refine the EsteR toolkit. Using the sensitivity analysis, we identified suitable model parameters and analyzed how stable the statistical models were in terms of changes in their parameters. Furthermore, the front end of the web application was improved with the results of the conducted cognitive walk-throughs and focus group interviews regarding its user-friendliness.
在新冠疫情期间，德国当地卫生当局负责管理和报告当前病例。自2020年3月起，工作人员必须通过监测和联系感染者以及追踪其接触者来遏制新冠病毒的传播。在EsteR项目中，我们实施了现有的和新开发的统计模型作为决策支持工具，以协助当地卫生当局开展工作。
本研究的主要目标是以两种互补方式验证EsteR工具包：第一，调查我们的统计工具在后端提供的关于模型参数的答案的稳定性；第二，由测试用户评估我们的网络应用程序在前端的可用性和适用性。
为进行模型稳定性评估，对所开发的全部5个统计模型进行了敏感性分析。我们模型的默认参数以及模型参数的测试范围基于先前关于新冠病毒特性的文献综述。使用差异度量比较不同参数得出的答案，并使用等高线图进行可视化。此外，确定了一般模型稳定性的参数范围。对于网络应用程序的可用性评估，对位于2个不同当地卫生当局的6名遏制行动侦察员进行了认知走查和焦点小组访谈。首先要求他们使用工具完成小任务，然后表达他们对网络应用程序的总体印象。
模拟结果表明，一些统计模型对其参数变化比其他模型更敏感。对于每个单人用例，我们确定了一个区域，在该区域内相应模型可被评为稳定。相比之下，群组用例的结果高度依赖用户输入，因此，无法确定具有一般模型稳定性的参数区域。我们还提供了敏感性分析的详细模拟报告。在用户评估中，认知走查和焦点小组访谈表明，用户界面需要简化，并且需要更多信息作为指导。总体而言，测试人员认为该网络应用程序很有帮助，尤其是对新员工。
这项评估研究使我们能够完善EsteR工具包。通过敏感性分析，我们确定了合适的模型参数，并分析了统计模型在参数变化方面的稳定性。此外，根据所进行的认知走查和焦点小组访谈关于其用户友好性的结果，对网络应用程序的前端进行了改进。
