Kopka Marvin, Feufel Markus A
Division of Ergonomics, Department of Psychology and Ergonomics (IPA), Technische Universität Berlin, Berlin, Germany.
Front Digit Health. 2024 Oct 21;6:1411924. doi: 10.3389/fdgth.2024.1411924. eCollection 2024.
Digital health research often relies on case vignettes (descriptions of fictitious or real patients) to navigate ethical and practical challenges. Despite their utility, the quality and lack of standardization of these vignettes has often been criticized, especially in studies on symptom-assessment applications (SAAs) and self-triage decision-making. To address this, our paper introduces a method to refine an existing set of vignettes, drawing on principles from classical test theory. First, we removed any vignette with an item difficulty of zero and an item-total correlation below zero. Second, we stratified the remaining vignettes to reflect the natural base rates of symptoms that SAAs are typically approached with, selecting those vignettes with the highest item-total correlation in each quota. Although this two-step procedure reduced the size of the original vignette set by 40%, comparing self-triage performance on the reduced and the original vignette sets, we found a strong correlation ( = 0.747 to = 0.997, < .001). This indicates that using our refinement method helps identifying vignettes with high predictive power of an agent's self-triage performance while simultaneously increasing cost-efficiency of vignette-based evaluation studies. This might ultimately lead to higher research quality and more reliable results.
数字健康研究通常依靠病例 vignettes（虚构或真实患者的描述）来应对伦理和实际挑战。尽管它们很有用，但这些 vignettes 的质量和缺乏标准化常常受到批评，尤其是在症状评估应用（SAA）和自我分诊决策的研究中。为了解决这个问题，我们的论文引入了一种方法，利用经典测试理论的原则来完善现有的一组 vignettes。首先，我们删除了任何项目难度为零且项目总分相关性低于零的 vignette。其次，我们对其余的 vignettes 进行分层，以反映 SAA 通常处理的症状的自然基础率，在每个配额中选择项目总分相关性最高的那些 vignettes。尽管这个两步程序使原始 vignette 集的大小减少了 40%，但比较减少后的和原始 vignette 集上的自我分诊性能，我们发现了很强的相关性（= 0.747 至 = 0.997，< 0.001）。这表明使用我们的完善方法有助于识别对个体自我分诊性能具有高预测力的 vignettes，同时提高基于 vignette 的评估研究的成本效益。这最终可能导致更高的研究质量和更可靠的结果。
