Jing Xia, Cimino James J, Patel Vimla L, Zhou Yuchun, Shubrook Jay H, De Lacalle Sonsoles, Draghi Brooke N, Ernst Mytchell A, Weaver Aneesa, Sekar Shriram, Liu Chang
Department of Public Health Sciences, Clemson University, Clemson, SC.
Informatics Institute, School of Medicine, University of Alabama, Birmingham, Birmingham, AL.
medRxiv. 2023 Oct 31:2023.05.30.23290719. doi: 10.1101/2023.05.30.23290719.
To compare how clinical researchers generate data-driven hypotheses with a visual interactive analytic tool (VIADS, a visual interactive analysis tool for filtering and summarizing large data sets coded with hierarchical terminologies) or other tools.
We recruited clinical researchers and separated them into "experienced" and "inexperienced" groups. Participants were randomly assigned to a VIADS or control group within the groups. Each participant conducted a remote 2-hour study session for hypothesis generation with the same study facilitator on the same datasets by following a think-aloud protocol. Screen activities and audio were recorded, transcribed, coded, and analyzed. Hypotheses were evaluated by seven experts on their validity, significance, and feasibility. We conducted multilevel random effect modeling for statistical tests.
Eighteen participants generated 227 hypotheses, of which 147 (65%) were valid. The VIADS and control groups generated a similar number of hypotheses. The VIADS group took a significantly shorter time to generate one hypothesis (e.g., among inexperienced clinical researchers, 258 seconds versus 379 seconds, = 0.046, power = 0.437, ICC = 0.15). The VIADS group received significantly lower ratings than the control group on feasibility and the combination rating of validity, significance, and feasibility.
The role of VIADS in hypothesis generation seems inconclusive. The VIADS group took a significantly shorter time to generate each hypothesis. However, the combined validity, significance, and feasibility ratings of their hypotheses were significantly lower. Further characterization of hypotheses, including specifics on how they might be improved, could guide future tool development.
比较临床研究人员使用可视化交互式分析工具（VIADS，一种用于筛选和汇总用层次术语编码的大数据集的可视化交互式分析工具）或其他工具生成数据驱动假设的方式。
我们招募了临床研究人员，并将他们分为“经验丰富”和“经验不足”两组。参与者在组内被随机分配到VIADS组或对照组。每位参与者通过遵循出声思维协议，与同一位研究促进者就相同的数据集进行为期2小时的远程假设生成研究。记录屏幕活动和音频，进行转录、编码和分析。由七位专家对假设的有效性、重要性和可行性进行评估。我们进行了多层次随机效应建模以进行统计检验。
18名参与者生成了227个假设，其中147个（65%）有效。VIADS组和对照组生成的假设数量相似。VIADS组生成一个假设的时间明显更短（例如，在经验不足的临床研究人员中，分别为258秒和379秒，P = 0.046，检验效能 = 0.437，组内相关系数 = 0.15）。在可行性以及有效性、重要性和可行性的综合评分方面，VIADS组的得分显著低于对照组。
VIADS在假设生成中的作用似乎尚无定论。VIADS组生成每个假设的时间明显更短。然而，他们假设的有效性、重要性和可行性综合评分显著更低。对假设进行进一步的特征描述，包括如何改进它们的具体细节，可为未来工具的开发提供指导。
