Department of Information Technology, Analytics, and Operations, Mendoza College of Business, University of Notre Dame, Notre Dame, IN, US.
Department of Psychology and Human Development, Peabody College, Vanderbilt University, Nashville, TN, US.
Int J Med Inform. 2023 Apr;172:105006. doi: 10.1016/j.ijmedinf.2023.105006. Epub 2023 Feb 10.
Low health literacy is a concern among US Veterans. In this study, we evaluated NoteAid, a system that provides lay definitions to medical jargon terms in EHR notes to help Veterans comprehend EHR notes. We expected that low initial scores for Veterans would be improved by using NoteAid.
We recruited Veterans from the Amazon Mechanical Turk crowd work platform (MTurk). We also recruited non-Veterans from MTurk as a control group for comparison. We randomly split recruited MTurk Veteran participants into control and intervention groups. We recruited non-Veteran participants into mutually exclusive control or intervention tasks on the MTurk platform. We showed participants de-identified EHR notes and asked them to answer comprehension questions related to the notes. We provided participants in the intervention group with EHR note content processed with NoteAid, while NoteAid was not available for participants in the control group.
We recruited 94 Veterans and 181 non-Veterans. NoteAid leads to a significant improvement for non-Veterans but not for Veterans. Comparing Veterans recruited via MTurk with non-Veterans recruited via MTurk, we found that without NoteAid, Veterans have significantly higher raw scores than non-Veterans. This difference is not significant with NoteAid.
That Veterans outperform a comparable population of non-Veterans is a surprising outcome. Without NoteAid, scores on the test are already high for Veterans, therefore, minimizing the ability of an intervention such as NoteAid to improve performance. With regards to Veterans, understanding the health literacy of Veterans has been an open question. We show here that Veterans score higher than a comparable, non-Veteran population.
Veterans on MTurk do not see improved scores when using NoteAid, but they already score high on the test, significantly higher than non-Veterans. When evaluating NoteAid, population specifics need to be considered, as performance may vary across groups. Future work investigating the effectiveness of NoteAid on improving comprehension with local Veterans and developing a more difficult test to assess groups with higher health literacy is needed.
低健康素养是美国退伍军人关注的问题。在这项研究中，我们评估了 NoteAid，这是一个系统，它为电子健康记录（EHR）中的医学术语提供通俗定义，以帮助退伍军人理解 EHR 记录。我们预计，使用 NoteAid 将改善退伍军人的初始低分数。
我们从亚马逊土耳其机器人（MTurk）的众包工作平台招募退伍军人。我们还从 MTurk 招募非退伍军人作为对照组进行比较。我们随机将招募的 MTurk 退伍军人参与者分为对照组和干预组。我们招募非退伍军人参与者在 MTurk 平台上进行相互排斥的对照组或干预任务。我们向参与者展示了去识别的 EHR 记录，并要求他们回答与记录相关的理解问题。我们为干预组的参与者提供了经过 NoteAid 处理的 EHR 记录内容，而对照组的参与者则无法使用 NoteAid。
我们招募了 94 名退伍军人和 181 名非退伍军人。NoteAid 可显著提高非退伍军人的成绩，但对退伍军人没有影响。与通过 MTurk 招募的退伍军人相比，我们发现，没有 NoteAid，退伍军人的原始分数明显高于非退伍军人。但使用 NoteAid 后，这种差异不再显著。
退伍军人的表现优于可比的非退伍军人群体，这是一个令人惊讶的结果。没有 NoteAid，退伍军人的测试分数已经很高，因此，干预措施（如 NoteAid）提高表现的能力最小化。关于退伍军人，了解退伍军人的健康素养一直是一个悬而未决的问题。我们在这里表明，退伍军人的得分高于可比的非退伍军人群体。
使用 NoteAid 时，MTurk 上的退伍军人的分数没有提高，但他们在测试中的得分已经很高，明显高于非退伍军人。在评估 NoteAid 时，需要考虑人口统计学因素，因为不同群体的表现可能会有所不同。需要进一步研究 NoteAid 在提高当地退伍军人理解能力方面的有效性，并开发更难的测试来评估具有更高健康素养的群体。
