Department of Neurology, Massachusetts General Hospital, Boston, MA, United States.
Harvard Medical School, Boston, MA, United States.
J Med Internet Res. 2022 Aug 30;24(8):e40384. doi: 10.2196/40384.
Electronic health records (EHRs) with large sample sizes and rich information offer great potential for dementia research, but current methods of phenotyping cognitive status are not scalable.
The aim of this study was to evaluate whether natural language processing (NLP)-powered semiautomated annotation can improve the speed and interrater reliability of chart reviews for phenotyping cognitive status.
In this diagnostic study, we developed and evaluated a semiautomated NLP-powered annotation tool (NAT) to facilitate phenotyping of cognitive status. Clinical experts adjudicated the cognitive status of 627 patients at Mass General Brigham (MGB) health care, using NAT or traditional chart reviews. Patient charts contained EHR data from two data sets: (1) records from January 1, 2017, to December 31, 2018, for 100 Medicare beneficiaries from the MGB Accountable Care Organization and (2) records from 2 years prior to COVID-19 diagnosis to the date of COVID-19 diagnosis for 527 MGB patients. All EHR data from the relevant period were extracted; diagnosis codes, medications, and laboratory test values were processed and summarized; clinical notes were processed through an NLP pipeline; and a web tool was developed to present an integrated view of all data. Cognitive status was rated as cognitively normal, cognitively impaired, or undetermined. Assessment time and interrater agreement of NAT compared to manual chart reviews for cognitive status phenotyping was evaluated.
NAT adjudication provided higher interrater agreement (Cohen κ=0.89 vs κ=0.80) and significant speed up (time difference mean 1.4, SD 1.3 minutes; P<.001; ratio median 2.2, min-max 0.4-20) over manual chart reviews. There was moderate agreement with manual chart reviews (Cohen κ=0.67). In the cases that exhibited disagreement with manual chart reviews, NAT adjudication was able to produce assessments that had broader clinical consensus due to its integrated view of highlighted relevant information and semiautomated NLP features.
NAT adjudication improves the speed and interrater reliability for phenotyping cognitive status compared to manual chart reviews. This study underscores the potential of an NLP-based clinically adjudicated method to build large-scale dementia research cohorts from EHRs.
电子健康记录 (EHR) 具有较大的样本量和丰富的信息，为痴呆症研究提供了巨大的潜力，但目前用于表型认知状态的方法不可扩展。
本研究旨在评估自然语言处理 (NLP) 支持的半自动注释是否可以提高图表审查表型认知状态的速度和组内一致性。
在这项诊断研究中，我们开发并评估了一种半自动 NLP 支持的注释工具 (NAT)，以促进认知状态的表型。临床专家使用 NAT 或传统图表审查对马萨诸塞州综合医院 (MGB) 医疗保健的 627 名患者的认知状态进行了裁决。患者病历包含来自两个数据集的 EHR 数据：(1) MGB 责任制医疗组织 100 名医疗保险受益人的 2017 年 1 月 1 日至 2018 年 12 月 31 日记录，以及 (2) COVID-19 诊断前 2 年至 COVID-19 诊断日期的 527 名 MGB 患者的记录。提取了相关期间的所有 EHR 数据；处理和总结了诊断代码、药物和实验室测试值；通过 NLP 管道处理临床记录；并开发了一个网络工具来呈现所有数据的综合视图。认知状态被评为认知正常、认知障碍或不确定。评估了 NAT 与手动图表审查在认知状态表型评估方面的评估时间和组内一致性。
NAT 裁决提供了更高的组内一致性 (Cohen κ=0.89 对 κ=0.80)，并且与手动图表审查相比，速度显著提高 (平均时间差 1.4 分钟，SD 1.3 分钟；P<.001；中位数比 2.2，最小值-最大值 0.4-20)。与手动图表审查具有中度一致性 (Cohen κ=0.67)。在与手动图表审查不一致的情况下，NAT 裁决能够通过其对突出相关信息的综合视图和半自动 NLP 功能产生具有更广泛临床共识的评估。
与手动图表审查相比，NAT 裁决提高了表型认知状态的速度和组内一致性。这项研究强调了基于 NLP 的临床裁决方法在从 EHR 构建大型痴呆症研究队列方面的潜力。
