Zolnoori Maryam, Vergez Sasha, Xu Zidu, Esmaeili Elyas, Zolnour Ali, Anne Briggs Krystal, Scroggins Jihye Kim, Hosseini Ebrahimabad Seyed Farid, Noble James M, Topaz Maxim, Bakken Suzanne, Bowles Kathryn H, Spens Ian, Onorato Nicole, Sridharan Sridevi, McDonald Margaret V
Columbia University Irving Medical Center, New York, NY 10032, United States.
School of Nursing, Columbia University, New York, NY 10032, United States.
JAMIA Open. 2024 Dec 10;7(4):ooae130. doi: 10.1093/jamiaopen/ooae130. eCollection 2024 Dec.
As artificial intelligence evolves, integrating speech processing into home healthcare (HHC) workflows is increasingly feasible. Audio-recorded communications enhance risk identification models, with automatic speech recognition (ASR) systems as a key component. This study evaluates the transcription accuracy and equity of 4 ASR systems-Amazon Web Services (AWS) General, AWS Medical, Whisper, and Wave2Vec-in transcribing patient-nurse communication in US HHC, focusing on their ability in accurate transcription of speech from Black and White English-speaking patients.
We analyzed audio recordings of patient-nurse encounters from 35 patients (16 Black and 19 White) in a New York City-based HHC service. Overall, 860 utterances were available for study, including 475 drawn from Black patients and 385 from White patients. Automatic speech recognition performance was measured using word error rate (WER), benchmarked against a manual gold standard. Disparities were assessed by comparing ASR performance across racial groups using the linguistic inquiry and word count (LIWC) tool, focusing on 10 linguistic dimensions, as well as specific speech elements including repetition, filler words, and proper nouns (medical and nonmedical terms).
The average age of participants was 67.8 years (SD = 14.4). Communication lasted an average of 15 minutes (range: 11-21 minutes) with a median of 1186 words per patient. Of 860 total utterances, 475 were from Black patients and 385 from White patients. Amazon Web Services General had the highest accuracy, with a median WER of 39%. However, all systems showed reduced accuracy for Black patients, with significant discrepancies in LIWC dimensions such as "Affect," "Social," and "Drives." Amazon Web Services Medical performed best for medical terms, though all systems have difficulties with filler words, repetition, and nonmedical terms, with AWS General showing the lowest error rates at 65%, 64%, and 53%, respectively.
While AWS systems demonstrated superior accuracy, significant disparities by race highlight the need for more diverse training datasets and improved dialect sensitivity. Addressing these disparities is critical for ensuring equitable ASR performance in HHC settings and enhancing risk prediction models through audio-recorded communication.
随着人工智能的发展，将语音处理集成到家庭医疗保健（HHC）工作流程中越来越可行。音频记录的通信增强了风险识别模型，自动语音识别（ASR）系统是关键组成部分。本研究评估了4种ASR系统——亚马逊网络服务（AWS）通用版、AWS医疗版、Whisper和Wave2Vec——在美国HHC中对患者与护士沟通内容进行转录的准确性和公平性，重点关注它们对黑人和白人英语患者语音进行准确转录的能力。
我们分析了纽约市一家HHC服务机构中35名患者（16名黑人患者和19名白人患者）与护士交流的音频记录。总体而言，有860条话语可供研究，其中475条来自黑人患者，385条来自白人患者。使用单词错误率（WER）来衡量自动语音识别性能，并以人工黄金标准为基准。通过使用语言查询和单词计数（LIWC）工具比较不同种族群体的ASR性能来评估差异，重点关注10个语言维度以及包括重复、填充词和专有名词（医学和非医学术语）在内的特定语音元素。
参与者的平均年龄为67.8岁（标准差=14.4）。交流平均持续15分钟（范围：11 - 21分钟），每位患者的单词中位数为1186个。在860条总话语中，475条来自黑人患者，385条来自白人患者。AWS通用版的准确性最高，WER中位数为39%。然而，所有系统对黑人患者的准确性都有所降低，在LIWC维度如“情感”“社交”和“驱动力”等方面存在显著差异。AWS医疗版在医学术语方面表现最佳，不过所有系统在处理填充词、重复内容和非医学术语时都存在困难，AWS通用版在这方面的错误率分别为65%、64%和53%，是最低的。
虽然AWS系统表现出卓越的准确性，但种族方面的显著差异凸显了需要更多样化的训练数据集以及提高方言敏感性。解决这些差异对于确保HHC环境中ASR性能的公平性以及通过音频记录通信增强风险预测模型至关重要。
