Xue Wei, van Hout Roeland, Cucchiarini Catia, Strik Helmer
Centre for Language and Speech Technology (CLST), Radboud University, Nijmegen, Netherlands.
Centre for Language Studies (CLS), Radboud University, Nijmegen, Netherlands.
Clin Linguist Phon. 2023 Jan 2;37(1):52-76. doi: 10.1080/02699206.2021.2009918. Epub 2021 Dec 27.
Speech intelligibility is an essential though complex construct in speech pathology. In this paper, we investigated the interrater reliability and validity of two types of intelligibility measures: a rating-based measure, through Visual Analogue Scales (VAS), and a transcription-based measure called Accuracy of Words (AcW), through two forms of orthographic transcriptions, one containing only existing words (EWTrans) and one allowing all sorts of words, including both existing words and pseudowords (AWTrans). Both VAS and AcW scores were collected from five expert raters. We selected speakers with various severity levels of dysarthria (SevL) and employed two types of speech materials, i.e. meaningful sentences and word lists. To measure reliability, we applied Generalizability Theory, which is relatively unknown in the field of pathological speech and language research but enables more comprehensive analyses than traditional methods, e.g., the intraclass correlation coefficient. The results convincingly indicate that five expert raters were sufficient to provide reliable rating-based (VAS) and transcription-based (AcW) measures, and that reliability increased as the number of raters or utterances increased. Generalizability Theory has proved effective in systematically dealing with reliability issues in our experimental design. We also investigated construct and concurrent validity. Construct validity was addressed by exploring the correlations between VAS and AcW within and across speech materials. Concurrent validity was addressed by exploring the correlations between our measures, i.e. VAS and AcW, and two external measures, i.e. phoneme intelligibility and SevL. The correlations corroborate the validity of VAS and AcW to assess speech intelligibility, both in sentences and word lists.
言语清晰度是言语病理学中一个重要但复杂的概念。在本文中，我们研究了两种清晰度测量方法的评分者间信度和效度：一种是基于视觉模拟量表（VAS）的基于评分的测量方法，另一种是基于转录的称为单词准确性（AcW）的测量方法，通过两种正字法转录形式，一种只包含现有单词（EWTrans），另一种允许各种单词，包括现有单词和伪词（AWTrans）。VAS和AcW分数均来自五位专家评分者。我们选择了具有不同构音障碍严重程度（SevL）的说话者，并使用了两种类型的言语材料，即有意义的句子和单词列表。为了测量信度，我们应用了概化理论，该理论在病理性言语和语言研究领域相对鲜为人知，但比传统方法（例如组内相关系数）能够进行更全面的分析。结果令人信服地表明，五位专家评分者足以提供可靠的基于评分（VAS）和基于转录（AcW）的测量方法，并且信度随着评分者数量或话语数量的增加而提高。概化理论已被证明在系统处理我们实验设计中的信度问题方面是有效的。我们还研究了结构效度和同时效度。通过探索言语材料内部和之间VAS和AcW的相关性来解决结构效度问题。通过探索我们的测量方法（即VAS和AcW）与两种外部测量方法（即音素清晰度和SevL）之间的相关性来解决同时效度问题。这些相关性证实了VAS和AcW在评估句子和单词列表中的言语清晰度方面的有效性。
