Department of Speech, Language, and Hearing Sciences, University of Colorado Boulder.
Laboratory for Interdisciplinary Statistical Analysis, Department of Applied Mathematics, University of Colorado Boulder.
Am J Audiol. 2020 Dec 9;29(4):762-773. doi: 10.1044/2020_AJA-20-00022. Epub 2020 Sep 23.
Purpose To overcome methodology limitations for studying auditory development in young children, we have recently developed an observer-based procedure that uses a conditioned, play-based, motor response (see Bonino & Leibold, 2017). The purpose of this article was to examine interrater reliability for the method. Method Video recordings of test sessions of 2- to 4-year-old children ( = 17) were examined. Detection of a 1000-Hz warble tone was measured with the Play Observer-Based, Two-Interval (PlayO2I) method in each of two conditions: for a fixed intensity level (30 dB SPL) or for a variable intensity level signal (0-30 dB SPL). All test sessions were scored independently by three observers (one real-time, two offline). Observer consensus was evaluated with Fleiss' kappa statistic. To determine if summary data were similar across the observers of each test session, the proportion of correct trials (fixed-level condition) or threshold (variable-level condition) were computed. Results The strength of observer consensus was classified as "almost perfect" and "substantial" for the fixed-level and variable-level conditions, respectively. Follow-up analysis of the variable-level data indicated that differences in observer consensus were seen based on the signal level, the type of response behavior provided by the child, and the confidence level of the real-time observer. Resulting summary data were similar across the three observers of each test session: no significant differences for estimates of the proportion of correct trials or threshold. Conclusions Results from this study confirm strong interrater reliability for the method. The PlayO2I method is a powerful tool for measuring detection and discrimination abilities in young children. Supplemental Material https://doi.org/10.23641/asha.12978197.
目的 为了克服研究幼儿听觉发育的方法学局限性，我们最近开发了一种基于观察者的程序，该程序使用条件反射、基于游戏的运动反应（参见 Bonino 和 Leibold，2017）。本文的目的是检验该方法的评分者间信度。 方法 对 2 至 4 岁儿童的测试视频记录进行了检查（n = 17）。使用 PlayO2I 方法，在两种条件下测量了 1000Hz 啁啾声的检测：固定强度水平（30 dB SPL）或可变强度水平信号（0-30 dB SPL）。所有测试均由三名观察者（一名实时，两名离线）独立评分。使用 Fleiss'kappa 统计评估观察者一致性。为了确定每个测试会话的观察者的汇总数据是否相似，计算了正确试验的比例（固定水平条件）或阈值（可变水平条件）。 结果 观察者一致性的强度被归类为“几乎完美”和“显著”，分别用于固定水平和可变水平条件。对可变水平数据的后续分析表明，观察者一致性的差异取决于信号水平、儿童提供的反应行为类型以及实时观察者的置信水平。每个测试会话的三位观察者的汇总数据相似：正确试验比例或阈值的估计值没有显著差异。 结论 本研究的结果证实了该方法具有很强的评分者间信度。PlayO2I 方法是测量幼儿检测和辨别能力的有力工具。 补充材料 https://doi.org/10.23641/asha.12978197.
