Department of Psychiatry and Behavioral Sciences, Dell Medical School, The University of Texas at Austin, Austin, Texas, USA.
Department of Population Health, Dell Medical School, The University of Texas at Austin, Austin, Texas, USA.
Psychiatry Res. 2023 May;323:115168. doi: 10.1016/j.psychres.2023.115168. Epub 2023 Mar 12.
Minimal guidance is available in the literature to develop protocols for training non-clinician raters to administer semi-structured psychiatric interviews in large, multi-site studies. Previous work has not produced standardized methods for maintaining rater quality control or estimating interrater reliability (IRR) in such studies. Our objective is to describe the multi-site Texas Childhood Trauma Research Network (TX-CTRN) rater training protocol and activities used to maintain rater calibration and evaluate protocol effectiveness.
Rater training utilized synchronous and asynchronous didactic learning modules, and certification involved critique of videotaped mock scale administration. Certified raters attended monthly review meetings and completed ongoing scoring exercises for quality assurance purposes. Training protocol effectiveness was evaluated using individual measure and pooled estimated IRRs for three key study measures (TESI-C, CAPS-CA-5, MINI-KID [Major Depressive Episodes - MDE & Posttraumatic Stress Disorder - PTSD modules]). A random selection of video-recorded administrations of these measures was evaluated by three certified raters to estimate agreement statistics, with jackknife (on the videos) used for confidence interval estimation. Kappa, weighted kappa and intraclass correlations were calculated for study measure ratings.
IRR agreement across all measures was strong (TESI-C median kappa 0.79, lower 95% CB 0.66; CAPS-CA-5 median weighted kappa 0.71 (0.62), MINI-MDE median kappa 0.71 (0.62), MINI-PTSD median kappa 0.91 (0.9). The combined estimated ICC was ≥0.86 (lower CBs ≥0.69).
The protocol developed by TX-CTRN may serve as a model for other multi-site studies that require comprehensive non-clinician rater training, quality assurance guidelines, and a system for assessing and estimating IRR.
在文献中几乎没有关于如何为非临床评估者制定方案以在大型多地点研究中进行半结构化精神科访谈的指导。以前的工作没有为这种研究制定出标准化的方法来维持评估者的质量控制或估计评估者间信度（IRR）。我们的目的是描述多地点德克萨斯州儿童创伤研究网络（TX-CTRN）评估者培训方案和用于维持评估者校准和评估方案有效性的活动。
评估者培训利用了同步和异步的教学学习模块，认证涉及对录像模拟量表管理的评论。认证评估者参加每月的审查会议，并完成正在进行的评分练习，以进行质量保证。使用三个关键研究措施（TESI-C、CAPS-CA-5、MINI-KID [重大抑郁发作 - MDE 和创伤后应激障碍 - PTSD 模块]）的个别措施和汇总估计 IRR 来评估培训方案的有效性。通过三位认证评估者对这些措施的录像记录进行评估，以评估协议统计数据，使用刀叉（在视频上）进行置信区间估计。为研究措施的评分计算了 Kappa、加权 Kappa 和组内相关系数。
所有措施的 IRR 一致性都很强（TESI-C 中位数 Kappa 为 0.79，置信区间下界为 0.66；CAPS-CA-5 中位数加权 Kappa 为 0.71（0.62），MINI-MDE 中位数 Kappa 为 0.71（0.62），MINI-PTSD 中位数 Kappa 为 0.91（0.9）。合并的估计 ICC 为≥0.86（置信区间下界≥0.69）。
TX-CTRN 制定的方案可作为其他需要全面非临床评估者培训、质量保证指南以及评估和估计 IRR 系统的多地点研究的模型。
