Floyd Erin, Adler Shelley R, Crane Rebecca S, Brewer Judson, Moran Patricia, Richler Robert, Hartogensis Wendy, Kuyken Willem, Hecht Frederick M
Osher Center for Integrative Medicine, University of California, San Francisco, CA, USA.
Department of Medicine, University of Wisconsin Hospital and Clinics, Madison, WI, USA.
Glob Adv Integr Med Health. 2023 May 17;12:27536130221149966. doi: 10.1177/27536130221149966. eCollection 2023 Jan-Dec.
The Mindfulness-Based Interventions: Teaching Assessment Criteria (MBI:TAC) is an important tool for assessing teacher skill and aspects of the fidelity of mindfulness-based interventions, but prior research on and implementation of the MBI:TAC has used video recordings, which can be difficult to obtain, share for assessments, and which increase privacy concerns for participants. Audio-only recordings might be a useful alternative, but their reliability is unknown.
To assess evaluator perception of the rating process and inter-rater reliability of MBI:TAC ratings using audio-only recordings.
We prepared audio-only files from video recordings of 21 previously rated Mindfulness-Based Stress Reduction teachers. Each audio recording was rated by 3 trained MBI:TAC assessors drawn from a pool of 12 who had previously participated in rating the video recordings. Teachers were rated by evaluators who had not viewed the video recording and did not know the teacher. We then conducted semi-structured interviews with evaluators.
On the 6 MBI:TAC domains, the intraclass correlation coefficients (ICCs) for audio recordings ranged from .53 to .69 using an average across 3 evaluators. Using a single rating resulted in lower ICCs (.27-.38). Bland-Altman plots showed audio ratings had little consistent bias compared to video recordings and agreed more closely for teachers with higher ratings. Qualitative analysis identified 3 themes: video recordings were particularly helpful when rating less skillful teachers, video recordings tended to provide a more complete picture for rating, and audio rating had some positive features.
Inter-rater reliability of the MBI:TAC using audio-only recordings was adequate for many research and clinical purposes, and reliability is improved when using an average across several evaluators. Ratings using audio-only recordings may be more challenging when rating less experienced teachers.
基于正念的干预教学评估标准（MBI:TAC）是评估教师技能和基于正念的干预实施保真度方面的重要工具，但此前对MBI:TAC的研究和实施使用的是录像，录像难以获取、用于评估时难以分享，且会增加对参与者隐私的担忧。仅音频记录可能是一种有用的替代方式，但其可靠性未知。
使用仅音频记录评估评估者对MBI:TAC评分过程的看法以及评分者间的可靠性。
我们从21名先前已评分的基于正念减压疗法教师的录像中准备了仅音频文件。每个音频记录由3名经过培训的MBI:TAC评估者进行评分，这3名评估者从之前参与录像评分的12名评估者中选取。教师由未观看录像且不认识该教师的评估者进行评分。然后我们对评估者进行了半结构化访谈。
在MBI:TAC的6个领域中，使用3名评估者的平均评分，音频记录的组内相关系数（ICC）范围为0.53至0.69。使用单一评分得出的ICC较低（0.27 - 0.38）。布兰德 - 奥特曼图显示，与录像相比，音频评分几乎没有一致的偏差，并且对于评分较高的教师，两者的一致性更高。定性分析确定了3个主题：在对技能较差的教师进行评分时，录像特别有帮助；录像往往能提供更完整的评分画面；音频评分有一些积极特征。
对于许多研究和临床目的，使用仅音频记录的MBI:TAC评分者间可靠性是足够的，并且使用多名评估者的平均评分时可靠性会提高。在对经验较少的教师进行评分时，仅使用音频记录的评分可能更具挑战性。
