Fan Shih-Chen, Tsai Shao-Tong, Wang Yi-Ching, Lee Meng-Lin, Huang Sheau-Ling, Hsieh Ching-Lin
Department of Occupational Therapy, College of Medical Science and Technology, Chung Shan Medical University, Taichung, Taiwan.
Occupational Therapy Room, Chung Shan Medical University Hospital, Taichung, Taiwan.
Br J Occup Ther. 2024 Jul;87(7):424-433. doi: 10.1177/03080226241239574. Epub 2024 Mar 28.
The Gap-Kalamazoo Communication Skills Assessment Form (GKCSAF) is widely used in medical education, yet its reliability in real occupational therapy clinical settings remains unexplored. This study aimed to assess the intra-rater and inter-rater reliability, as well as random measurement error, of the GKCSAF in occupational therapy.
Five independent raters evaluated audio-recordings and transcripts of conversations involving 30 patients treated by 22 assessors (7 therapists and 15 students). Both direct and coded ratings were used.
For direct ratings, intra-rater reliability was moderate for total score (intraclass correlation coefficient (ICC) = 0.76), but poor for inter-rater (ICC = 0.31). minimal detectable change (MDC%) was acceptable for the same rater (17.8%) but not for different raters (38.3%). Weighted kappa values indicated poor to fair reliability (-0.01 to 0.34) for each domain score. Coded ratings showed moderate intra-rater reliability (ICC = 0.69) and poor inter-rater reliability (ICC = 0.22). MDC% was acceptable for the same rater (24.8%) but not for different raters (65.5%). Weighted kappa values indicated poor to fair reliability (-0.02 to 0.33) for each domain score.
GKCSAF displays acceptable intra-rater but poor inter-rater reliability in occupational therapy clinical scenarios. Multiple raters are advised for enhanced reliability, while coding might not significantly enhance it. It is advisable to use the GKCSAF cautiously in occupational therapy education, ensuring adequate training, and possibly incorporating multiple raters for assessment consistency.
间隙-卡拉马祖沟通技能评估表（GKCSAF）在医学教育中被广泛使用，但其在实际职业治疗临床环境中的可靠性尚未得到探索。本研究旨在评估GKCSAF在职业治疗中的评分者内信度、评分者间信度以及随机测量误差。
五名独立评分者对涉及22名评估者（7名治疗师和15名学生）治疗的30名患者的对话录音和文字记录进行评估。同时使用了直接评分和编码评分。
对于直接评分，总分的评分者内信度中等（组内相关系数（ICC）=0.76），但评分者间信度较差（ICC=0.31）。同一评分者的最小可检测变化（MDC%）可接受（17.8%），但不同评分者的不可接受（38.3%）。加权kappa值表明每个领域得分的信度从差到一般（-0.01至0.34）。编码评分显示评分者内信度中等（ICC=0.69），评分者间信度较差（ICC=0.22）。同一评分者的MDC%可接受（24.8%），但不同评分者的不可接受（65.5%）。加权kappa值表明每个领域得分的信度从差到一般（-0.02至0.33）。
在职业治疗临床场景中，GKCSAF显示出可接受的评分者内信度，但评分者间信度较差。建议使用多名评分者以提高信度，而编码可能不会显著提高信度。在职业治疗教育中谨慎使用GKCSAF是明智的，要确保进行充分培训，并可能纳入多名评分者以保证评估的一致性。
