El-Den Sarira, Moles Rebekah J, Zhang Randi, O'Reilly Claire L
Faculty of Medicine and Health, School of Pharmacy, The University of Sydney, Sydney, NSW 2006, Australia.
Pharmacy (Basel). 2021 Jan 24;9(1):28. doi: 10.3390/pharmacy9010028.
Mental Health First Aid (MHFA) training teaches participants how to assist people experiencing mental health problems and crises. Observed behavioral assessments, post-training, are lacking, and the literature largely focuses on self-reported measurement of behaviors and confidence. This study explores the reliability of an observed behavioral assessment rubric used to assess pharmacy students during simulated patient (SP) role-play assessments with mental health consumers. Post-MHFA training, pharmacy students ( = 528) participated in SP role-play assessments ( = 96) of six mental health cases enacted by consumers with lived experience of mental illness. Each assessment was marked by the tutor, participating student, and consumer (three raters). Non-parametric tests were used to compare raters' means scores and pass/fail categories. Interrater reliability analyses were conducted for overall scores, as well as pass/fail categories using intra-class correlation coefficient (ICC) and Fleiss' Kappa, respectively. Test re-test reliability analyses were conducted using Pearson's correlation. For interrater reliability analyses, the intra-class correlation coefficient varied from poor-to-good to moderate-to-excellent for individual cases but was moderate-to-excellent for combined cases (0.70; CI 0.58-0.80). Fleiss' Kappa varied across cases but was fair-to-good for combined cases (0.57, < 0.001). For test re-test reliability analyses, Pearson's correlation was strong for individual and combined cases (0.87; < 0.001). Recommended modifications to the rubric, including the addition of barrier items, scoring guides, and specific examples, as well as the creation of new case-specific rubric versions, may improve reliability. The rubric can be used to facilitate the measurement of actual, observed behaviors post-MHFA training in pharmacy and other health care curricula.
心理健康急救（MHFA）培训教导参与者如何帮助有心理健康问题和危机的人。目前缺乏对培训后行为的观察性评估，并且文献主要集中在行为和信心的自我报告测量上。本研究探讨了一种观察性行为评估量表的可靠性，该量表用于在与心理健康消费者进行模拟患者（SP）角色扮演评估期间评估药学专业学生。在MHFA培训后，药学专业学生（n = 528）参与了由有精神疾病生活经历的消费者演绎的六个心理健康案例的SP角色扮演评估（n = 96）。每次评估由导师、参与的学生和消费者（三名评分者）打分。使用非参数检验比较评分者的平均得分和及格/不及格类别。使用组内相关系数（ICC）和Fleiss' Kappa分别对总分以及及格/不及格类别进行评分者间信度分析。使用Pearson相关性进行重测信度分析。对于评分者间信度分析，个别案例的组内相关系数从差到好再到中等至优秀不等，但合并案例的组内相关系数为中等至优秀（0.70；CI 0.58 - 0.80）。Fleiss' Kappa因案例而异，但合并案例的Fleiss' Kappa为一般至良好（0.57，p < 0.001）。对于重测信度分析，个别案例和合并案例的Pearson相关性都很强（0.87；p < 0.001）。建议对该量表进行修改，包括增加障碍项目、评分指南和具体示例，以及创建新的特定案例量表版本，这可能会提高信度。该量表可用于促进对MHFA培训后药学及其他医疗保健课程中实际观察到的行为的测量。
