From the Division of Management Consulting, Department of Anesthesia, University of Iowa, Iowa City, Iowa.
Department of Anesthesia, University of Iowa, Iowa City, Iowa.
Anesth Analg. 2020 Sep;131(3):909-916. doi: 10.1213/ANE.0000000000004779.
Annual and/or semiannual evaluations of pain medicine clinical faculty are mandatory for multiple organizations in the United States. We evaluated the validity and psychometric reliability of a modified version of de Oliveira Filho et al clinical supervision scale for this purpose.
Six years of weekly evaluations of pain medicine clinical faculty by resident physicians and pain medicine fellows were studied. A 1-4 rating (4 = "Always") was assigned to each of 9 items (eg, "The faculty discussed with me the management of patients before starting a procedure or new therapy and accepted my suggestions, when appropriate").
Cronbach α of the 9 items equaled .975 (95% confidence interval [CI], 0.974-0.976). A G coefficient of 0.90 would be expected with 18 raters; the N = 12 six-month periods had mean 18.8 ± 5.9 (standard deviation [SD]) unique raters in each period (median = 20).Concurrent validity was shown by Kendall τb = 0.45 (P < .0001) pairwise by combination of ratee and rater between the average supervision score and the average score on a 21-item evaluation completed by fellows in pain medicine. Concurrent validity also was shown by τb = 0.36 (P = .0002) pairwise by combination of ratee and rater between the average pain medicine supervision score and the average operating room supervision score completed by anesthesiology residents.Average supervision scores differed markedly among the 113 raters (η = 0.485; CI, 0.447-0.490). Pairings of ratee and rater were nonrandom (Cramér V = 0.349; CI, 0.252-0.446).Mixed effects logistic regression was performed with rater leniency as covariates and the dependent variable being an average score equaling the maximum 4 vs <4. There were 3 of 13 ratees with significantly more averages <4 than the other ratees, based on P < .01 criterion; that is, their supervision was reliably rated as below average. There were 3 of 13 different ratees who provided supervision reliably rated as above average.Raters did not report higher supervision scores when they had the opportunity to perform more interventional pain procedures.
Evaluations of pain medicine clinical faculty are required. As found when used for evaluating operating room anesthesiologists, a supervision scale has excellent internal consistency, achievable reliability using 1-year periods of data, concurrent validity with other ratings, and the ability to differentiate among ratees. However, to be reliable, routinely collected supervision scores must be adjusted for rater leniency.
美国的多个组织都要求对疼痛医学临床教师进行年度和/或半年度评估。为此，我们评估了经改良 de Oliveira Filho 等人临床监督量表在这方面的有效性和心理测量可靠性。
研究了住院医师和疼痛医学研究员对疼痛医学临床教师进行的六年每周评估。对 9 个项目中的每个项目（例如，“教员在开始程序或新疗法之前与我讨论了患者的管理，并在适当的情况下接受了我的建议”）进行 1-4 分（4=“总是”）的评分。
9 项的 Cronbach α 为.975（95%置信区间[CI]，0.974-0.976）。18 名评估者的 G 系数应为 0.90；N=12 个为期 6 个月的时间段，每个时间段的平均独特评估者为 18.8±5.9（标准差[SD]）（中位数=20）。平均监督评分与疼痛医学研究员完成的 21 项评估中的平均评分之间的 Kendall τb=0.45（P<0.0001）表明了同时效度。平均疼痛医学监督评分与麻醉学住院医师完成的平均手术室监督评分之间的 τb=0.36（P=0.0002）也表明了同时效度。113 名评估者的平均监督评分差异显著（η=0.485；CI，0.447-0.490）。评估者和评估者的配对不是随机的（Cramér V=0.349；CI，0.252-0.446）。进行了混合效应逻辑回归，将评估者的宽松度作为协变量，因变量为平均评分等于最大值 4 分与<4 分。基于 P<0.01 标准，有 3 名评估者的平均评分<4 分明显多于其他评估者，这表明他们的监督确实被评为低于平均水平。有 3 名不同的评估者提供的监督确实被评为高于平均水平。当评估者有机会进行更多介入性疼痛程序时，他们并没有报告更高的监督评分。
需要对疼痛医学临床教师进行评估。与用于评估手术室麻醉师时一样，监督量表具有出色的内部一致性、使用 1 年数据可实现的可靠性、与其他评分的同时有效性，以及区分评估者的能力。然而，为了可靠，必须对常规收集的监督评分进行调整，以消除评估者的宽松度。
