Department of Anesthesia, University of Iowa, 200 Hawkins Drive, 6-JCP, Iowa City, IA, USA.
Department of Business Analytics, University of Iowa, Iowa City, USA.
Health Care Manag Sci. 2020 Dec;23(4):640-648. doi: 10.1007/s10729-020-09518-0. Epub 2020 Sep 18.
Daily evaluations of certified registered nurse anesthetists' (CRNAs') work habits by anesthesiologists should be adjusted for rater leniency. The current study tested the hypothesis that there is a pairwise association by rater between leniencies of evaluations of CRNAs' daily work habits and of didactic lectures. The historical cohorts were anesthesiologists' evaluations over 53 months of CRNAs' daily work habits and 65 months of didactic lectures by visiting professors and faculty. The binary endpoints were the Likert scale scores for all 6 and 10 items, respectively, equaling the maximums of 5 for all items, or not. Mixed effects logistic regression estimated the odds of each ratee performing above or below average adjusted for rater leniency. Bivariate errors in variables least squares linear regression estimated the association between the leniency of the anesthesiologists' evaluations of work habits and didactic lectures. There were 29/107 (27%) raters who were more severe in their evaluations of CRNAs' work habits than other anesthesiologists (two-sided P < 0.01); 34/107 (32%) raters were more lenient. When evaluating lectures, 3/81 (4%) raters were more severe and 8/81 (10%) more lenient. Among the 67 anesthesiologists rating both, leniency (or severity) for work habits was not associated with that for lectures (P = 0.90, unitless slope between logits 0.02, 95% confidence interval -0.34 to 0.30). Rater leniency is of large magnitude when making daily clinical evaluations, even when using a valid and psychometrically reliable instrument. Rater leniency was context dependent, not solely a reflection of raters' personality or rating style.
麻醉医师对注册护士麻醉师（CRNA）日常工作习惯的评估应根据评估者的宽松程度进行调整。本研究检验了一个假设，即评估者对 CRNA 日常工作习惯和客座教授和教师的讲座评估的宽松程度之间存在成对关联。历史队列是麻醉医师对 CRNA 日常工作习惯的 53 个月和客座教授和教师的讲座的 65 个月的评估。二项端点是所有 6 项和 10 项的李克特量表评分，分别等于所有项目的 5 分最大值，或没有。混合效应逻辑回归估计了每个被评估者在调整评估者宽松程度后表现优于或低于平均水平的几率。双变量误差变量最小二乘线性回归估计了麻醉医师对工作习惯和讲座评估宽松程度之间的关联。在 107 名评估者中有 29 名（27%）比其他麻醉医师更严格地评估 CRNA 的工作习惯（双侧 P < 0.01）；34 名（32%）评估者更宽松。在评估讲座时，81 名评估者中有 3 名（4%）更严格，8 名（10%）更宽松。在对两者进行评估的 67 名麻醉医师中，工作习惯的宽松程度（或严格程度）与讲座的宽松程度（或严格程度）无关（P = 0.90，对数之间的无单位斜率为 0.02，95%置信区间为-0.34 至 0.30）。即使使用有效且心理测量可靠的工具，在进行日常临床评估时，评估者的宽松程度也很大。评估者的宽松程度是上下文相关的，不仅仅是评估者个性或评分风格的反映。
