Dexter Franklin, Epstein Richard H, Dillman Dawn, Hindman Bradley J, Mueller Rashmi N
From the Department of Anesthesia, University of Iowa, Iowa City, Iowa.
Department of Anesthesiology, Perioperative Medicine and Pain Management, University of Miami, Miami, Florida.
Anesth Analg. 2025 Mar 1;140(3):723-731. doi: 10.1213/ANE.0000000000007076. Epub 2024 Jul 11.
At all Joint Commission-accredited hospitals, the anesthesia department chair must report quantitative assessments of anesthesiologists' and nurse anesthetists' (CRNAs') clinical performance at least annually. Most metrics lack evidence of usefulness, cost-effectiveness, reliability, or validity. Earlier studies showed that anesthesiologists' clinical supervision quality and CRNAs' work habits have content, convergent, discriminant, and construct validity. We evaluated predictive validity by testing for (expected) small but statistically significant associations between higher quality of supervision (work habits) and reduced probabilities of cases taking longer than estimated.
Supervision quality of each anesthesiologist was evaluated daily by assigned trainees using the 9-item de Oliveira Filho scale. The work habits of each CRNA were evaluated daily by assigned anesthesiologists using a 6-item scale. Both are scored binary, 1 if all items are rated the maximum, 0 otherwise. From 40,718 supervision evaluations and 53,722 work habit evaluations over 8 fiscal years, 16 mixed-effects logistic regression models were estimated, with raters as fixed effects and ratees (anesthesiologists or CRNAs) as random effects. Empirical Bayes means in the logit scale were obtained for 561 anesthesiologist-years and 605 CRNA-years. The binary-dependent variable was whether the case took longer than estimated from the historical mean time for combinations of scheduled procedures and surgeons. From 264,060 cases, 8 mixed-effects logistic regression models were fitted, 1 per fiscal year, using ratees as random effects. Predictive validity was tested by pairing the 8 one-year analyses of clinical supervision, and the 8 one-year analyses of work habits, by ratee, with the 8 one-year analyses of whether OR time was longer than estimated. Bivariate errors in variable linear least squares linear regressions minimized total variances.
Among anesthesiologists, 8.2% (46/561) had below-average supervision quality, and 17.7% (99/561), above-average. Among CRNAs, 6.3% (38/605) had below-average work habits, and 10.9% (66/605) above-average. Increases in the logits of the quality of clinical supervision were associated with decreases in the logits of the probabilities of cases taking longer than estimated, unitless slope = -0.0361 (SE, 0.0053), P < .00001. Increases in the logits of CRNAs' work habits were associated with decreases in the logits of probabilities of cases taking longer than estimated, slope = -0.0238 (SE, 0.0054), P < .00001.
Predictive validity was confirmed, providing further evidence for using supervision and work habits scales for ongoing professional practice evaluations. Specifically, OR times were briefer when anesthesiologists supervised residents more closely, and when CRNAs had better work habits.
在所有经联合委员会认证的医院中，麻醉科主任必须至少每年报告对麻醉医生和麻醉护士（CRNAs）临床表现的定量评估。大多数指标缺乏有用性、成本效益、可靠性或有效性的证据。早期研究表明，麻醉医生的临床监督质量和麻醉护士的工作习惯具有内容效度、收敛效度、区分效度和结构效度。我们通过测试更高质量的监督（工作习惯）与手术时间超过预期的概率降低之间（预期的）微小但具有统计学意义的关联来评估预测效度。
每天由指定的实习生使用9项de Oliveira Filho量表评估每位麻醉医生的监督质量。每天由指定的麻醉医生使用6项量表评估每位麻醉护士的工作习惯。两者均采用二元评分，若所有项目均评为最高分则为1分，否则为0分。在8个财政年度的40718次监督评估和53722次工作习惯评估中，估计了16个混合效应逻辑回归模型，将评分者作为固定效应，被评分者（麻醉医生或麻醉护士）作为随机效应。获得了561个麻醉医生年和605个麻醉护士年的logit量表中的经验贝叶斯均值。二元因变量是手术时间是否比根据预定手术和外科医生组合的历史平均时间估计的时间长。在264060例病例中，拟合了8个混合效应逻辑回归模型，每个财政年度1个，将被评分者作为随机效应。通过将按被评分者分组的8项临床监督的一年分析和8项工作习惯的一年分析与8项手术时间是否长于预期的一年分析配对，来测试预测效度。变量线性最小二乘线性回归中的双变量误差使总方差最小化。
在麻醉医生中，8.2%（46/561）的监督质量低于平均水平，17.7%（99/561）高于平均水平。在麻醉护士中，6.3%（38/605）的工作习惯低于平均水平，10.9%（66/605）高于平均水平。临床监督质量的logit增加与手术时间超过预期概率的logit降低相关，无量纲斜率=-0.0361（标准误，0.0053），P<.00001。麻醉护士工作习惯的logit增加与手术时间超过预期概率的logit降低相关，斜率=-0.0238（标准误，0.0054），P<.00001。
预测效度得到证实，为使用监督和工作习惯量表进行持续的专业实践评估提供了进一步的证据。具体而言，当麻醉医生更密切地监督住院医生时，以及当麻醉护士有更好的工作习惯时，手术时间会更短。
