The Accuracy of the Learning-Curve Cumulative Sum Method in Assessing Brachial Plexus Block Competency.

BACKGROUND

The learning-curve cumulative sum method (LC-CUSUM) and its risk-adjusted form (RA-LC-CUSUM) have been proposed as performance-monitoring methods to assess competency during the learning phase of procedural skills. However, scarce data exist about the method's accuracy. This study aimed to compare the accuracy of LC-CUSUM forms using historical data consisting of sequences of successes and failures in brachial plexus blocks (BPBs) performed by anesthesia residents.

METHODS

Using historical data from 1713 BPB performed by 32 anesthesia residents, individual learning curves were constructed using the LC-CUSUM and RA-LC-CUSUM methods. A multilevel logistic regression model predicted the procedure-specific risk of failure incorporated in the RA-LC-CUSUM calculations. Competency was defined as a maximum 15% cumulative failure rate and was used as the reference for determining the accuracy of both methods.

RESULTS

According to the LC-CUSUM method, 22 residents (84.61%) attained competency after a median of 18.5 blocks (interquartile range [IQR], 14-23), while the RA-LC-CUSUM assigned competency to 20 residents (76.92%) after a median of 17.5 blocks (IQR, 14-25, P = .001). The median failure rate at reaching competency was 6.5% (4%-9.75%) under the LC-CUSUM and 6.5% (4%-9%) for the RA-LC-CUSUM method ( P = .37). The sensitivity of the LC-CUSUM (85%; 95% confidence interval [CI], 71%-98%) was similar to the RA-LC-CUSUM method (77%; 95% CI, 61%-93%; P = .15). Identical specificity values were found for both methods (67%; 95% CI, 29%-100%, P = 1).

CONCLUSIONS

The LC-CUSUM and RA-LC-CUSUM methods were associated with substantial false-positive and false-negative rates. Also, small lower limits for the 95% CIs around the accuracy measures were observed, indicating that the methods may be inaccurate for high-stakes decisions about resident competency at BPBs.