Simulation-based assessment to evaluate cognitive performance in an anesthesiology residency program.

BACKGROUND

Problem solving in a clinical context requires knowledge and experience, and most traditional examinations for learners do not capture skills that are required in some situations where there is uncertainty about the proper course of action.

OBJECTIVE

We sought to evaluate anesthesiology residents for deficiencies in cognitive performance within and across 3 clinical domains (operating room, trauma, and cardiac resuscitation) using simulation-based assessment.

METHODS

Individual basic knowledge and cognitive performance in each simulation-based scenario were assessed in 47 residents using a 15- to 29-item scenario-specific checklist. For every scenario and item we calculated group error scenario rate (frequency) and individual (resident) item success. For all analyses, alpha was designated as 0.05.

RESULTS

Postgraduate year (PGY)-3 and PGY-4 residents' cognitive items error rates were higher and success rates lower compared to basic and technical performance in each domain tested (P < .05). In the trauma and resuscitation scenarios, the cognitive error rate by PGY-4 residents was fairly high (0.29-0.5) and their cognitive success rate was low (0.5-0.68). The most common cognitive errors were anchoring, availability bias, premature closure, and confirmation bias.

CONCLUSIONS

Simulation-based assessment can differentiate between higher-order (cognitive) and lower-order (basic and technical) skills expected of relatively experienced (PGY-3 and PGY-4) anesthesiology residents. Simulation-based assessments can also highlight areas of relative strength and weakness in a resident group, and this information can be used to guide curricular modifications to address deficiencies in tasks requiring higher-order processing and cognition.