Reliability of results produced through objectively structured assessment of technical skills (OSATS) for endotracheal intubation (ETI).

OBJECTIVE

To estimate the reliability of an objectively structured assessment of technical skills (OSATS) in patients for Endotracheal Intubation (ETI), using task-specific checklists during administration of general anaesthesia.

STUDY DESIGN

Co-relational reliability study.

PLACE AND DURATION OF STUDY

Department of Anaesthesia and Medical Education, Pakistan Institute of Medical Sciences, Islamabad, from September to November 2010.

METHODOLOGY

Ten first year residents of the department of anaesthesia who had successfully completed a one-day course of ETI on mannequins and had performed 25 ETI in actual patients under supervision, were asked to perform ETI on anaesthetized patients in the operation theater after consent. The procedure was directly observed by two expert observers (ER1 and ER2) and one inexperienced observer (novice, NR) using a validated task oriented checklist. Each trainee repeated the procedure on another patient and was reassessed using the same checklist. Inter-rater for test and retest, and expert to novice reliabilities along with internal consistency were calculated.

RESULTS

The inter-rater reliability was 0.92 in test and was 0.98 in retest (Ebel's calculation). Spearman Rank Correlation Coefficient (SRCC) for ER 1 and NR was 0.76 in test and 0.89 in retest. Cronbach's alpha for internal consistency was 0.93 for first test and 0.97 for retest.

CONCLUSION

The OSATS for ETI in real patients demonstrated excellent inter-rater test reliability, inter-rater retest reliability, expert-novice reliability and internal consistency. This instrument, therefore, proves promising as a tool for competency-based evaluations.