Data mining of the E-pelvis simulator database: a quest for a generalized algorithm for objectively assessing medical skill.

Inherent difficulties in evaluating clinical competence of physicians has lead to the widespread use of subjective skill assessment techniques. Inspired by an analogy between medical procedure and spoken language, proven modeling methods in the field of speech recognition were adapted for use as objective skill assessment techniques. A generalized methodology using Markov Models (MM) was developed. The database under study was collected with the E-Pelvis physical simulator. The simulator incorporates an array of five contact force sensors located in key anatomical landmarks. Two 32-state fully connected MMs are used, one for each skill level. Each state in the model corresponds to one of the possible combinations of the 5 active contact force sensors distributed in the simulator. Statistical distances measured between models representing subjects with different skill levels are sensitive enough to provide an objective measure of medical skill level. The method was tested with 41 expert subjects and 41 novice subjects in addition to the 30 subjects used for training the MM. Of the 82 subjects, 76 were classified correctly (92%). Moreover, unique state transitions as well as force magnitudes for corresponding states (expert/novice) were found to be skill dependent. Given the white box nature of the model, analyzing the MMs provides insight into the examination process performed. This methodology is independent of the modality under study. It was previously used to assess surgical skill in a minimally invasive surgical setup using the Blue DRAGON, and it is currently applied to data collected using the E-Pelvis.