Verbal protocols for assessing the usability of clinical decision support: the retrospective sense making protocol.

We compare the effectiveness of two types of verbal protocol, concurrent think aloud vs. retrospective sense making, for evaluating the usability of a clinical decision support tool. Thirty-five medical and nursing students participated in a usability experiment. Participants were asked to complete seven tasks using the system under evaluation. Eighteen students were allocated to the concurrent think aloud group and the remainder followed the retrospective protocol. The retrospective protocol was significantly more sensitive than the concurrent protocol in recording unique usability problems related to users' cognitive behaviour. These problems concerned the interpretation and comprehension of statistical output, search results and system messages. These findings can be explained by the retrospective protocol's greater ability to detect compound usability problems, capturing the cognitive dimensions of users' interactions with the interface in greater depth. Evaluations of clinical decision support systems should take an evidence-based approach to selecting protocols.