Monitoring the monitor: automated statistical tracking of a clinical event monitor.

At Columbia-Presbyterian Medical Center, a clinical event monitor processes a set of rules called Medical Logic Modules (MLMs), which generate messages (interpretations, warnings, suggestions) for health care providers. The automated statistical tracker (AST) monitors the operation of the clinical event monitor for the purpose of detecting malfunctions in MLMs or in the clinical event monitor itself. The AST follows the number of messages generated by each MLM each day and issues an alert to a system administrator if the current count of messages seems unusual compared to the MLM's past activity. The AST is based upon a combination of Poisson and normal distributions. The AST was implemented using Unix shell scripts and put into operation. Of two malfunctions that occurred during a prospective study of 12 MLMs over 85 days, the AST automatically detected one that might otherwise have gone undetected, and the system administrator detected the other during routine review of the AST's daily report. The AST's performance was compared to that of five human subjects, and it was found to rank third among the six total subjects. The AST generated eight false-positive alerts during the study period (false-positive rate = 0.009 alerts/MLM day); seven of these were also picked by the human subjects. Subsequent experience has proven the AST to be useful and efficient for a system with 20 to 60 MLMs.