Evaluation of a belief-network-based reminder system that learns from utility feedback.

PRETRIEVE is a belief-network-based, unsolicited information-retrieval system that performs machine learning based on user feedback. We report here on the document-ordering and document-retrieval performance of PRETRIEVE. We developed a test collection of 410 judgments of document utility in a simulated medical order-entry context. We characterized the validity of these judgments, which were elicited from domain experts, by measuring interrater and intrarater reproducibility. We developed a measure of the quality of document orderings similar to the ROC-curve analysis used to evaluate document-retrieval systems. We found that the ordering performance of the PRETRIEVE system was (1) substantially better than random, (2) somewhat less than ideal, and (3) superior to that of versions of the PRETRIEVE system that used relevance feedback instead of utility feedback. Under a set of assumptions, which we make explicit, we found that the documents retrieved by a version of PRETRIEVE that modeled time cost were of higher utility than those retrieved by a similar rule-based system.