Affirmative actions: can the discriminant accuracy of a test be determined in the face of selection bias?

Clinical estimates of test efficacy can be distorted by the differential referral of positive and negative test responders for outcome verification. Accordingly, a series of computer simulations was performed to quantify the effects of various degrees of this selection bias on the observed true-positive rate, false-positive rate, and discriminant accuracy of a hypothetical test. The error in observed true- and false-positive rates was positive with respect to diagnosis, and negative with respect to prognosis. The magnitude of error was highly correlated with the magnitude of bias associated with the test response (primary selection bias), but not with the magnitude of bias associated with additional independent factors (secondary selection bias). Mathematical correction for preferential referral based on the test response using a previously published algorithm completely removed the correlation with primary selection bias for both diagnosis and prognosis. Although a significant correlation with secondary selection bias persisted at intermediate base rates, its magnitude was small. Discriminant accuracy was assessed in terms of area under a receiver operating characteristic (ROC) curve. Biased values of true- and false-positive rates were distributed along the curve defined by the actual true- and false-positive rates of the test for both diagnosis and prognosis. As a result, the areas under ROC curves calculated from biased true- and false-positive rates were within 2% of the areas calculated from the actual rates. Only when the primary and secondary observations were independent with respect to one outcome and dependent with respect to the other outcome did a systematic error appear in ROC area.(ABSTRACT TRUNCATED AT 250 WORDS)