Comparison of visual grading and free-response ROC analyses for assessment of image-processing algorithms in digital mammography.

OBJECTIVE

To compare two methods for assessment of image-processing algorithms in digital mammography: free-response receiver operating characteristic (FROC) for the specific task of microcalcification detection and visual grading analysis (VGA).

METHODS

The FROC study was conducted prior to the VGA study reported here. 200 raw data files of low breast density (Breast Imaging-Reporting and Data System I-II) mammograms (Novation DR, Siemens, Germany)-100 of which abnormal-were processed by four image-processing algorithms: Raffaello (IMS, Bologna, Italy), Sigmoid (Sectra, Linköping, Sweden), and OpView v. 2 and v. 1 (Siemens, Erlangen, Germany). Four radiologists assessed the mammograms for the detection of microcalcifications. 8 months after the FROC study, a subset (200) of the 800 images was reinterpreted by the same radiologists, using the VGA methodology in a side-by-side approach. The VGA grading was based on noise, saturation, contrast, sharpness and confidence with the image in terms of normal structures. Ordinal logistic regression was applied; OpView v. 1 was the reference processing algorithm.

RESULTS

In the FROC study all algorithms performed better than OpView v. 1. From the current VGA study and for confidence with the image, Sigmoid and Raffaello were significantly worse (p<0.001) than OpView v. 1; OpView v. 2 was significantly better (p=0.01). For the image quality criteria, results were mixed; Raffaello and Sigmoid for example were better than OpView v. 1 for sharpness and contrast (although not always significantly).

CONCLUSION

VGA and FROC discordant results should be attributed to the different clinical task addressed.

ADVANCES IN KNOWLEDGE

The method to use for image-processing assessment depends on the clinical task tested.