Automatic labels are as effective as manual labels in digital pathology images classification with deep learning.

The increasing availability of biomedical data is helping to design more robust deep learning (DL) algorithms to analyze biomedical samples. Currently, one of the main limitations to training DL algorithms to perform a specific task is the need for medical experts to manually label the data. Automatic methods to label data exist; however, automatic labels can be noisy, and it is not completely clear in which situations they can be used to train DL models. This paper aims to investigate under which circumstances automatic labels can be used to train a DL model for the classification of whole slide images. The analysis involves multiple architectures, such as convolutional neural networks and vision transformer, and 10,604 WSIs as training data, collected from three use cases: celiac disease, lung cancer, and colon cancer, which include respectively binary, multiclass, and multilabel data. The results identify 10% as the percentage of noisy labels before a performance drop-off, so to train effective models for the classification of WSIs, reaching, respectively, F1-scores of 0.906, 0.757, and 0.833. Therefore, an algorithm generating automatic labels needs to stay within this range to be adopted, as shown by the application of Semantic Knowledge Extractor Tool as a tool to automatically extract concepts and use them as labels. Automatic labels are as effective as manual labels in this case, achieving solid performance comparable to that obtained by training models with manual labels.