Scalable Nuclei Detection in HER2-SISH Whole Slide Images via Fine-Tuned Stardist with Expert-Annotated Regions of Interest.

Breast cancer remains a critical health concern worldwide, with histopathological analysis of tissue biopsies serving as the clinical gold standard for diagnosis. Manual evaluation of histopathology images is time-intensive and requires specialized expertise, often resulting in variability in diagnostic outcomes. In silver in situ hybridization (SISH) images, accurate nuclei detection is essential for precise histo-scoring of HER2 gene expression, directly impacting treatment decisions. This study presents a scalable and automated deep learning framework for nuclei detection in HER2-SISH whole slide images (WSIs), utilizing a novel dataset of 100 expert-marked regions extracted from 20 WSIs collected at the University of Malaya Medical Center (UMMC). The proposed two-stage approach combines a pretrained Stardist model with image processing-based annotations, followed by fine tuning on our domain-specific dataset to improve generalization. The fine-tuned model achieved substantial improvements over both the pretrained Stardist model and a conventional watershed segmentation baseline. Quantitatively, the proposed method attained an average F1-score of 98.1% for visual assessments and 97.4% for expert-marked nuclei, outperforming baseline methods across all metrics. Additionally, training and validation performance curves demonstrate stable model convergence over 100 epochs. These results highlight the robustness of our approach in handling the complex morphological characteristics of SISH-stained nuclei. Our framework supports pathologists by offering reliable, automated nuclei detection in HER2 scoring workflows, contributing to diagnostic consistency and efficiency in clinical pathology.