Deep learning (DL) algorithms have been proven to be very effective in a wide range of computer vision applications, such as segmentation, classification, and detection. DL models can automatically assess complex medical image scenes without human intervention and can be applied as a second reader to provide an additional opinion for the physician. To predict the axillary lymph node (ALN) metastatic status in patients with early-stage breast cancer, a deep learning-based computer-aided prediction system for ultrasound (US) images was proposed. A total of 153 women with breast tumor US images were involved in this study; there were 59 patients with metastasis and 94 patients without ALN metastasis. A deep learning-based computer-aided prediction (CAP) system using the tumor region and peritumoral tissue in ultrasound (US) images were employed to determine the ALN status in breast cancer. First, we adopted Mask R-CNN as our tumor detection and segmentation model to obtain the tumor localization and region. Second, the peritumoral tissue was extracted from the US image, which reflects metastatic progression. Third, we used the DL model to predict ALN metastasis. Finally, the simple linear iterative clustering (SLIC) superpixel segmentation method and the LIME explanation algorithm were employed to explain how the model makes decisions. The experimental results indicated that the DL model had the best prediction performance on tumor regions with 3 mm thick peritumoral tissue, and the accuracy, sensitivity, specificity, and AUC were 81.05% (124/153), 81.36% (48/59), 80.85% (76/94), and 0.8054, respectively. The results indicated that the proposed CAP system could help determine the ALN status in patients with early-stage breast cancer. The results reveal that the proposed CAP model, which combines primary tumor and peritumoral tissue, is an effective method to predict the ALN status in patients with early-stage breast cancer.