A novel shifted window (Swin) Transformer coffee bean grading model called Swin-HSSAM has been proposed to address the challenges of accurately classifying green coffee beans and low identification accuracy. This model integrated the Swin Transformer as the backbone network; fused features from the second, third, and fourth stages using the high-level screening-feature pyramid networks module; and incorporated the selective attention module (SAM) for discriminative power enhancement to enhance the feature outputs before classification. Fusion Loss was employed as the loss function. Experimental results on a proprietary coffee bean dataset demonstrate that the Swin-HSSAM model achieved an average grading accuracy of 96.34% for the three grading as well as the nine defect subdivision levels, outperforming the AlexNet, VGG16, ResNet50, MobileNet-v2, Vision Transformer (ViT), and CrossViT models by 3.86%, 2.56%, 0.44%, 4.05%, 5.36%, and 5.40% percentage points, respectively. Evaluations on a public coffee bean dataset revealed that, compared with the aforementioned models, the Swin-HSSAM model improved the average grading accuracy by 1.01%, 0.13%, 4.75%, 0.85%, 0.73%, and 0.27% percentage points, respectively. These results indicate that the Swin-HSSAM model not only achieved high grading accuracy but also exhibited broad applicability, providing a novel solution for the automated grading and identification of green coffee beans.