Convolutional neural networks (CNNs) are widely used for artificial intelligence (AI)-based image classification. Residual network (ResNet) is a new technology that facilitates the accuracy of image classification by CNN-based AI. In this study, we developed a novel AI model combined with ResNet to diagnose colorectal polyps. In total, 127,610 images consisting of 62,510 images with adenomatous polyps, 30,443 with non-adenomatous hyperplastic polyps, and 34,657 with healthy colorectal normal mucosa were subjected to deep learning after annotation. Each validation process was performed using 12,761 stored images of colorectal polyps by a 10-fold cross validation. The efficacy of the ResNet system was evaluated by sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and diagnostic accuracy. The sensitivity, specificity, PPV, NPV, and diagnostic accuracy for adenomatous polyps at WLIs were 98.8%, 94.3%, 90.5%, 87.4%, and 92.8%, respectively. Similar results were obtained for adenomatous polyps at narrow-band imagings (NBIs) and chromoendoscopy images (CEIs) (NBIs vs. CEIs: sensitivity, 94.9% vs. 98.2%; specificity, 93.9% vs. 85.8%; PPV, 92.5% vs. 81.7%; NPV, 93.5% vs. 99.9%; and overall accuracy, 91.5% vs. 90.1%). The ResNet model is a powerful tool that can be used for AI-based accurate diagnosis of colorectal polyps.