The high nutritional and medicinal value of apples has contributed to their widespread cultivation worldwide. Unfavorable factors in the healthy growth of trees and extensive orchard work are threatening the profitability of apples. This study reviewed deep learning combined with computer vision for monitoring apple tree growth and fruit production processes in the past seven years. Three types of deep learning models were used for real-time target recognition tasks: detection models including You Only Look Once (YOLO) and faster region-based convolutional network (Faster R-CNN); classification models including Alex network (AlexNet) and residual network (ResNet); segmentation models including segmentation network (SegNet), and mask regional convolutional neural network (Mask R-CNN). These models have been successfully applied to detect pests and diseases (located on leaves, fruits, and trunks), organ growth (including fruits, apple blossoms, and branches), yield, and post-harvest fruit defects. This study introduced deep learning and computer vision methods, outlined in the current research on these methods for apple tree growth and fruit production. The advantages and disadvantages of deep learning were discussed, and the difficulties faced and future trends were summarized. It is believed that this research is important for the construction of smart apple orchards.