An apple localization and grading algorithm was proposed based on an improved SSD convolutional neural network to achieve fast and accurate automatic grading of apple fruit diameter and shape. The efficiency of apple grading was improved by improving the input layer of the original SSD network. Channel separation was performed on the color apple image obtained from the top, and the two channels in the separation channel that had the most significant impact on the apple recognition accuracy were extracted. A fused image was composed of the two channels and the apple depth image from the top based on the binocular camera. The longitudinal diameter-related information of the apple was calculated in the fused image. Moreover, multiple apple shape grading and information output based on the fused image were realized through this method. The depthwise-separable convolution module was used to replace part of the standard convolution in the original SSD network backbone feature extraction network, which achieved the light weighting of the network. The recognition recall, accuracy, mAP and F1 values of the trained model under the verification set were 93.68%, 94.89%, 98.37% and 94.25%, respectively. By comparing and analyzing the differences in recognition accuracy among the four input layers, the experimental results showed that the highest recognition and grading mAP for apples was achieved when the image channel combination of the input layer was DGB. The actual recognition localization and grading effects of the original SSD, Faster R-CNN and YOLO v5 algorithms for apples with different numbers of fruits were compared by using the same input layer and evaluated in terms of mAP. The experimental results showed that the improved SSD had a comparable mAP to the original SSD for dense apples, which was higher than that of Faster R-CNN by 1.33 percentage points and higher than YOLO v5 by 14.23 percentage points. The advantages of the algorithm localization and grading efficiency were verified under different hardware conditions. The detection time of an image was 5.71ms under GPU and 15.96ms under CPU, and the actual frame rate of the detected video reached 175.17f/s and 62.64f/s. The research result can provide a theoretical basis for automated grading equipment to accurately locate and grade apples in a high-speed environment.