Different forms of mechanical damage affect the germination and growth of castor seeds and the quality of castor oil after oil extraction. Therefore, it is very important to identify and classify castor seeds with mechanical damage. The classification of castor seeds with seed shells missing and castor seeds with cracks and intact castor seeds (without damage) was taken as an example. The training set and test set of castor seeds were constructed, which included two convolutional layers (eight convolution nuclei per convolutional layer), two pooling layers and one full connecting layer (128 nodes). In order to improve the accuracy and real-time performance of the convolutional neural network, the network structure was adjusted and the batch_size parameters were optimized to obtain better network structure and batch_size. The sample was expanded by turning up and down, and the learning rate and regularization coefficient of the optimizer were changed to conduct a combination test on the network, so as to obtain a combination with better accuracy and efficiency. Finally, the over-fitting of the convolutional neural network model was reduced through Dropout optimization. The experimental results showed that the average test accuracy of the network model was 92.52% when the convolution layer was 5, pooling layers was 5 and the batch_size was 32. In combination test, the Sgdm optimizer can improve the classification performance of the network by updating the network. Data amplification can increase the diversity of samples and thus reduce the over-fitting phenomenon. After the over-fitting of the convolutional neural network model was reduced by Dropout optimization, the average test accuracy of the convolutional model was 93.45%, which was 0.93 percentage points higher than that before optimization. When the learning rate was 0.01 and the regularization coefficient was 0.0005, the classification accuracy of the model could reach 94.82% after dropout optimization. The accuracy of missing seed shell castor seeds was 95.60%, the accuracy rate of cracked castor seeds was 93.33%, the accuracy rate of intact castor seeds was 95.51%, and the average detection time of a single castor seed image was 0.1435s. Finally, the system for castor seeds damage classification was developed. The results of verification of the algorithm showed that the accuracy of seed shell missing castor seeds was 96.67%, that of cracked castor seeds was 80.00%, and that of complete castor seeds was 86.67%. The combined test convolutional neural network model had a high recognition accuracy in the classification of damaged castor seeds, and the convolutional model can be applied to the detection system for the real-time classification of castor seeds.