In order to recognize the sound and broken grains of wheat quickly and accurately, an image detection system of wheat grain integrity based on convolution neural network (CNN) was designed and implemented, and successfully applied to actual detection. The images of sound and broken kernels were captured and the image database and morphological characteristics database of single wheat grain were established after some image processing (segmentation and filtering). Both databases were divided into a training set and validation set according to the ratio of 7∶3. Four typical convolutional neural networks (LeNet-5, AlexNet, VGG-16 and ResNet-34) were used to build wheat grain integrity recognition model and compared with the other two traditional algorithms of machine learning (SVM and BP neural network). The results showed that the training speed of the two traditional models was faster, and SVM gave the highest accuracy of 92.25%. By contrast, all four kinds of convolutional neural networks had an accuracy rate of about 98%. Among them, the accuracy of test set of AlexNet, which had the best recognition performance, was 98.02%, and the recognition speed of it was at a rate of 0.827ms per grain. Therefore, a wheat grain integrity image detection system was developed based on this model, and used for actual detection. The detection results showed that the detecting time of 100 wheat grains was 26.3s, among which, the average image acquisition time was 21.2s, and the average image processing and recognition time was 5.1s, and the average recognition accuracy was 96.67%. The system was easy to operate, which had stable performance, and provided a reference for the design of wheat grain image detection system. 