In order to solve the problem that the color sorter can not recognize the intact and the broken Camellia oleifera seeds, an integrity recognition algorithm of Camellia oleifera seed based on convolution neural network was proposed and the image database of Camellia oleifera seed was constructed. The network structure simplification and hyper-parameter optimization was conducted to improve the classification accuracy and real-time performance of the model. Firstly, the batch normalization (BN) layer of the model was selected by the comparison experiment, which speeded up the training of the model and improved the generalization performance of the model. Moreover, the Swish function was chosen as the model activation function, which improved the recognition accuracy and speeded up the convergence of the model. Furthermore, the depth and width of the network were changed to compress the size of the model and shorten the training time. In depth, the model included four convolution layers and one fully connected layer. And in the width, the number of local receptive fields (LRFs) in the convolution layers and the number of nodes in the fully connected layer were compressed. And the second and third convolution layers were replaced by the depthwise convolutions. After the structural improvement, the model was transferred to CO-Net, which was more suitable for the integrity identification of Camellia oleifera seeds. Besides, the hyper-parameters (batch size and learning rate) that affected the performance of the model were optimized. The final model (CO-Net) not only improved the classification accuracy but also speeded up the training convergence speed and enhanced the generalization performance of the model. The results showed that the accuracy of the optimized network was 98.05%, the training time was only 0.58h, and the model size was only 1.65MB. The average time of detecting an image of Camellia oleifera seed was only 13.91ms, which can meet the requirements of realtime sorting of Camellia oleifera seed.