In view of the demand for online detection of sprouted potatoes, a lightweight convolutional neural network was proposed to detect sprouted potatoes. Firstly, the acquired potato samples were collected based on the grading line, and the samples were expanded through data enhancement. The Shuffle-Net lightweight convolutional neural network was built, and the effects of different learning rates and learning rate decay strategies on the model were compared. Experiment results showed that when the learning rate was 0.001 and the decay strategy was W-EP, the performance was the best. The overall recognition accuracy of sprouted potato and healthy potato was 97.8%, the single sample recognition time was 0.14s, and the model memory footprint was 5.2MB. The experimental results were evaluated, the precision was 98.0%, the recall was 97.1%, the specificity was 98.4%, and the harmonic mean was 97.5%. The VGG11, Alex-Net, and Res-Net101 models were selected for comparison with the model. It was found that the recognition accuracy of the model was greatly improved compared with that of the VGG11 and Alex-Net, and the recognition speed of a single sample was 5 times higher than that of Res-Net101. Compared with VGG11, it was nearly 7 times higher, and the model volume was greatly reduced compared with that of VGG11, Alex-Net, and Res-Net101. In the experiment, the internal convolution of the model was visually analyzed and the results were misjudged. It was found that when the buds were dark, short and at the edge of the tuber, misjudgment would be caused. It can be concluded that this experimental model realized the accurate and effective identification of sprouted potato, and it also had the advantages of fast identification speed, small size and strong portability, which can provide theoretical support for the external nondestructive testing and classification of agricultural products.