In response to the problems of excessive parameter quantity, high computational cost, and low accuracy in the recognition model of soybean abnormal seeds, an improved lightweight neural network MobileNetV3 model was proposed, which reduced the number of layers, accelerated the training and inference speed of the model, increased the nonlinear discrimination ability of the model by adding fully connected layers and softmax layers, and facilitated the output of multiple classification tasks, by using global average pooling instead of global maximum pooling to reduce information loss, and increasing the model's generalization ability by adding a Dropout layer and removing the SE Block attention mechanism in MobileNetV3. The experimental results showed that after comparing the soybean seed image data with the traditional convolutional neural networks AlexNet, VGG16, and lightweight neural network MobilenetV3, the AlexNet algorithm's final mean average precision (mAP) was 87.3%, and the VGG16 algorithm's mAP was 87.7%. The difference in mAP between the two was small, but there was a significant difference in model size and training time during the training process, the AlexNet model had a model size of 7070kB and a training time of 5420.59s, while the VGG16 model had a model size of 19674kB and a training time of 8282.68s. Overall, AlexNet was relatively better. The recognition and training of the lightweight neural network MobileNetV3 model resulted in a model size of 32153kB, a training time of 6298.29s, and an mAP of 90.6%, which was higher than that of the two traditional algorithms and more suitable for the classification and recognition of abnormal soybean seeds. In order to improve training accuracy and speed, the structure of the MobileNetV3 network model was adjusted and improved. The optimized Opt-MobileNetV3 network model mAP reached 95.7%, which was 5.1 percentage points higher than that of the traditional MobileNetV3 neural network mAP. The model size was 9317kB, reduced by 22836kB, and training time was saved by 696.57s. The optimized model achieved reducing model size, improving accuracy, and faster training speed, which can meet the task of identifying abnormal soybean seeds.