In order to improve the precision of the detection and recognition of the potato seedling leaf bud and improve the efficiency of the automatic seedling production system, an improved recognition network based on the YOLO v4 network was proposed. The residual block in the feature extraction part CSPDarknet53 was replaced with Res2Net, and the depthwise separable convolution was used to reduce the computation. In this way, the receptive field of the convolutional neural network can be enlarged, the finer feature information of leaf bud can be got, and the missed detection rate of potato leaf bud can be reduced. Furthermore, a spatial feature pyramid (D-SPP module) based on dilated convolution was designed and embedded in the output of the three feature layers of the feature extraction part to improve the recognition and localization precision of potato leaf bud target. The ablation experiment was used to verify the effectiveness of the improved strategies. The experiment results showed that the recognition precision, the recall rate, the comprehensive evaluation index F1 value and the average precision of the improved network were 95.72%, 94.91%, 95% and 96.03% respectively. Comparing with the common networks such as Faster R-CNN, YOLO v3 and YOLO v4, the improved network had the better recognition performances, thus the production efficiency automatic seedling production system can be enhanced.