The identification of vegetable seedlings offers many useful applications in precision agriculture，such as automated weeding, variable rate fertilization and precise spraying of diseased plants. Aiming to recognize vegetable seedlings accurately and rapidly in natural environment, multiple kinds of vegetable seedlings were taken as study object, and a two-stage based lightweight detection model was proposed. In order to improve the speed and efficiency of image feature extraction, a mixed depth wise convolution that naturally mixed up multiple kernel sizes in a single convolution was applied as backbone network to process input images. Moreover, the feature pyramid networks (FPN) was introduced to integrate different feature maps of backbone network with the aim of improving the identification accuracy of deep learning detection model for multi-scale targets. By minimizing network channel dimensions and decreasing the number of full connection layers in detection head，the two-stage based detection model parameters and computational complexity were greatly reduced. In addition, a distance-IoU (DIoU) loss was proposed for bounding box regression to make the predicted box match with the target box perfectly. Experimental results showed that the mean average precision and recognition speed of multiple kinds of vegetable seedlings based on the proposed model were 97.47% and 19.07 f/s, respectively, and model size was 60 MB. The average accuracy of detection model can obtain 88.55%，when a crop size was less than 32 pixels×32 pixels or leaves occlusion occurred. It was demonstrated that the two-stage based lightweight detection model havd good generalization and robustness performance compared with that by other models，such as Faster R-CNN and R-FCN. The approach presented obtained high accurate rate and fast inference speed in the recognition of vegetable seedlings, which opened a new journey for the research of vegetable detection in precision agriculture.