Traditional methods of image processing for crop detection under agricultural natural environment are easily affected by small samples and subjective judgment, so they have many disadvantages such as low recognition rate and low robustness. Deep learning can self-study according to data set, and has a strong ability to express feature. Therefore, a new broccoli seedlings detection approach based on Faster R-CNN model was proposed. Data acquisition was the first step to build deep learning model, and the diversity of data can improve the generalization ability of the model. According to the characteristics of field environment, broccoli seedlings images with different light intensities, different ground moisture contents and different weed densities were collected. The sample size was expanded by images rotation and noise enhancement, and data set was transformed as PASCAL VOC format. And then the Faster R-CNN model was trained by using data set. Contrast experiment was designed on ResNet101, ResNet50 and VGG16 networks. The results showed that ResNet101 network with the deepest network layer and smaller parameter space was the best feature extraction network. The average detection accuracy was 90.89%, and the average time-consuming was 249ms. Based on that, the network super-parameters were optimized and the average accuracy of model detection reached 91.73%, when Dropout value was 0.6. The results showed that this approach can effectively detect broccoli seedlings in agricultural natural environment, and provided a hopeful solution for crop detection in the field of agriculture.