China is one of the main planting sites of citrus. Since citrus is the economic pillar of farmers from many producing regions and the raw ingredients of many fruit processing facilities, there is a strong connection between citrus output and economic benefits. The output can influence farmers’ income and facilities’ productivity directly. By estimating the output of citrus, the facilities can analyze the production and marketing situation and adjust the pricing policy in time, which is significant to the macro-control of citrus market. For a long time, the agricultural production in China relies mainly on manual work, which has high labor intensity and low efficiency. A precise visual detection of citrus can estimate the output. Also, it can provide technical support for the citrus picking robot. Therefore, it is of great significance to the study of visual detection of green citrus under natural environment. Green citrus has similar color feature to the background, which makes the visual detection of fruits difficult to be implemented. Based on deep learning technology, the visual detection of green citrus was studied by using faster RCNN. The image acquisition experiment of green citrus was designed firstly. Then 2160 images were acquired and 1500 of them were selected from artificial selection. These 1500 images contained different amounts of fruit, different areas of scale and different illuminating angles. Totally 1200 images were selected randomly as training set. The rest 300 images were left for verification. Then the experimental environment of deep learning was configured, the image acquisition experiment was designed and the sample set of green citrus was set up. Making tuning of hyper-parameters and setting the learning rate as 0.01, batch size as 128 and momentum as 0.9 to train the model. The MAP of test set by using trained model was 85.49%. Comparison experiment of Faster RCNN and Otsu method was conducted under different lighting environments, different sizes of citrus and different amounts of citrus within an image. Defining value F as comparative evaluation index to analyze the detection result of the two methods. The F value of Faster RCNN under different lighting conditions was 77.45%, which was 59.53% when Otsu method was used. The F value of different amounts of citrus were 82.58% and 60.34%. With images of citrus in different sizes, the F values were 73.53% and 49.44%. Results above showed that the given method had better detection result. It can provide technical support for automatic production in orchard and visual detection of picking robot.