Citrus detection and location is the foundation of citrus automated picking systems, in light of the outdoor natural picking environment, a citrus visual feature recognition model was designed based on deep convolution neural network with good robustness for typical interfering factors, such as illumination change, uneven brightness, similar foreground and background, mutual occlusion of fruit, branches and leaves, shadow coverage and so on. The model included a deep convolutional network structure which can steadily extract the visual features of citrus under natural environment, a deep pool structure which can extract highlevel semantic features to get citrus feature map, a citrus location prediction model based on nonmaximum suppression method. Moreover, the proposed model was trained by transfer learning method. Each raw image was segmented into several subimages before citrus detection to enhance the ability of multiscale object detection, and reduce the computing time of citrus detection. A testing dataset, which contained representative interference factors of natural environment, was used to test the citrus detection model, and the proposed detection model had good robustness and realtime performance. The average detection accuracy and the average loss value of the model was 86.6% and 7.7, respectively, meanwhile, the average computing time for detecting citrus from single image was 80ms. The citrus detecting model constructed by deep convolution neural network was suitable for the citrus harvesting in the natural environment.