For citrus picking robot in natural environment, the accurate recognition and location vision system is one of the key factors ensuring the efficiency and safety of picking operations. In order to make the robot not only acquire the location information of the picking target accurately but also the surrounding obstacles, a novel obstacle recognition and location system based on Kinect V2 and improved you only look once (YOLO V3) algorithm was proposed. Firstly, five classification principles of citrus tree in natural orchard were defined, including one class that the fruit can be picked directly and four obstacle classes. Secondly, three maximum pooling layers were added to the convolution module of the YOLO V3 structure and K-means clustering analysis was conducted on anchor box to enhance the feature extraction performance of branches and leaves of the convolution neural network. Finally, threedimensional coordinates of the classification targets were obtained by using the Kinect V2 depth mapping to guide obstacleavoiding picking operation. The experimental results showed that the F-scores of obstacles and normal fruits were 83.6% and 91.9%, respectively, the positioning error was 5.9mm and the processing time of each frame was 0.4s, the success picking rate was 80.51% and success rate of obstacle avoidance was 75.79%. The research results provided a basis and guide for the picking path planning and obstacle avoidance of robotic harvesting task in natural scene.