In order to realize precise detection of green walnut in natural environment, Faster R-CNN algorithm was improved with three methods for higher adaptability, including batch normalization processing of convolution layer, improved RPN using bi-linear interpolation and the establishment of mixed loss function to strengthen the cohesion of the model. The pre-trained VGG16 network was used as feature extractor, and SGD and Adam optimization methods were adopted to training model respectively. The improved Faster R-CNN model was compared with Faster R-CNN model under the same test conditions. Images of different resolution were used as inputs to explore the impact of image sizes on model performance. Precision, recall rate and F1 value were used as the accuracy indexes of the model, and average detection time per image was used to evaluate the speed performance. The investigation showed that the model trained by Adam optimizer was more stable, its precision was 97.71%, the recall rate was 94.58%, and the F1 value was 96.12%. The single image detection time was 0.227s. The precision of improved Faster R-CNN was 5.04 percentage points higher than that of the unimproved Faster R-CNN model, the recall rate was increased by 4.65 percentage points and the F1 index was increased by 4.84 percentage points. Besides, image detection time per image was decreased by 0.148s. The proposed method was verified to obtain the success rate of 91.25% in the walnut garden environment. The proposed method had high precision, fast speed and good robustness for walnut recognition under natural condition, which can provide a basis for the robot to recognize and pick walnuts in a complex environment quickly for a long time.