Reliable pedestrian detection in agriculture field is one of the key technologies for unmanned agricultural vehicles. For the complex environment in the orchard, it is difficult to accurately detect the obstacle information. To solve this problem, an improved single shot multibox detector (SSD) deep learning object detection method was proposed to detect pedestrian in the field obstacles. The lightweight network framework MobileNetV2 was used as the basic network in the SSD model to reduce the time and computational effort of extracting image features. For auxiliary layer of the SSD model, the inverse residual block combined with the dilated convolution was used as the basic structure for position prediction so that the multi-scale features can be integrated and at the same time avoiding the information loss caused by the down sampling operation. Based on the Tensorflow deep learning framework, different motion states (motion and static), different pose states (normal and unnormal) and different object scales (large, medium and small) pedestrian detection experiment in orchard were carried out on the open data set in orchard environment of the National Robotics Engineering Center of Carnegie Mellon University and the accuracy and speed of these different situations were compared. Result showed that the average precision and recall rate of the improved SSD pedestrian detection model in agriculture reached 97.46% and 91.65%, respectively, higher than 96.87% and 88.51% of the original SSD model, and the parameter quantity was decreased by seven times. The speed was accelerated by three times and the detection speed was 62.50 frames per second. The model had good robustness and could detect the pedestrian in the field environment, which could provide a basis for the obstacle avoidance of the unmanned agriculture machinery.