The real-time detection of pedestrian and agricultural vehicles is very important for the navigation and path planning of autonomous agricultural vehicles. In the field, obstacles are difficult to be detected due to crops occlusion and background interference. A real-time pedestrian and agricultural vehicles detection model in natural field scene was proposed, which effectively improved the feasibility of pedestrian and agricultural vehicles visual detection to embedded platform in the independent operation of agricultural machinery. This detection model was improved based on You only look once version 3 tiny (YOLOv3-tiny). A third prediction layer was got by merging the features of YOLOv3-tiny’s shallow layer and the features of second YOLO prediction layer, thus more smaller anchors resulted in the detection ability improvement of small targets. Both the squeeze and excitation attention module (SEAM) and the convolutional block attention module (CBAM) were applied in the key feature maps of the network, thus the model’s anti-background disturbance capability was increased. A data set included 9405 images of pedestrian and agricultural vehicles with different shooting angles and natural field scenes was set, and 7054 images were used for training while the remained 2351 images were used for testing. Tests showed that the memory size of the improved model was reduced to 1/3 and 2/3 of that of the YOLOv3 and single shot multibox detector (SSD) models, the improved model’s mean average precision (mAP) was increased by 11 percentage points, and the small target recall (R) rate was increased by 14 percentage points while compared with that of YOLOv3-tiny. On the Jetson TX2 embedded hardware platform, the single frame detection time of the improved model was 122ms, which can meet the requirements of real-time detection.