Pesticide spraying in orchard is an important inspection content of fruit quality and safety, and the reliable record of pesticide spraying behavior is an important part of fruit traceability system. Aiming at solving the problem that it was difficult to accurately grasp the real situation of pesticide application in the farmer professional cooperatives during the fruit planting in China, monitoring of the spraying behavior in orchard based on the interaction of human posture estimation and scenes was proposed. Firstly, the fine tuned YOLO v5 model was used to complete the precise detection of sprayers and fruit tree targets, and the features of scene interaction were extracted. Then, the OpenPose model was used to recognize human skeleton and extract human posture features. Finally, the distance and angle of the above features were calculated respectively, and fused into 11244 sets of feature vectors, which were trained by the SVM model to complete the detection of orchard spraying behavior. In order to verify the effectiveness of the algorithm, totally 92 videos with different illuminations, different distances, different numbers of people and different occlusion degrees were tested. The results showed that the ACC of the algorithm was 85.66%, the MAE was 42.53%, the RMSE was 44.59%, the RMSEP was 44.34% and the RPD was 1.56. Simultaneously, the effectiveness of spraying behavior recognition in orchard was validated under different illuminations, occlusions, distance change and single spraying among multiple people. Experimental results showed that it was feasible to apply the model to the detection of orchard spraying behavior. The research result could provide technical reference for the standardization and reliability of orchard management in the fruit traceability system.