Aiming at the existing detection methods, it is difficult to accurately detect the information of kiwifruit pests and diseases on single plants in orchards over a large area, and the information obtained by ground or remote sensing data alone is incomplete. By building the ground data collection equipment, together with the remote sensing images collected by the UAV, more comprehensive information on kiwifruit canopy leaf pests and diseases was obtained from both air and ground perspectives. The Pytorch deep learning framework was selected and the YOLO v5s model was used for target detection of pest and disease leaves. When calculating the infestation rate of a single fruit tree, the pixel values of infested leaves and canopy leaves were counted by image processing instead of number counting. During the calculation of canopy pixel values, K-means cluster analysis and Otsu method threshold segmentation algorithm were compared, and both methods were more accurate, with the latter taking less time and being simpler to operate. As a result, the precision rate of the detection model was 99.54%, the recall rate was 99.24%, and the mean values of target detection and classification loss in the validation set were 0.08469 and 0.00083, respectively. Meanwhile, totally 20 disease and pest data from UAV and ground were selected, respectively, and the predicted values of the number of pest and disease leaves obtained from the detection model were compared with the real values labeled manually, and the mean absolute value errors of the disease and pest detection models from remote sensing and ground were 3.5, 2.5, 0.9, and 0.45, respectively. The detection effect of the ground-based data was better than that of the remote sensing data. The research result can provide a basis for the establishment of kiwifruit orchard pest and disease detection system, and also provide guidance for the fine management of kiwifruit orchards.