In order to accurately detect litchi diseases and insect pests with complex background in natural environment in real time, the data set of litchi diseases and insect pests was constructed and the detection model of litchi diseases and insect pests was proposed for diagnosis and control. Based on YOLO v4, GhostNet, the lighter and faster lightweight network, was used as the backbone network to extract features. According to the core design of GhostNet, Ghost Module, a lower cost convolution, was used to replace the traditional convolution in the neck structure. Based on the lightweight YOLO v4-G model, the feature fusion method and attention mechanism called CBAM were used to improve the YOLO v4-G. The detection accuracy was improved without losing the detection speed and the lightweight degree of the model. Finally, the YOLO v4-GCF detection model of litchi diseases and insect pests was proposed. The dataset contained 3725 images of litchi diseases and insect pests. Litchi diseases included sooty mold, anthracnose and algal spot. Litchi insect pests included leaf mite and Dasineura sp. The experimental results showed that the average accuracy of five kinds of diseases and insect pests targets detected by YOLO v4-GCF detection model in train set, validation set and test set was 95.31%, 90.42% and 89.76%, respectively. The detection time of a single image was 0.1671s, and the size of the model was 39.574MB. Compared with the YOLO v4, the model size was reduced by 84%, the detection speed was increased by 38% and the average accuracy in the test set was improved by 4.13 percentage points. At the same time, the average accuracy was 17.67,12.78 and 25.94 percentage points higher than those of YOLO v4-tiny, EfficientDet-d2 and Faster R-CNN, respectively. The proposed YOLO v4-GCF detection model of litchi diseases and insect pests can effectively inhibit the interference of complex background, and accurately and quickly detect targets of litchi diseases and insect pests in the images, which can provide reference for crop diseases and insect pests detection research with complex and unstructured background in natural environment.