Aiming at the problems of large differences in disease spots and high similarity between disease spots and background in tea leaf blight (TLB) disease images collected by UAV, a lightweight network LiTLBNet for the accurate and real-time detection of TLB disease in UAV images of tea gardens in the field was designed. A lightweight M-Backbone was used to extract the distinguishing features of the TLB spots, which reduced missed detections caused by the large differences in the scales, colors, and shapes of the disease spots in the images. The SE and ECA modules were introduced into the LNeck of LiTLBNet to help the network learn more comprehensive features in the channel dimension and reduce false detections caused by the similarities between disease spots and backgrounds. The largest feature maps were deleted to reduce the calculations and the network size, and furthermore, the training samples were also augmented by rotating them by different angles, adding noise to the images, and constructing synthetic images to improve the generalization of LiTLBNet by using a small number of samples. Experimental results showed that the precision of LiTLBNet was 75.1%, and the mAP was 78.5%, which was similar to that of YOLO v5s. However, the size of LiTLBNet was only 2.0MB, which was 13.9% of the size of YOLO v5s. The proposed method can be effectively used for the real-time and accurate UAV remote sensing monitoring of TLB disease in tea gardens with a relatively large area.