Tomatoes are a globally important economic crop with a wide planting area. In order to ensure tomato food safety and improve the economic benefits of tomatoes, accurate surface defect detection and quality grading of tomatoes are necessary. However, traditional tomato defect detection mainly relies on manual sorting during harvesting, which results in low efficiency and high missed detection rate. What’s more, new defects generated during procurement and transportation (such as dents, cracks, etc.) would be ignored. Therefore, an improved YOLO v11 method (Tomato defect detection YOLO, TDD-YOLO) was proposed for surface defect detection of tomatoes automatically, including white spot, hyperplasia, depression, crack, and deterioration. Firstly, a HE-Head layer of the YOLO 11 was constructed by fusing the wavelet depth separable convolution module to detect small targets, such as white spot, while maintaining its lightweight design. Secondly, the WC3k2 module was used to replace the original C3k2 module of YOLOv 11 to expand the receptive field of the model in the feature extraction stage, and a lightweight dynamic upsampling method was used to replace the original upsampling. These two improvements of YOLO v11 were to reduce the number of parameters and improve the realtime performance. Finally, an adaptive threshold focus loss function was used to improve the model’s attention to various classification label samples in response to the diversity and complexity of tomato surface defect types and distributions. Several experiments were carried out to evaluate the performance of the proposed method. The experimental results showed that the proposed TDD-YOLO method effectively improved the detection accuracy while keeping the model parameters basically unchanged. The overall recognition precision was 89.0% and the recall rate was 84.9%, which was increased by 2.9 percentage points and 5.6 percentage points comparing with that of YOLO v11, respectively. In comparative experiments, the proposed model had better detection performance than all published YOLO series models, Faster R-CNN and EfficientDet on detecting tomato surface defects. The proposed method achieved a detection speed of 142.89f/s, meeting the real-time detection speed requirements of industrial production applications. This work can provide important technical support for standardized and industrialized tomato detection and inspection.