Aiming to achieve real-time and accurate detection of multi-scale rice pests in complex backgrounds, a dataset containing images of various rice pests was constructed and a pest detection model called YOLO v8-FDI was proposed. The model was based on the YOLO v8n architecture, utilizing the more efficient FasterNet as its backbone network. This design optimized the network structure while maintaining sensitivity to pest features. Dynamic Head technology was incorporated, allowing the model to dynamically adjust the detection heads in the output layer. This improved the model’s accuracy and generalization for pests of different types and sizes. Furthermore, the Inner-IoU loss function was adopted to automatically adjust scaling factors during loss calculation process. This accelerated training convergence and further improved model performance. Experimental results showed that the YOLO v8-FDI model processed a single pest image in an average time of 12.43 m, achieving a processing speed of 80 frames per second (FPS), meeting real-time requirements for practical applications. On the test set, the model’s detection precision, mean average precision mAP@0.5:0.95 and F1 score were 97.7%, 94.0%, and 97.2%, respectively. Compared with YOLO v3-tiny, YOLO v5n, YOLO v7-tiny, YOLO v8n,YOLO v9t,and YOLO v10n, precision was improved by 5.2, 2.7, 6.7, 3.4, 2.2, and 3.2 percentage points, mAP@0.5:0.95 was increased by 10.8, 5.4, 18.1, 2.3, 1.0, and 6.4 percentage points, and F1 score was raised by 2.6, 2.0, 4.9, 1.2, 1.3, and 2.9 percentage points. The novelty lied in the improvements made to the YOLO v8n architecture by integrating FasterNet, Dynamic Head, and the Inner-IoU loss function. These enhancements significantly improved the model’s accuracy and generalization, offering strong technical support for real-time and accurate pest monitoring in complex backgrounds.