The wide distribution of pests in the field leads to difficulties in image data acquisition, and most of the traditional detection models use complex feature pyramid network (FPN) to enhance detection accuracy, which affects the real-time detection to some extent. To this end, the trap lamp device was designed to construct the pest dataset FieldPest5 and the detector YOLOF, which does not use the FPN structure, was improved to propose a pest detection model YOLOF_PD that balanced detection accuracy and efficiency. Firstly, the Cutout data augmentation method was added to alleviate the occlusion problem in the pest images, and the complete intersection over union (CIoU) loss function was used to obtain better box regression positions. Secondly, the adaptive coordinate attention (ACA) mechanism was proposed to enhance the information representation capability of the model. Specifically, the global maximum pooling (GMP) path was added to the global average pooling (GAP) path of the original coordinate attention (CA) mechanism, and the weights of different paths were updated adaptively by using learnable parameters. Finally, the Dilated_Dwise_ACA encoder was proposed to improve the performance of YOLOF for smallscale pest detection. Improvements were made to the projector and residual modules in the dilated encoder. The ACA attention mechanism was introduced after the 3×3 convolution in the projector module, and in the Residual module 3×3 depth-separable convolution and 1×1 pointwise convolution were fused. The experimental results showed that the improved YOLOF_PD model mAP achieved 93.7% on the FieldPest5 test set, which was 2.1 percentage points higher than that of the model before improvement, and the detection speed was 42.4f/s, which can meet the requirements of fast pest detection. Compared with Cascade R-CNN, RetinaNet and ATSS, YOLOF_PD achieved good performance in terms of detection effect and detection speed. The research result can lay a solid foundation for field pest data collection as well as real-time pest detection. 