Realizing identification and counting of vegetable pests captured by yellow plates under complex conditions in the field is an essential prerequisite for targeted prevention and treatment pests and diseases of crop. Because of the small size, the large number and uneven distribution of pests trapped by yellow plates, it brings a great challenge to both manual and machine identification of pests. The current mainstream machine learning model Faster R-CNN was introduced to identify and count the main pests such as diamondback moth, striped flea beetle and bemisia tabaci on the yellow plates. It also proposed a modified Faster R-CNN pest detection algorithm (Mobile terminal pest Faster R-CNN, MPF R-CNN) based on Faster R-CNN. This algorithm combined ResNet101 network with FPN network as a feature extraction network and designed a variety of different size anchor pairs in the RPN network to judge the foreground and background of features. This algorithm also adopted ROIAlign instead of ROIPooling for feature mapping and a dual loss function for algorithm parameter control. The experimental analysis of 2440 sample images showed that the average accuracy of MPF R-CNN in the detection of bemisia tabaci, striped flea beetle, diamondback moth and other large pests (body length greater than 5mm) in the realistic and complex natural environment were 87.84%, 86.94%, 87.42% and 86.38%, respectively. The average accuracy in the low density of 0~480 on 35cm×25cm yellow plate was 93.41%, and the mean accuracy in the case of the medium density of 480~960 was 89.76%. There was no significant difference between the detection accuracy in sunny and rainy days in medium and low density and the determination coefficient between the counting result of this algorithm and the insect count was 0.9255. Simultaneously, the average recognition time of the algorithm was 1.7s when it was put into the Mi 7 mobile terminal system with the architecture of “WeChat applet + cloud storage server + algorithm server” for application test. The results showed that the present algorithm can support the current portable applications in terms of accuracy and speed and can provide technical support for the rapid mobile monitoring and identification of vegetable pests, which had a good promotion prospect.