In order to deal with the problem of low automation and low recognition accuracy in the current rice planthopper image recognition research, an image classification algorithm based on transfer learning and Mask R-CNN was proposed. Firstly, according to biological characteristics of rice planthopper, the self-developed wild insect image collection device was utilized to obtain insect images automatically. Then, the dataset was divided into two categories: rice planthopper and non-rice planthopper by the image label tool VIA, and was trained in the ResNet50 framework with transfer learning. Finally, the Mask R-CNN image classification experiments were carried out based on rice planthopper images, non-rice planthopper images, insect images with disturbances and those images which were adhesive and overlapping, respectively. Moreover, experiments were compared with SVM, BP neural network, which were traditional image classification algorithms, and Faster R-CNN algorithm. Experiment results showed that the method based on transfer learning and Mask R-CNN could distinguish the rice planthopper and non-rice planthopper images effectively and the average classification accuracy reached 0.923 under the same sample conditions, which could provide information support for the prevention and early warning of rice planthoppers.