With the rapid development of Internet of Things and artificial intelligence, the detection and treatment of crop diseases are gradually developing towards intelligence. Using computer vision methods to identify crop diseases accurately and efficiently was of great significance to ensure the normal growth of crops. In order to extract the highlevel semantic features of images and solve the problem of different image sizes of various plant diseases and insect pests, a multilevel extremely efficient spatial pyramid (EESP) model based on deep learning was proposed. Firstly, the image was preprocessed, and then the proposed model was constructed. In order to extract characteristic information of different scales, the cavity ratio was different in each layer. By integrating the information of each layer, different characteristics of various crop pest images were obtained. Finally, crop pests and diseases were identified through image classification method. The data set included 61 pests and disease categories of 10 crops. After 300 epochs training, the experiments showed that the Top1 accuracy of the proposed model reached 88.4%, which was effectively improved by about 3 percentage points compared with that of traditional methods, and it was found that using the threelayer EESP model can obtain the best effect. It had certain practical value and can be applied in actual scenarios.