Litchi diseases and insect pests are not only various, but also have a long onset cycle. The difficulty in prevention and control is an important limiting factor affecting the production and quality of litchi. To better assist the prevention and control of diseases and insect pests in litchi and promote the healthy development of the litchi industry, a high-precision, stable and suitable identification model SHTNet was proposed for the collected image data set of litchi diseases and insect pests. Firstly, the attention mechanism SimAM was introduced into the lightweight convolutional neural network ShuffleNetV2 model. Without additional network parameters, the effective extraction of important features was improved to enhance the characteristics of litchi pests and diseases and suppress background features. Secondly, while ensuring the accuracy of model recognition, the activation function Hardswish was used to reduce the amount of the network model parameters, making the network more lightweight. Thirdly, the transfer learning method was adopted on the improved model to transfer the knowledge learned from the source data (Mini-ImageNet data set) to the target data (the litchi diseases and insect pests image data set after data enhancement), enhancing the model’s adaptability to recognize different types of litchi diseases and insect pests. The experimental results showed that the accuracy of the proposed model SHTNet reached 84.9%, which was improved by 8.8 percentage points; the precision rate reached 78.1% with an increase of 9 percentage points; the recall rate was increased by 8.8 percentage points to 73.2%; the F1 was 75.8% with an increase of 10.2 percentage points. This ultimately improved model comprehensive performance, which was superior to that of ResNet34, ResNeXt50 and MobileNetV3-large models. Therefore, the final model proposed SHTNet had better robustness and strong generalization ability, laying a solid technical foundation for diseases and insect pests of litchi real-time online identification application platform implementation.