Aiming at the problems of low accuracy and poor generalization ability caused by the imbalance of samples between data sets, shooting angles, light changes and other actual imaging and environmental factors caused by apple leaf diseases, a type of asymmetric shuffle convolution neural network ASNet was proposed. Firstly, by adding an improved scSE attention mechanism module to the ResNeXt backbone network to enhance the network feature extraction; secondly, for the relatively scattered feature distribution of most leaf diseases, the asymmetric shuffle convolution module was used to replace the original residual module to expand the receptive field of the convolution kernel and the enhanced feature extraction ability, thereby improving the recognition accuracy and generalization ability of the model; finally, the use of channel squeeze and channel shuffling in the asymmetric shuffle convolution module made up for the grouping convolution. The defect of insufficient correlation between channels reduced the problem of low recognition accuracy of traditional network models caused by the imbalance between leaf diseases. Under the COCO data set evaluation index, the experimental results showed that compared with the Mask R-CNN whose backbone network was ResNeXt-50, the average test accuracy of this model reached 96.8%, which was increased by 5.2 percentage points, and the model size was reduced to 321 MB, a decrease of 170 MB. Tested by 240 field-collected and AI Challanger crop disease identification challenge apple leaf images, the test results showed that the average segmentation accuracy of the proposed model ASNet for apple black rot, rust, scab and healthy leaves reached 94.7%.