To reduce the loss of crop diseases, a large number of chemicals are used for disease control. However, due to the untimely and inaccurate judgment of the disease, chemical agents are abused, which also has a great impact on the ecological environment and food safety. Therefore, it is urgent to develop accurate crop disease recognition based on images. In the image recognition of crop diseases, the shape, region, and color of leaf spots are the main indexes to distinguish different types of diseases. In order to extract more accurate and rich color features and spatial features of leaf spots, and solve the problems of coarse fine-grained classification of disease severity and low recognition accuracy, a fusion color mask and self-attention network (FCMSAN) was proposed. FCMSAN was composed of a color mask network (CMN) and a channel adaptive self-attention network (CASAN). CMN can improve the ability of color feature extraction by learning the color features of leaf spots. CASAN can extract the disease spot features in the global scope and add the location features and channel adaptive features of the disease spot, which can locate the leaf disease spot area accurately and comprehensively. Finally, the outputs of CMN and CASAN were fused by transfer fusion layer (TFL). The experimental results showed that the classification Top-1 accuracy reached 87.97% and the average F1 value can reach 84.48% in the fine-grained identification of 61 types of crop diseases. Finally, the effectiveness of the proposed method for crop disease recognition was verified by visualization experiments.