Plant diseases are one of the main causes of crop yield reduction, however, traditional manual diagnosis methods are costly and inefficient, which are difficult to adapt to the demands of modern agricultural production. Recognizing crop diseases automatically and accurately is hence of great importance. Currently, most studies have focused on images taken by professionals for academic purposes, rather than by farmers in actual agricultural production. However, images taken in real applications by farmers are with far more complex backgrounds and hence alleviating the performance of many state-of-art methods. A grape leaf disease dataset were construted under natural complex environments where images were taken by farmers in actual agricultural production. And a network architecture named MANet was proposed for efficient recognition of grape leaf diseases under natural complex environment. The inverted residual module was embedded to build the model, which significantly lowered the number of model parameters. Moreover, the attention mechanism SENet module was used to improve the ability of the model to extract key disease features from complex background images and suppress other irrelevant information. In addition, a multi-scale convolution (MConv) module was designed to extract and fuse multi-scale features of disease images. The experimental results indicated that the proposed model presented a superior performance relative to other most advanced methods. On the public crop disease dataset, MANet achieved the highest average recognition accuracy of 99.65%. And even on the complex background crop disease dataset of the construction, the average recognition accuracy of grape diseases reached 87.93%, which was still better than other state-of-the-art models. Therefore, the proposed model can effectively recognize grape leaf diseases and has certain potential for practical applications.