Aiming at the difficulty of grape downy mildew grading detection under the complex background of natural environment, a method of grape downy mildew grading detection based on semantic segmentation combined with K-means clustering and random forest was proposed to realize the rapid grading of grape downy mildew. The image data set of grape downy mildew under the complex background of natural environment was constructed, and the semantic segmentation model of grape leaf was established by HRNet v2+OCR network to extract grape leaf image. The K-means clustering algorithm was used to decompose grape leaf image into several subregion images, and a small number of data sets were marked for random forest learning to realize grape leaf disease spot segmentation and extraction from leaf image. At the same time, in the process of grape leaf extraction and disease spot extraction, an image size transformation method was designed to solve the problem of low accuracy caused by image resolution. The accuracy of grape leaf segmentation model based on HRNet v2+OCR network was 98.45%, and the mean intersection over union was 97.23%. The accuracy rates of downy mildew grading of grape leaf front, back and both sides were 52.59%, 73.08% and 63.32%, respectively, and the accuracy rates of disease grade error less than or equal to grade 2 were 88.67%, 96.97% and 92.98%, respectively. The research results showed that the grape downy mildew grading detection method based on K-means clustering and random forest could accurately segment grape leaf and grape downy mildew spots under the complex background of natural environments, and achieve grape downy mildew rapid grading, providing method and model support for precise control of grape downy mildew.