Potato disease is one of the most important influencing factors for agricultural high quality. Traditional methods of image processing for disease identification under light of the outdoor natural environment are easily affected by typical interfering factors, such as illumination change, uneven brightness, similar foreground and so on. Therefore, these factors will lead to low recognition rate and low robustness. To improve the accuracy and stability of disease identification, a disease recognition method of deep convolutional neural network and composite feature words was proposed. Firstly, the Faster R-CNN model was trained by the migration learning technology, disease areas were detected with leaf image. Secondly, color feature and SIFT feature were extracted from the entire patch region set by high-density sampling method, and color feature and SIFT feature vocabulary were established. Then, the K-means algorithm was used to cluster the two types of apparent feature vocabularies to construct a composite feature dictionary. Finally, the features extracted from the disease area were mapped in the compound dictionary to obtain the feature histogram, and the identification model of the disease was trained by the support vector machine. The experimental results showed that when the number of visual words in the couposite dictionary was 50, the robustness and real-time performance of disease recognition was better, the average recognition rate was 90.83%, as well as the single frame image average time-consuming was 1.68s. The average accuracy of model detection reached 84.16%, when the feature used a combination of color features and SIFT features. In addition, compared with the traditional bag of word recognition methods for the same data set, the proposed method could make the recognition accuracy increase by 25.45 percentage points.