The Diptera fruit fly adults of B.dorsalis Hendel, the B.tau Walker and the B.cucurbitae are the dominant species in the south of China. Because of its wide host range and high risk, it has been the most serious pest in the citrus growing areas in South China. Under the premise of accuracy, how to reduce the human and material resources for monitoring insect pests is an urgent problem to be solved. From the view of image recognition, this paper studied the morphological characteristics of the harmful flies, and proposed a classification algorithm. In the algorithm, Hough transform was used to detect the lines of fly wings to correct the direction of fly and define the effective area of the stripe by lines. Filtering in HSV space was used to detect the scutellum of fly waist and abdomen. A combination of the two ways separate the mesonotum from the whole fly. According to definition formula of characteristic factor of the central stripe, four shape feature parameters are extracted to form the feature vector after digital processing. Feature data sets were built by collecting feature vectors in 90 sample images, and the BP neural network was trained to get the neural network model parameters for the classification of the flies. Experimental results showed that the recognition effect of this method on Diptera fruit fly adults had a good accuracy and real-time, under the condition that the distribution of the wings of the flies and the distribution of the pectoral fin stripes were clear. It greatly reduces the requirement of image clarity, and is more suitable for dynamic identification of video streaming devices. The recognition accuracy of B.dorsalis was 95.45%, the B.tau Walker was 93.33%, the B.cucurbitae was 97.83%. The overall accuracy rate was 95.56%.The average time of single recognition was about 500ms, which can meet the needs of practical applications. The identification model proposed in this study has good expansibility for Diptera adults.