Aiming at the ambiguity and uncertainty of lesion field image border, using inference rule of fuzzy logic and self-adaptive of neural network, the self-adaptive and fuzzy neural network model was proposed to be the decisionsystem for extracting the diseased spots, and the initial values of adjusting parameters were optimized by using genetic algorithm which enhanced the speed of network training, overcame the local minimum of traditional gradient descent method. The experimental result showed that model had many advantages including accuracy, convergence, stability, robustness, and was easy to implement when implied in extracting the diseased spots of potato early blight.