In order to detect plant leaf element deficiency, a visual detection method of soybean leaf element deficiency based on neural network was proposed. After analyzing the characteristics of soybean deficient leaves, deep learning technology was used. The Mask R-CNN model was used to segment the leaf images collected by a fixed camera, and the VGG16 model was adopted to classify the deficient leaves. Firstly, after collecting hydroponic soybean images, the outline of soybean leaves was marked manually in the images, establishing training set and test set of segmentation task. Through pre-training, the initial parameters of the Mask R-CNN model were determined, and then using lower learning rate to train the model. For segmentation task on single leaf images and on multiple leaves on a complex background in the test set, the Matthews correlation coefficient (MCC) of the final trained model reached 0.847 and 0.788 respectively. The training set and test set of the soybean leaf image classification task were established by segmenting the leaves through the trained Mask R-CNN network and manually marking them. The initial parameters of the VGG16 model were determined through pre-training, and then the whole connection layers of the VGG16 model were replaced before training to adapt to the leaf classification task. The classification accuracy of the final trained model on the test set was 89.42%. When analyzing the result, the leaves with obvious deficiency features were classified into two types of nitrogen deficiency and four types of phosphorus deficiency to discuss the inadequacy of the method. The average running time of the algorithm to detect a picture of 1 million pixels was 0.8s. The algorithm had a good detection result on the classification of soybean leaf deficiency under complex background, which can provide technical support for the estimation of plant deficiency in agricultural automation production.