Soybean is an important crop in agriculture and plays an important role in the agricultural field of the world. With the increase of population and disasters caused by abnormal climate, how to cultivate more adaptive high-yield crop varieties has become a major problem faced by breeding experts. A soybean detection method was proposed based on deep learning to reduce the influence of illumination, growth difference and occlusion. In view of characteristics of soybean and accuracy of deep learning, single shot multibox detector (SSD) was improved. Compared with the SSD, the improved SSD (IM-SSD) had better anti-interference ability and self-learning ability. The first step was to build datasets by taking 3695 photos of harvested soybean plants under the fixed and defined light condition and blue background described. And the training set was randomly changed by images translation, rotation and scaling to expand the datasets. Then training datasets were used to train and construct network model. To study the effect of base network and the input size of networks, one contrast experiment was designed on SSD and IM-SSD. Through the analysis of the experimental results,the average accuracy of IM-SSD was 7.79 percentage points higher than that of SSD300 and 3.83 percentage points higher than that of SSD512, respectively. Compared with SSD, IM-SSD was improved in soybean pod and stem detection. Nevertheless, the location of the stem by IM-SSD was discontinuous. A method of stem extraction was proposed, which used IM-SSD results and ant colony optimization (ACO) algorithm to extract the whole stem. The experimental results showed that the IM-SSD and the stem extraction method could accurately locate pod and stem of soybean plants. Finally, some phenotypic information of soybean plants was obtained, including the number of pods of the whole plant, plant height, effective branch number, main stem and plant type.