Jujube is a high-value fruit throughout the world. Detection for the surface defects of jujubes is the prerequisite to realize its automatic grading. For the difficulty of red jujube location and defect detection in images, a kind of locating method based on search of the shortest path between frames and framework named ensemble-convolution neural network (E-CNN) were introduced. As for locating method, image coordinate system was established at first. With image preprocessing, the location coordinates of each jujube target were obtained and these location coordinates were mapped into the spatial coordinate system. Combining judgment rules of the shortest path between frames, the location coordinates of targets were updated and transmitted with the video time sequences. Also, by using the method with video data, it could be quickly and efficiently to build data sets. Based on “Bagging” ensemblelearning and “returning” training method, the basic convolutional neural network tree models were built and then according to output of each basic tree model, the final result of the model was obtained by “voting”. The results of experiments showed that the location accuracy of 100% was achieved with this locating method, avoiding complicated mechanical and circuit design. At the same time, by using E-CNN model, the average recognition accuracy and recall rate reached 98.48% and 98.39%, respectively. And the classification accuracy was greater than those of color feature classification model (86.62%), texture feature classification model (86.40%), and basic convolution neural network model (95.82%). The model had high recognition accuracy and strong robustness, and can provide reference for other agricultural products sorting and detection.