In order to solve the difficulty of fruit recognition in near color background, an algorithm for identifying and locating fruits from the depth image was presented based on the near-spherical morphological characteristics of fruits. A depth camera was used to get depth image of a fruit tree. The gradient vectors of each pixel point from the depth image were calculated. The gradient vector was considered as a vector field of motion and the divergence of the vector field was calculated. Searching for divergence center points from vector fields according to the principle of maximum divergence. The fruit center point was selected from the divergence center point by using the difference of the contour image between the fruit and the leaf. The fruit boundary points were searched in eight directions from the center point of the fruit. The fruit images in the closed area formed by connecting the fruit boundary points were imported into the point cloud. Finally, the point cloud was used to find the fitting circle of the target fruit according to the random sample consensus (RANSAC) algorithm, and the size and spatial location of the fruit were obtained. The algorithm discarded the color features commonly used in traditional algorithms but used only the depth information in the depth image for fruit recognition and positioning. It can overcome the drawbacks of traditional algorithms affected by color, illumination and other factors. Because the algorithm did not use color image information at all, it can not only recognize and locate green fruits, but also enable the harvesting robot to work in dark environment. The research result can provide a method for fruits recognition and location in complex environment.