A visual positioning system of litchi picking manipulator was established based on the binocular stereo vision. By analyzing the color features of mature litchi, the YCbCr color model was selected. The Otsu algorithm was combined with fuzzy C-means clustering (FCM) to segment litchi fruit and peduncle. The experimental results showed that the effective recognition accuracy of litchi fruit and peduncle reached to 94.2%. The maximum distance between the fruit centroid and peduncle was calculated to determine litchi picking point. The color image matching method based on hue component of color image and the limiting binding method were used to achieve the stereo match of picking point, which realized the spatial position of picking point. Through the analysis of position error, the linear interpolation method was used to realize the positioning error compensation. The positioning experimental results showed that the depth value error was less than 10mm, which could satisfy the visual accurate position requirements of litchi manipulator.