Fruit and vegetable production occupy an important position in agriculture with wide market and huge economic benefit. Currently, due to the diversity of picking object, most of fruit harvesting in our country depends on manual work. It’s not only time-consuming, but also technic-demanding. The labor cost of harvesting tends to occupy one-third to one-half of the whole labor cost in fruit production process. Thus, fruit harvesting robot needs to be developed to increase the efficiency and lower the costs. Since the working task of harvesting robot grows in natural environment with various shapes and complex structure, visual system needs to be built to recognize the target. This article focusing on litchi picking process, a visual system for litchi images was built and used to recognize litchi. Firstly, a visual system for litchi picture acquisition was built and a method of nighttime litchi recognition and picking point calculation was proposed. For comparison, pictures of same cluster of litchis were captured at daytime with different natural illumination and nighttime with artificial illumination. By analyzing color features of same litchi picture in different color models, the YIQ color model was proved to be the model with best practicability for nighttime litchi recognition and picking point calculation. The background of nighttime picture was firstly removed using Otsu algorithm, then fruit was segmented from stem using Fuzzy C-means clustering algorithm. Circle detection was performed to recognize fruits respectively using Hough circle fitting method. The experiments showed that nighttime litchi recognition accuracy was 95.3% with the average recognition time of 0.46s, and the method for litchi recognition at night time had better accuracy and higher real-time. This research provided technical support for visual localization technology of litchi picking robots. Based on machine vision, the recognition of litchi fruit was realized. It could provide technical support for litchi picking robot, bring practical significance with high harvest efficiency and low labor cost.