It is a big problem for China to have not enough workforces in picking tea sprout today, which usually result in missing the tea sprout's optimum picking period. The picking machine currently used improves the picking efficiency indeed. However, the old leaves are picked together with the sprout, and the sprout is damaged at some time. Hence it is necessary to develop an intelligent picking machine that can recognize and pick the sprout with the minimum sprout damage. This paper presents an image process method based on the tea sprout color and shape features, which was used to recognize the “Wuzi Xianhao” (a famous green tea of Shaanxi, China) tea sprout grown at Xixiang County of Shaanxi Province. Firstly, the green component of the tea leaf image in RGB color space was extracted. Secondly, green component image was segmented using double threshold method. Finally, the edge of tea sprout is detected according to its shape feature. Experimental results indicate that this recognizing method of the tea sprout based on color and shape features can reach the right detection rate of 94%, and provide an effective method for automatic tea sprout picking.