Fresh tea leaf and stem separation for coarse tea not only improves the quality of finished tea, but also reduces heat energy consumption. To realize the separation, the key issue is the recognition of tea stems and petioles. Tilted chute was used to constrain the direction of fresh tea. Then, the problem was transformed into determining whether the tea sprout was facing up or down. At first, according to the stem end length, fresh tea was classified into two categories: for those with long stem ends, an algorithm based on the feature size was proposed, and the side with longer feature size was recognized as tea sprout. For those with short stem ends, the “local extreme points of fresh tea contour” was utilized, and the side with more local extreme points was recognized as tea sprout. Then, 168 randomly obtained images and a background image was used to validate the algorithm, and a recognition rate of 93.3% was reached. The average time to recognize the direction of a single fresh tea was 17.8 ms which fulfilled the requirements of real-time processing.