Tea is a high-value crop throughout the world. Most fresh tea leaves are picked by machines, then various grades are mixed together including broken leaves and leaf stalks. In order to improve quality, the fresh tea leaves picked by machines need to be further classified. However, traditional methods such as winnowing and screening can only sort tea leaves roughly. A new kind of intelligent fresh-tea-leaf sorting system was proposed based on computer vision technology and deep learning method, which can identify and sort tea leaves automatically and accurately. In this system, convolution neural network (CNN) was used to recognize the images of fresh tea leaves, and there was a seven-layer network structure in the CNN identification model. Through image segmentation and scale transformation, the original image was normalized as the input of CNN. CNN was able to learn the characteristics of images independently and can avoid many complicated feature extraction. The preprocessed images were rotated and mapped to serve as the training set, which enhanced the generalization ability of CNN identification model. Meanwhile, the training performance was greatly improved by sharing weights and using a declining learning rate. Experiment results showed that the system can effectively sort out several kinds of tea leaves, single bud, a bud with a leaf, a bud with two leaves, a bud with three leaves, single leaf and leaf stalk. The identification accuracy was more than 90%.