Effective detection of tea buds is an important prerequisite for improving the precision of mechanical picking and planning the picking route to avoid harming tea plants. Considering the problems of low detection accuracy, poor robustness and slow speed of traditional target detection algorithm in complex background, Faster R-CNN was applied to recognize tea bud in complex background. Firstly, collected pictures were processed by equal cutting, label making and data enhancement to make VOC2007 dataset. The deep learning model on detecting tea bud types (single bud and one bud with one leaf/two leaves) was trained after setting up the environment and adjusting the model parameters, and the trained model was evaluated. The results showed that the average precision (AP) was 54%, and the root mean square error (RMSE) were 3.32 when the tea bud type was not distinguished. When distinguishing tea bud types, the AP of single bud and one bud with one leaf/two leaves were 22% and 75%, with RMSE of 2.84. When single bud was removed, the AP of one bud with one leaf/two leaves was 76%, with RMSE of 2.19. Compared with tea bud detection algorithm based on excess green and image binarization (traditional target detection algorithm), the deep learning target detection algorithm was superior to traditional target detection algorithm, with RMSE of 5.47, in accuracy and speed, especially under complex background. Deep learning algorithm demonstrated an important application prospect in realizing tea bud detection and automatic picking in intelligent tea garden image real-time detection system.