The precise recognition of tea buds is the prerequisite and foundation of intelligent bud picking. The method of bud identification using vision and deep learning is gradually established, but it relies excessively on high-performance hardware, which is not conducive to the deployment of mobile tea picking robots. To solve this problem, a mobile recognition solution for tea buds based on the Compact-YOLO v4 algorithm was proposed. Firstly, the Backbone and Neck networks of the YOLO v4 algorithm were improved by replacing the Backbone network with GhostNet and the traditional convolution in the Neck network with Ghost convolution, the size of the improved model was only one-fifth of the original one. Secondly, the training method of transfer learning was applied to improve the model accuracy. The experiments showed that the Compact-YOLO v4 algorithm model had P, R, mAP and F1 score values of 51.07%, 78.67%, 72.93% and 61.45%, respectively. Finally, the algorithm model was transplanted to the PRO-RK3568-B mobile development board to reduce the hardware performance requirements of the model inference calculation by three ways: converting the model, quantization processing, and improving the deployment environment. The aim of optimizing the model inference process and reducing the pressure on edge computing on mobile was eventually achieved, while also ensuring the accuracy of tea bud recognition, providing a theoretical and practical basis for the practical application of tea bud picking robots.