Compared with the artificial sensory evaluation method, the tea bud grading based on deep learning and computer technology can reduce the time cost and greatly improve the accuracy, but the commonly used recognition model has the problem of large redundant calculation and large model specifications. For this reason, the tea buds picked from the Hongfeng Mountain Yun Tea Farm in Guizhou were used as the research object, and the tea samples were divided into three grades based on the workers’ experience. The multiscale convolutional block attention module (MCBAM) and multiscale depth shortcut (MDS) were embedded in the ShuffleNet-V2 0.5x basic unit, a tea bud grading model (ShuffleNet-V2 0.5x-SMAU) was proposed, which focused on the feature information in tea samples that was conducive to grading. The models pre-trained on two different source domains was taken as the teacher and student model. A tea bud grading method was proposed which combined dual migration and knowledge distillation. With the help of dark knowledge, the classification performance of the grading model and the ability to resist over-fitting were further enhanced. The results showed that the classification accuracy of the method can achieve 100%, 92.70% and 89.89% respectively for the three different grade samples in the test set under the condition of ensuring the lightweight of the model, which was better than the comprehensive performance of the complex network model. The application was more advantageous in production scenarios with limited storage resources and low hardware levels.