In response to the problems of low efficiency and strong subjectivity in obtaining information about the current stage of wheat development that relies on manual observation, a wheat image dataset consisting of four key growth stages of winter wheat: winterovering stage, green-turning stage, jointing stage, and heading stage, totaling 4599 images were constructed. A lightweight model FSST (fast shuffle swin transformer) based on FasterNet was proposed to carry out intelligent recognition of these four key growth stages. Firstly, based on the partial convolution of FasterNet, the Channel Shuffle mechanism was introduced to improve the computational speed of the model. Secondly, the Swin Transformer module was introduced to achieve feature fusion and self attention mechanism, it can improve the accuracy of identifying key growth stages of wheat. Then the structure of the whole model was adjusted to further reduce the network complexity, and the Lion optimizer was introduced into the training to accelerate the training speed of the model. Finally, model validation on the self-built wheat dataset with four key growth stages was performed. The results showed that the parameter quantity of the FSST model was only 1.22×107, the average recognition accuracy was 97.22%, the F1 score was 78.54%, and the FLOPs was 3.9×108. Compared with that of the FasterNet, GhostNet, ShuffleNetV2 and MobileNetV3 models, the recognition accuracy of the FSST model was higher, the operation speed was faster, and the recognition time was reduced by 84.04%, 73.74%, 72.22% and 77.01%, respectively. The FSST model proposed can effectively identify the key growth stage of wheat, and had the characteristics of fast, accurate, and lightweight recognition. It can provide a reference for optimizing the application of deep learning models in smart agriculture and offerring information technology support for real-time monitoring of field crop growth on resource-constrained mobile devices.