The MobileViT as the main feature extraction network was employed in order to accomplish quick and precise post-harvest screening of loquats in the paper. A lightweight network model called MobileViT-CBAM was developed as a result of strengthening the network’s capacity to extract detailed features in both channel and spatial dimensions by inserting convolutional block attention module (CBAM) after Layer1 and Layer2. The method outperformed MobileViT in terms of defect recognition accuracy, showing gains of 1.17 percentage points on the validation set and 1.23 percentage points on the test set for things like scars, mechanical damage, and decaying fruits. According to experimental results, the MobileViT-CBAM model performed better in terms of accuracy (97.86%) than VGG16, ResNet34, and MobileNetV2. It also had the advantage of having a small memory footprint (3.768 MB) and a rapid inference time (42 ms per image). It was possible to use this lightweight network model on embedded systems. The research offered an effective and precise technique for external quality inspection of loquats and other agricultural products by providing a theoretical framework for fault recognition in the construction of an online detection system for loquats.