Aiming at the problems of large diversity of defective eggs, as well as the strong subjectivity and poor real-time detection of artificial detection, and the potential risk of food safety for end-consumers, a non-destructive testing system based on deep learning for defective eggs on mobile device was proposed to realize real-time detection of cracked eggs and bloody eggs. An improved lightweight convolutional neural network MobileNetV2_CA model was firstly established. MobileNetV2 network was taken as the original framework, it was further optimized by embedding coordinate attention mechanism, adjusting width factor, transfer learning and other parameters. The PC detection was also performed for comparison. Results showed that the MobileNetV2_CA model presented the validation accuracy of 93.93%, the recall rate of 94.73%, and the average detection time of 9.9ms for a single egg, which was 3.60 percentage points higher, 4.30 percentage points higher, and 2.62ms shorter than the original MobileNetV2 model, respectively. The parameter score of MobileNetV2_CA model was only 2.36×106, which was 31.59% lower than the original MobileNetV2 network model. In addition, the NCNN deep learning framework was used to train MobileNetV2_CA model, which was further applied to Android mobile terminal through format conversion. The verification of mobile terminal detection of NCNN deep learning training model was investigated and compared with TensorFlow Lite deep learning model. Results showed that the NCNN deep learning model had an average recognition accuracy of 92.72%, an average detection time of 22.1ms for a single egg, and the library file size of 2.7MB, indicating its better performance than TensorFlow Lite and meeting the requirement of practical applications. The effectiveness of the proposed system based on deep learning was finally demonstrated.