Citrus fruit infested by Bactrocera dorsalis can cause consumer panic and huge economic losses, which makes it important to sort it out during processing. Since there are no obvious characteristics on the fruit surface and manual sorting usually features low efficiency, new techniques for automated sorting are needed. The feasibility of combining an agricultural X-ray machine and an RGB camera on the processing line was explored for non-destructive detection. A multi-modal data fusion method using X-ray and RGB images was firstly proposed, and a CNN-LSTM detection model was then developed which can detect the fruit infested by Bactrocera dorsalis with high precision. The process of the fruit rolling on the processing line was simulated and six X-ray and RGB sequential images were captured respectively, which formed the dataset. The effectiveness of multi-modal data fusion was verified by integrating it into four lightweight detection models, including ResNet18-LSTM, GoogleNet-LSTM, SqueezeNet-LSTM and MobileNetV2-LSTM. Results showed that for each network, the performance using multimodal data fusion outperformed that using unimodal data. ResNet18-LSTM obtained the highest detection accuracy, reaching 97.3% by using multi-modal image fusion and 95.7% by using feature fusion, respectively, and the accuracy based on single-modal X-ray and RGB data was 93.2% and 89.3%, respectively. These results demonstrated the potential to develop an online non-destructive detection system for citrus fruit infested by Bactrocera dorsalis.