The present method has some difficulties in recognizing cassava leaf diseases in a field environment, such as covering and stacking between leaves. Based on the YOLOX network, cassava leaf disease detection (CDD) model was proposed. Firstly, the cassava leaf disease image data under complex background was augmented to reduce the recognition difficulty caused by environmental impact. Secondly, built on the YOLOX network, the lightweight multi-scale feature extraction (LME) module was used to strengthen fine-grained feature extraction and reduce the amount of model calculation. At the same time, the channel attention mechanism was embedded to improve the representation ability of the network. Finally, the quality focal loss was used as a part of the classification loss to assist the network convergence and improve the accuracy of target classification. In conclusion, the proposed CDD model can detect cassava leaf disease under complex background. The amount of network parameters was 5.04×106 and the mean average precision was 93.53%, which was 6.02 percentage points higher than that of the non-optimized network model. Comprehensive detection ability was better than that of previous models. Therefore, the proposed method CDD had faster and more accurate detection ability for cassava leaf diseases in the field, and provided a reference method for realizing intelligent field detection.