Rice disease is one of the important factors affecting rice yield. Early prediction of rice disease is very important for rice disease prevention. In order to realize the prediction of rice bacterial leaf blight disease, hyperspectral images of leaves under the stress of bacterial leaf blight disease were collected continuously for seven days from inoculation to early onset. The Savitzky-Golay algorithm was used to preprocess hyperspectral images, and the principal component analysis (PCA) and random forest (RF) algorithms were used to extract spectral features. The prediction model of multi-task learning (MTL) and long-short term memory (LSTM) network fusion was constructed to predict the incidence rate and incubation period of rice diseases. The MTL-LSTM model was optimized by using the whale optimization algorithm (WOA). The experimental results showed that PCA and RF can effectively extract spectral features from hyperspectral and reduce the dimension of hyperspectral images, and the performance of the prediction model based on spectral features was better than that of the prediction model based on full spectra. The modeling time of the former was about 98% lower than that of the latter. The prediction model constructed based on time series hyperspectral achieved the expected results in the prediction of the incidence probability and latency. The WOA-MTL-LSTM model, constructed based on the first ten characteristic wavelengths, achieved the best prediction performance. The R2 of the test set for the prediction of the incidence probability and latency was 0.93 and 0.85, the RMSE was 0.34 and 2.12, and the RE was 0.33% and 1.21%, respectively. The prediction performance of MTL-LSTM can be improved by WOA algorithm, and the R2 of disease probability and incubation period was increased by 0.05. The results indicated that RF extracted characteristic wavelengths can effectively characterize the full spectrum. The WOA-MTL-LSTM model based on time-series hyperspectral can accurately predict the incidence rate and incubation period of bacterial leaf blight disease, which provided technical support for the prevention of rice bacterial leaf blight disease.