Early prediction for the growth and development of plants was an important component of the intelligent breeding process. However, it is difficult to accurately predict and simulate plant phenotypes. A prediction model of plant growth and development was proposed based on spatiotemporal long short-term memory (ST-LSTM) to predict future growth and development of plant. Firstly, the plant masks were recognized and extracted by the pre-trained Mask R-CNN model and the background of the plant image was removed by morphological operations. Then, the plant growth and development prediction data set was constructed. After that, utilizing the spatial and temporal dependence of plant growth and development, the image sequence of plants future growth and development was predicted by the prediction model for plant growth and development using the spatial and temporal depth characteristics integrated from the image sequence of early plant growth and development. The results showed that the image sequence predicted by the proposed model had high consistency and similarity with the actual image sequence of growth and development. At the first prediction time node, the structural similarity index measure was 0.8741, the mean square error was 17.10, and the peak signal to noise ratio was 30.83. The prediction determination coefficient (R2) of canopy leaf area, crown width, and leaf number were 0.9619, 0.9087 and 0.9158, respectively. Finally, the research realized the prediction of growth and development based on the image sequence of plant growth and development, which would effectively reduce the time, land and labor cost of repeated experiments in the field, and provided a reference for improving breeding efficiency.