In order to accurately predict the nitrogen content in different scales of apple leaves at flowering, young fruit and fruit expansion periods, a combined color characteristics based prediction model of apple leaf nitrogen content was proposed. Firstly, the image of apple leaves was obtained and 17 color features, including R,G,B monochromatic components and 14 color combination parameters were extracted, and the key influencing factors of nitrogen content of apple leaves in different periods were extracted by principal component analysis to eliminate the correlation between the original variables and reduce the input vector dimension of the model. Secondly, the PCA-SVM, PCA-BP and PCA-ELM prediction models were established in different periods, the prediction effect and accuracy of apple leaf nitrogen content were compared, and the best prediction model in different periods was obtained. Finally, the best prediction model was used to predict the nitrogen content of apple leaves in different periods, and the parameters of the best prediction model were optimized by adaptive genetic algorithm. The results showed that the prediction accuracy of PCA-SVM model was higher than that of PCA-BP and PCA-ELM model in different growth periods; the mean absolute error of PCA-SVM prediction model in flowering period, young fruit period and fruit expansion period was 0.640 g/kg, 0.558 g/kg and 0.544 g/kg, and mean absolute percentage error was 0.057 g/kg, 0.050 g/kg and 0.064 g/kg, and root mean square error was 0.800g/kg, 0.747 g/kg and 0.737 g/kg, which was better than that of the prediction model before optimization. The model had good prediction performance and generalization ability, which can provide theoretical basis for orchard precision fertilization management, improving fruit quality, avoiding resource waste and environmental pollution.