The automatic classification and recognition of tree image has important practical application value. Relevant research on traditional tree species recognition includes leaf recognition, flower recognition, bark texture recognition, and wood texture recognition. In order to solve the problem of recognizing the tree image with complex background in nature scenes, a tree species recognition method based on the overall tree image and ensemble of transfer learning was proposed. Four pretraining models of AlexNet, VggNet-16, Inception-V3 and ResNet-50 were firstly used on ImageNet largescale datasets to extract features. They were then transferred to the target tree dataset to train four different classifiers. An ensemble model was finally established by the relative majority voting method and the weighted average method. A new tree image dataset called TreesNet was built and experiments were designed based on the dataset, including the comparative experiments of transfer learning and conventional methods.The experimental results showed that data augmentation can effectively solve the overfitting problem and the training model had better generalization ability and higher recognition rate. The image recognition accuracy of the tree species in the complex background with the method proposed reached 99.15%, which had a better effect on overall tree image recognition compared with the conventional classification methods of Knearest neighbor (KNN), support vector machine (SVM) and back propagation neural network (BP).