In order to detect the mango yield on trees rapidly, a green mango visual detection method based on unmanned aerial vehicle (UAV) was proposed. The deep learning technology and the YOLOv2 model were adopted to detect the mango images captured by UAV. Firstly, totally 471 images of the mango on trees were collected by the UAV. To meet the demand of diversity, totally 360 images included different shooting distances and different lighting situations were selected. Among which, 300 images were selected randomly as the training set, the other 60 images were used as the test set. Also, the shooting plan of the whole tree was designed. By image collecting and image mosaic, the integrated images of five mango trees were worked out for the yield estimating experiment of mango. After image collection, these images were marked manually and used to build the training set and the test set. The batch size and the initial learning rate were determined by experiments. During the model training, the learning rate was reduced gradually as the training times were changed. The mean average precision (MAP) of the trained model on the training set was 86.43%. By designing the experiments, the accuracy of mango recognition with images that containing different fruit numbers and different lighting conditions was worked out. Also, the yield estimation experiment was designed. The experimental results showed that the average running time of an image using the given algorithm was 0.08s, while the accuracy of the teat set was 90.64% and the false recognition rate was 9.36%; the highest recognition accuracy of image with different numbers of fruits was 94.55% and the lowest was 88.05%. The recognition accuracy was 93.42% under the condition of direct sunlight, and the recognition accuracy was 87.18% under the condition of backlight. The average error of the yield estimation of mango tree was 12.79%. The result demonstrated that the algorithm was effective for mango in natural environment, which can provide technical support for estimating the yield of fruits and vegetables in intelligent agricultural production.