The flowering intensity of litchi can directly affect the yield and quality of the fruit, so the detection of litchi flowers is very important for orchard planting strategies. Dense litchi flower detection has important challenges due to serious occlusion. Existing research methods ignore the interaction between dense suggestion boxes in the process of non-maximum suppression. In order to improve the detection precision and reduce the missed detection rate, a detection method was proposed based on polyphyletic loss function. This method included an aggregation loss term in the loss function to force the proposal box to approach and compactly locate the corresponding object. At the same time, the segmentation loss of the bounding box specially designed for the dense crop scene was added to keep the prediction box away from the surrounding objects and improve the robustness of detecting a large number of flowers. Compared with Faster R-CNN, Focus Loss, AdaptiveNMS and Mask R-CNN, the experiment showed that the recognition precision of this method on the standard apple blossom dataset was about 2 percentage points higher than that of other methods, which verified the detection versatility of this method. At the same time, the mean average precision of this method in the self-built litchi flower dataset was 87.94%, the F1 score was 87.07%, and the miss rate was 13.29%. Compared with Faster R-CNN, Focus Loss, AdaptiveNMS and Mask R-CNN, the accuracy of the method was improved by 20.09 percentage points, 14.10 percentage points, 8.35 percentage points and 4.86 percentage points, respectively, with high detection performance. Therefore, the method proposed can effectively detect the dense litchi flowers, and provide an important reference for dense crop detection in complex scenes.