In the process of planting sandalwood trees on a large scale, there are problems such as low efficiency, high cost, and difficulty in the supervision of manual ranking of missing seedlings, and the necessary companion plants for each sandalwood tree and other crops interspersed between the trees, further deepening the difficulty of checking and replenishing. For these problems, a seedling deficiency detection and precise localization method in complex environment was proposed based on YOLOv4 algorithm and double regression strategy. Firstly, the YOLOv4 target detection model was used to achieve sandalwood plant detection from remote sensing images collected by UAV. Then the missing seedling localization algorithm (MSL) was constructed based on the double linear regression and extended column line fixing strategy: arbitrary sandalwood trees were selected as the benchmark, column regions were divided according to the pixel coordinates, and column lines were fitted to the sandalwood trees in each column region by using linear regression;for the omitted sandalwood trees that were not classified into columns after fitting, the attribution was judged again with the extended regression line strategy, and the column lines were optimized by linear regression again. Finally, the missing seedlings were calculated and localized according to the spacing at the time of planting. The results showed that the precision was 86.82%, the recall was 82.25%, the F1-score was 84.47%, and the running time was 8.19s, respectively. In summary, this method combined the rapidity of DJI UAV remote sensing image acquisition system, the accuracy of YOLOv4 algorithm and double regression strategy, which can be used to achieve realtime intelligent seedling deficiency detection and accurate localization of sandalwood trees under complex growth conditions.