Forest plays an important role in the terrestrial ecosystem. Acquirement of tree height has vital significances in global ecology analysis, environmental protection, national economic construction and other related fields. In the past decades, polarimetric interferometric synthetic aperture radar (PolInSAR) has been widely treated as an advanced microwave remote sensing technique to extract vegetation information of the earth surface. However, the existing methods for tree height inversion suffer seriously from the deviations of coherence estimation, especially in the case of high-resolution SAR images. In order to improve the accuracy of tree height estimation with single-baseline PolInSAR, a new tree height extraction algorithm with improved coherence estimation based on nonlocal sample selection was proposed. The optimized interferogram was firstly acquired from four pairs of quad polarimetric SAR images with the proposed algorithm, and then the independent identical distribution samples were selected which were adopted to estimate accurate coherence, and at last the classical three-stage height inversion processing procedures were employed to obtain the vegetation height. In the experiment, multiple pairs of PolInSAR data were simulated through the PolSARPro software designed by European Space Agency (ESA). From the experiments, it can be concluded that the proposed algorithm has the capacity to maintain the boundary information effectively and the accuracy of tree height estimation was increased by nearly 5%.