Damping estimation is difficult as a result of modal interference existing in closely spaced modes, a combinational method of inner product calculation and iterative algorithm was presented. Firstly, resonant frequency of the topmost spectral peak was counted by the application of FFT-FT. Secondly, inner product was conducted between response signal and the constructed exponential function. According to the theorem of Cauchy-Schwarz inequality, the attenuation coefficient was achieved by calculating the maximum of the inner product module, meanwhile, the amplitude and phase of the corresponding modal signal were obtained. One modal signal could be estimated from the above calculations. Thirdly, the estimated signal was subtracted from the overall response signal, and then estimated the next modal signal. In this way, an iteration process was formed by repeating the above diagnosing procedures, which could solve the modal interference. Both simulation calculation and experiment were conducted to validate the method.