In order to lay a certain foundation for optimization of mechanical structure vibration reduction, the viscoelastic damping material optimal layout was studied. Damping materials dosage was restricted to realize structural lightweight design. Based on the Hamilton variation principle and the deformation displacement relationship of damping structure, structure vibration differential equation was deduced. A topological optimization mathematical model for constrained damping structure that used structure maximum damping ratio as the optimization goal, structure damping materials consumption as constraint condition, structure damping element status as design variables was built. To search structure topological optimization iteration direction, the modal damping ratio of sensitivity was derived. The constrained damping element adding and deleting criterion was presented, the program for constrained damping structure was established, and independent mesh filter method was adopted to avoid the checkerboard. The bi-directional evolutionary structural optimization (BESO) can improve the utilization efficiency of damping materials, thus BESO was used to optimize the damping structure. After evolutionary structural optimization(ESO) for damping structure, the increases of the first and the third order damping ratios were 54.51% and 36.21%, respectively, while after BESO for damping structure, the increases were 76.69% and 58.36%, respectively, and the phenomenon of checkerboard was improved, the damping material layout was consistent with structural strain energy figure. To verify the result of topology optimization for damping layout, harmonic response analysis was carried out on the constrained layer damping structure, the simulation results showed that the structural response amplitude was low and vibration suppression effect was good by means of BESO method.