The algorithm optimization of hybrid grid for centrifugal pump optimizes tetrahedral mesh and tri-prism grid separately, which will lead to low quality and efficiency. To solve this problem, a tri-prism grid quality optimization objective function balancing triangle shape and orthogonality of prismatic side was established. Besides, the effects of six different methods of solving optimal solution on the results of optimizing hybrid grids were compared. Moreover, the impact of weight coefficient on optimization was also studied and the optimization results between the algorithm and a commercial software were compared. Results showed that the optimization results and efficiency were better when the conjugate gradient method and dichotomy were combined and weight coefficient value was 0.2. With the optimization algorithm, the quality of the worst grid cell and the whole grid was increased, comparing with the commercial software. In addition, the quality of the worst grid cell with the optimization algorithm was also increased compared with the commercial software. It indicates that the optimization algorithm is better than the existing algorithms.