CFD was used to study the mechanism of the performance curve instability of the mixed flow pump with vane diffuser characterized by the positive slop of the head-flow curve. The two parts of performance curve instability including the dramatically head drop and a slightly rise of the head when the flow rates decreased further. The reason of the two parts was that the decreasing of the effective diameter of the impeller blade TE was caused by the appearance of the vortices in the tip of the impeller outlet which was caused by the reverse flow. The flow separation and the flow pattern of the tested pump shifted to the centrifugal pump which led to the strong vortices in the impeller hub respectively and it was got by steady simulation. The unsteady simulation was used to study the reason of reverse at hub of impeller outlet and it was found that the stall is the main reason for the mixed flow pump shifting to the centrifugal pump. It was also found that asymmetry of pressure at vane inlet is the main reason of the stall for its propagation.