The hump characteristic is one of the detrimental characteristics of axial flow pump. If the pump runs in the hump region, some unsteady flows will occur, such as stall, backflow, cavitation and so on, which will reduce the pump performance. In order to analyze the hump characteristics, numerical method based on the RANS equations and SST k—ω turbulence model was used to simulate the instability flows in the axial flow pump with and without inlet guide vane. The hexahedral meshing scheme was used for the whole computational domain, particularly with O-grid around blades. The characteristics of swirling backflow vortex were firstly investigated. The influences of different distances between inlet guide vane and impeller blade on the swirling backflow vortices and the pressure fluctuations were studied. The effects of variable angles of inlet guide vane on pump performance were also discussed. The results showed that the predicted results were in good agreement with the experimental data. When the flow rate decreased, the backflow region was extended upstream. During the development of the backflow, the swirling vortices occurred in the shear layer between the swirling backflow and the main flow, which caused a lot of low frequency pressure fluctuations and consumed energies, leading to pump head decrease and hump characteristic. The swirling backflow vortices can be broken by inlet guide vane, resulting in decrease of amplitudes of low frequency pressure fluctuations and increase of pump head due to less energy consumed by the swirling vortices. Thus, the hump characteristic of axial flow pump can be improved by inlet guide vane. With the reduction of the distance between inlet guide vane and impeller blade, pump head was increased continuously. In addition, pump head was increased at negative angle of inlet guide vane, while it decreased at positive angle of inlet guide vane, which was resulted from the prewhirl regulation.