The small-scale unmanned helicopter is well-known by its hovering capabilities. However, it exhibits a nonlinear and complex dynamic phenomenon, and it is a loop unstable, high degree of inter axis coupling system. The goal of autonomous flight was realized based on an accurate and appropriate helicopter model. And system identification is the practical method to obtain the model. Aiming at the system identification of a small-scale unmanned helicopter in hover condition, a novel algorithm combined prediction error method with artificial bee colony algorithm (PEM-ABC) was proposed. In the proposed algorithm, the problem of system identification was turned into an optimization problem. The search scope was arranged by the PEM algorithm; in that case, the initial solutions can be obtained. And in the stage of employed bee search, an adaptive search strategy was adopted to increase the speed of convergence. In the stage of following bee search, a new probability of selection strategy was introduced to keep the diversity of the population. And in the stage of scout bee search, the chaotic search operator was used to improve the ability of global search. Through the actual flight data collected by airborne equipment, the model used in system identification was validated and analyzed. The results show that the unknown parameters can be estimated based on the proposed algorithm. Compared to PEM algorithm and traditional ABC algorithm, the identified accuracy of the proposed algorithm was better, which showed an important engineering application value.