The scale variation, deformation and occlusion are the important reasons for model drift. In order to overcome the effect of model drift on robust tracking, a multiple model tracking algorithm based on object proposals was proposed. Firstly, as object proposals can reflect the general object material properties, the proposed tracker replaced traditional sliding sampling with object proposals to adapt the displacement and scale variation in the tracking process. And then, in order to enhance the object representation ability, the deep convolutional feature was used to characterize the target. During this process, although the previous size of object proposals may be different, the deep convolutional feature of each object proposal can be extracted quickly by a ROI pooling layer, and each object proposals feature had the same length, which can help to model training and further improve the robustness of the tracker. Lastly, the multi-models selection mechanism was used to undo past bad model updates by selecting the best tracking model, and adjusting the searching area can achieve object re-detection. These measures can inhibit the effect of model drift on robust tracking. In order to verify the superiority of the algorithm, the OTB 2013 benchmark and UAV 20L benchmark, and some classic contrast algorithms recently were used to evaluate the proposed tracker. The results showed that the proposed tracker achieved the best performance on precision and success rate, and the effect of model drift on robust tracking can be effectively suppressed.