In the process of fine breeding of dairy goats, the accurate and real-time tracking of goat targets is an important basis for their behavior recognition and disease abnormality warning. Based on the DiMP tracking model, a kind specific target tracking model was designed for dairy goats, which can effectively overcome the disadvantage of insufficient positioning accuracy of DiMP algorithm in tracking specific targets. The migration training of the tracking algorithm was carried out by using the constructed dairy goat video tracking data training set to accelerate the convergence speed of the model and make the boundary box predicted by the evaluation network more fit the position and size of the real frame of the dairy goat. In the online tracking stage, aiming at the disadvantage that the target template only used the first frame features to produce the modulation vector of the whole sequence, which led to unrepresentative characteristics of the modulation vector relative to the whole tracking stage, the training set was used to produce the class modulation vector containing various poses of the dairy goat, and the proportion between the class modulation vector of the dairy goat and the modulation vector of the first frame was updated by exponential ablation to enhance the discrimination between characteristics and background of dairy goats in the boundary box regression task. The AUC and accuracy of the proposed algorithm on the test set were 76.20% and 60.19%, respectively, which were 6.17 and 14.18 percentage points higher than that of the DiMP method. The tracking speed was 30 frames per second (f/s),which met the requirements of real-time tracking. The experimental results showed that the proposed target tracking method can be used to monitor the movement of milk goats in complex scenes, and it can provide technical support for fine management of dairy goats.