Behavior recognition and tracking of group-housed pigs are an effective aid to monitor pigs’ health status in smart farming. In real farming scenarios, it is still challenging to automatically track the behavior of group-housed pigs by using computer vision techniques due to the pigs’ overlapping occlusion and illumination change, which cause the identity (ID) of pig to switch wrongly. To improve the situation, an improved DeepSORT algorithm of behavior tracking based on YOLO v5s was proposed. The improvement of the algorithm included two parts. One was that the trajectory processing and data association were improved in the scene where there was a fixed number of pigs. This reduced ID switch and enhanced tracking stability. The other was that the behavior information from YOLO v5s detection algorithm was introduced into the tracking algorithm, thereby achieving behavior recognition of pigs in tracking. The experimental results showed that YOLO v5s algorithm had a mAP of 99.3% and an F1 of 98.7% in object detection. In terms of re-identification, the Top-1 accuracy of the experiment was 99.88%. In terms of tracking, the method achieved a favorable performance with a MOTA of 91.9%, an IDF1 of 89.2% and an IDS of 33. Compared with the original DeepSORT algorithm, the proposed method improved 1.0 percentage points and 16.9 percentage points in MOTA and IDF1 respectively, and decreased 83.8% in IDS. This showed that the improved DeepSORT algorithm was able to achieve behavior tracking of group-housed pigs with stable ID. The method can provide technical support for no-contact automatic monitoring of pigs.