In the research on smart pig breeding, the pig instance segmentation method is one of the key technologies to realize automatic detection of pigs. However, in actual segmentation scenarios, there is occlusion and adhesion phenomenon, which makes pig segmentation difficult. Aiming at the difficulty of piglet segmentation in the farrowing room, an instance segmentation model OD_SeGAN was proposed based on YOLO v5s and generative adversarial network(GAN). This method extracted the piglet target through the target detection algorithm YOLO v5s, and inputed it into the semantic segmentation algorithm GAN to achieve segmentation, and used dilated convolution to replace the ordinary convolution in GAN to expand the network receptive field; secondly, a squeeze-incentive attention mechanism was used module to enhance the model’s ability to learn the global characteristics of piglets and improve the model’s segmentation accuracy. Experimental results showed that OD_SeGAN’s IoU on the test set was 88.6%, which was 3.4, 3.3, 4.1, 9.7, and 8.1 percentage points higher than YOLO v5s_Seg, Cascade_Mask_RCNN, Mask_RCNN, SOLO, and Yolact, respectively. OD_SeGAN was applied to the piglet litter average weight estimation task, and the Pearson correlation coefficient between the piglet litter average weight and the number of piglet pixels was measured to be 0.956. The OD_SeGAN proposed had good piglet segmentation performance in actual production scenarios, and can provide a technical basis for subsequent research such as piglet litter weight estimation.