Aiming to achieve automatic individual pig’s tracking and monitoring in pig group, an RGB-D image segmentation method based on the double-pyramid network was proposed to solve the segmentation difficulties caused by overlaps and adhesion body areas which were frequently exiting in images because of habits of huddle and crowd in piglets. The method was based on an instance segmentation network Mask R-CNN, modifying its feature extraction network, ResNet101, to a double-pyramid structure. Features were extracted from RGB and Depth images and combined to be inputted into a regional generation network. The network outputted regions of interest (ROI). The combined features and ROIs were then inputted into a head network, which included the classifications and regression and mask branches and outputted the locations of pigs and results of classification. Eventually, the individual pigs were segmented from images according to the outputs. The double-pyramid network was trained using 2000 groups of images, splitting to a training set and a validation set in a ratio of 4∶1 randomly. Experimental results showed that the double-pyramid network (Feature pyramid networks, FPN) can effectively address the segmentation for group pig images of adhesive pigs, and acquire the complete individual pig areas, the segmentation accuracy rate was up to 8925%. During the training process, the GPU used rate was lower to 7757%, the FPN outperformed the Mask R-CNN and PigNet networks both in the segmentation accuracy rate and running speed. The double-pyramid network represented its generalization and robustness on the segmentation for multi-behaviors and diversified adhesions in pig group images, which provided a new approach to automatically track individual pig in group pigs.