Pig drinking behavior is closely related to pig’s physical condition and piggery environment. Recording such data continuously is beneficial to the pig industry. However, it is difficult to get the detailed drinking data of each pig manually. An automated method is expected. RFID is used to detect pig drinking behavior recently. But this approach invades pigs and the piggery needs to be equipped with auxiliary facilities. There is no such concern by using video monitoring. Therefore, using machine vision to recognize pig drinking behavior was proposed. Firstly, to distinguish pigs from the background, threshold segmentation was used to get a binary image, in which pixels belonged to pigs were assigned to 1 and others were assigned to 0. From the binary image, each pig’s centroid and angle were computed and used to decide whether a pig was static or not. Drinking behavior is likely to happen when a pig stays in the drinking zone. Secondly, occupation index was computed to determine if a static pig was closed to the drinking nipple. Drinking behavior could be preliminarily judged through this way. Thirdly, a pig head detector was implemented by using deep learning algorithm to accurately confirm the occurrence of pig drinking behavior. At last, to confirm which pig was performing the drinking behavior, a pig identification detector was implemented. Through the multistep judgment, pig drinking behavior can be recognized precisely. Experiment showed that the precision rate of the proposed algorithm in the video data set was 92.11%, which was suitable to aid managerial decision making in pig production.