Accurate and efficient cow behavior recognition is helpful for timely disease detection and detection of abnormalities. It is the key to perceive cow health. By analyzing the behavior of cows at different periods in the cattle farm, a cow behavior recognition algorithm based on spatiotemporal features was proposed. The algorithm combined temporal shift module (TSM), feature attention unit (FAU) and long short-term memory (LSTM) networks on the basis of time-domain segment network (TSN). Firstly, TSM was used to fuse time information to improve timing modeling ability. The video frame after time sequence modeling was input to TSN. Secondly, FAU was used to integrate high resolution spatial information and low resolution semantic information to enhance the learning ability of spatial features of the algorithm. Finally, the past and current information were fused by LSTM to classify cow behavior. The results showed that the recognition accuracy of this algorithm for eating, walking, lying, and standing was 76.7%, 90.0%, 68.0% and 96.0%, respectively. And the average recognition accuracy was 82.6%. Compared with C3D, I3D and CNN-LSTM networks, the average recognition accuracy of this algorithm was 7.9 percentage points, 9.2 percentage points and 9.6 percentage points higher, respectively. The illumination variation had a certain impact on the recognition accuracy, but the proposed algorithm was relatively less affected by light. The results can provide technical support for cow health perception and disease prevention.