Ruminant behavior is closely related to dairy cow production, reproductive performance, disease and other factors. To overcome the shortage of non-contact dairy cow ruminant behavior analysis caused by cow movement or background interference, the FlowNet 2.0 optical flow algorithm was improved. Firstly, the vertical optical flow component was calculated instead of the optical flow velocity to construct an optical flow diagram to eliminate horizontal movement interference on optical flow analysis. Secondly, the optical flow threshold was set to avoid the interference of head movement flow in vertical optical flow. The area threshold of the ruminant area was calculated and the optical flow data was extracted in the region to avoid the influence of head movement flow of the target object. Finally, filter fitting was used to calculate the rumination curve, determine curve period, increase the difference between peak and valley, and improve the accuracy of counting of rumination frequency in dairy cows. The validity, robustness and accuracy of this method were validated by using 20 dairy cows and 30 ruminant videos in different scenarios. The results showed that the accuracy of improved FlowNet 2.0 optical flow algorithm was 99.39% and 5.75 percentage points higher than that of FlowNet 2.0 optical flow algorithm.