Daily behaviors are the important indicators of health status for calves. The suitability of using behavioural changes to provide an early indication of calf’s disease was studied. The possibility of achieving a realtime analysis of a number of specific changes in behaviours, such as lying, standing, walking, running, and jumping, is crucial for disease prevention. Considering the limitation for sensing animal behavior by contacting device and in order to improve the welfare of calves, a method based on video analysis was studied and applied to recognize calf basic behaviors. Firstly, a looping algorithm based on maximum connected region was proposed for fast detection of calf target under complex environment. Secondly, a realtime model was built to renew the background and detect the calf’s target quickly and accurately. Thirdly, the position of the centroid, the ratio of the height and width of the target outline, and differences of the centroid moving curve were extracted as the features of behaviors. These features could be the characterizations of the internal properties of behaviors constituted the sequence structure of calf behaviors. Finally, a classifier based on structure similarity of behavior features was designed to recognize basic behaviors of the calf. By testing 162 videos, the results demonstrated that the recognition rate of lying, standing, walking and runjump were 100%, 96.17%, 95.85% and 97.26%, respectively. On the basis of these research outcomes, the proposed method is feasible for computing calf behavioural indices and the realtime detection of behavioural changes, and also lays a foundation for recognizing and understanding senior behaviors of large animal.