Milk is one of the main sources for humans to obtain protein, and dairy industry is also an important pillar industry for agricultural personnel in China to increase their income. Detecting the estrus of dairy cows in time, artfical insemination them at the right time, and reducing cows emptiness are the key means to increase the milk production of dairy farms. As the methods of identifying dairy cow estrus based on physical signs such as activity or body temperature often cause stress reactions of cows and accuracy is also not high enough, a noncontacted automatic method for recognizing estrus behaviors of cows was proposed. In this method, an improved Gaussian mixture model to achieve target detection for moving cows was used. Then, interference images were removed based on the information of color and texture. Next, a cow behavior classification network model based on AlexNet was trained to identify cows’ mounting behavior. Finally, based on the classification model result, automatic recognition of estrus behavior of cows was realized. Experiments on the test video data sets showed that the accuracy rate of the method was 100%, and the recall rate was 88.24%. The method can be used for daily estrus monitoring of dairy farms, and it can also provide support for decisionmaking of their production management. The research can also serve as a reference for the automatic recognition of other large animals’ behaviors.