Aiming to improve the detection accuracy and speed of estrus behavior of dairy cows in a complex scene, a method of recognizing estrus behavior of dairy cows based on improved YOLO v3 model was proposed. To solve the problem of the cows’ size was inconsistent with the size of the object in the COCO dataset, which caused the original anchors were not applicable, new anchors were obtained by clustering new data sets and optimized by using linear expansion. As cows with a big size, the small difference between individuals and associations between behaviors, which was difficult to distinguish, a DenseBlock structure was introduced to the feature extraction network of YOLO v3 model to improve its detection performance on the large objects. Considered that the original bounding box loss function of YOLO v3 model was not invariant to the object scale, the FIoU and the center distance Dc of two boxes were used as the measuring method, and a new bounding box loss function was proposed to make it scale-invariant. Totally 50 images were extracted each from 96 video mounting behavior clips of dairy cows, according to the uncertainty position of cows’ mounting behavior in the active area and the character of the light changing of the active area, horizontally flipped, rotated ±15° and random brightness enhancement (decrease) were applied on them for data augmentation. The augmented data was divided into three parts as training sets, validation sets, and test sets, training sets and validation sets were used to train the improved model and the best training model was chosen as dairy cow estrus behavior recognition model with the indicators F1, mAP, accuracy rate P, and recall rate R. The experiment on test sets showed that the accuracy rate of the model was 99.15%, the recall rate was 97.62%, and the processing speed reached 31f/s, which could accurately and real-time identify cows’ estrus behavior in a complex breeding environment under all weather. The research could also provide a reference for other large livestock behavior recognition.