Aggressive attacking behaviors of artificial rearing male sika deer on heat period are increased dramatically, which causes damages to deer’s antlers and even deer themselves. Automatic monitoring of their aggressive attacking behaviors can provide an important basis for the research to reduce them. A dual-stream neural network (optical flow attention attacking recognition network, OAAR) was proposed, which was based on the attention mechanism and long-short memory sequences. It was used to achieve automatic recognition and detection of sika deer behaviors, including attacking, feeding, lying down, and standing. The OAAR network consisted of a per-network, a base network, and a time-sequential network. The pre-network consisted of the LK optical flow algorithm（lucas kanade optical flow algorithm), which was used to extract the information from the RGB data. In the base network, a self-attentive module was added to the ResNet-152 to build a new design ARNet152 (Attention ResNet-152), which was used to combine the RGB and optical flow information, extract the features, and input them into the time-sequential network. The time-sequential network was based on an attention long short term network (ALST), which was composed of an attention long-short memory sequence that can classify the behavior and give scores. The experimental dataset was composed of 10942 video segments, with a total of 310574 frames, which were divided into four major categories of behaviors, including aggression, foraging, standing, and lying. From the aggressive behaviors, three sub-categories were further divided, including hitting, kicking, and chasing. The training, validation, and test sets were divided at a ratio of 3∶1∶1. The results of the study showed that the OAAR model reached an accuracy of 97.45%, a recall rate of 97.46%, and an F1 value of 97.45% on the test set, and good classification results in ROC curves and improved discrimination in feature embedding maps. All the results of OAAR were better than the results of LSTM, I3D, and ITSN networks. Meanwhile, the online deer behavior identification and recording system based on the OAAR network was developed to improve the management level and production efficiency of the sika deer farming industry.