Rice panicle detection is the core research basis of automatic rice panicle counting and rice yield estimation. Due to the density and small size of rice panicle, the size of rice panicle varies greatly at different growth stages, which brings great challenges to the effective and accurate detection of rice panicle. In order to quickly and accurately count the number of rice panicle in the video monitoring area, a rice panicle detection method based on improved Faster R-CNN was proposed. In order to deal with the problem of small target of rice panicle,dilated convolution was introducedon the basis of Inception_ResNet-v2 to optimize the solution. For the problem that rice panicle size varied greatly in different growing periods, K-means clustering aiming at the scale of label box was designed, so as to provide prior knowledge for region proposal network and improve the detection accuracy. In addition, in view of the particularity of the detection target, ROIAlign was used instead of ROIPooling to improve the extraction accuracy of ROI. Using the Faster R-CNN as the basic network and combining the above optimization strategy, a method was proposed for rice panicle detection based on the improved Faster R-CNN. During the experimental test, three data sets were made based on the differences in phenotypic characteristics of rice panicle at different developmental stages, and selected 10 as the best cluster number feasible in practice according to the experimental results. A large number of results showed that the rice panicle detection mAP of this algorithm reached 80.3%, which was 2.4 percentage points higher than that of the original Faster R-CNN model without improved strategy. And compared with SSD and YOLO series model, it had a greater improvement.