Cotton boll is an important yield and quality organ of cotton. The research on phenotypic traits such as boll number per plant, boll length and width is of great importance in cotton genetics and breeding research. In order to obtain the accurate number of bolls, a boll tracking and counting method was proposed based on the improved Faster R-CNN and Deep Sort to realize cotton boll measurement based on the rotating video. First of all, a simple video captured device was designed for the cotton plant. And then the feature pyramid network (FPN), Guided Anchoring and Soft NMS methods were adopted to improve the original Faster R-CNN detection network, in which the FPN was used to promote the ability for small targets recognition, Guided Anchoring was applied to generate the Anchors with appropriate size, and the Soft NMS was adopted to mitigate the mistaken deletion of overlapping targets. As a result, the improved Faster R-CNN outperformed the other models, including RetinaNet,SSD, Faster R-CNN, YOLO v5 and YOLOF. The mAP75 and F1 of improved Faster R-CNN was 0.97 and 0.96 respectively, which was 0.02 and 0.01 higher than that of the original Faster R-CNN model. After that, Deep Sort was used to realize the match of the same target in different frames through Kalman filter and deep association metric, and the ID of the same target was matched. In order to solve the ID switch problem, the mask collision mechanism was developed. When the matched cotton boll passed through the mask region from right to left, the ID of the cotton boll would be recorded and the number of the cotton boll would be added, which was proved to significantly reduce the mistaken counting caused by ID switch. Finally, the specialized software was designed based on the improved Faster R-CNN, Deep Sort and mask collision mechanism. The results showed that the tracking result RMOTA was 0.91, which was 0.02 higher than that of Tracktor algorithm, and 0.15 better than that of Sort algorithm, respectively. The measurement results of coefficient of determination, mean square error, mean absolute error and mean absolute percentage error of the bolls number were 0.96, 1.19, 0.81 and 5.92% respectively, which had high consistence with the manual measurement, and it could realize the high precision counting of cotton bolls based on the specialized software. In conclusion, the research demonstrated an effective tool for cotton bolls measurement, which was beneficial to the cotton breeding research.