The use of machine vision to quickly and accurately estimate fruit yield is of great significance for the development of smart agriculture. In view of the characteristics of dwarf dense planting and wide distribution of kiwifruit cultivated in greenhouses, orchard crawler trolleys were used to shoot and obtain videos of kiwifruit orchards, and a dataset of kiwifruit detection and tracking was established combined with artificial labeling. Considering the small proportion and dense distribution of kiwifruit in the self-made dataset, the YOLO v7 model and Soft-NMS were proposed to detect kiwifruit in each frame. Based on the prediction results of the Kalman filter, the VGG16 network was introduced to extract the features of kiwifruit, and the Hungarian algorithm was used to complete the target matching of the before and after frames. Finally, the ID counting method based on YOLO v7+DeepSort tracking algorithm was used to realize kiwifruit yield estimation. The experimental results showed that the improved YOLO v7 model performed well on the kiwifruit detection dataset, with an F1 score of 90.09%. The average accuracy of the adopted tracking algorithm on the kiwifruit tracking dataset was 89.87%, the precision of each target can be correctly matched was 82.34% and a large video tracking speed of 20.19f/s. Under the condition of low environmental impact, the ID counting accuracy was 97.49%. This method can provide technical support for yield estimation and harvest planning in the intelligent management of kiwifruit orchards.