The use of computer vision technology for automatic grape berry detection during the berry thinning stage is of great significance for the intelligent monitoring of grape growth status and phenotypic analysis during berry thinning. However, compared with those at the mature stage, Muscat Hamburg grape berries at the berry thinning stage in natural environments are characterized by small size, dense distribution, severe occlusion and adhesion, and low foreground contrast, making accurate detection challenging. To address these problems, an improved YOLO v8n-based grape berry detection model, termed ASP – F – YOLO, was proposed to improve the detection accuracy of grape berries during the berry thinning stage in complex environments. Firstly, the network architecture was optimized for dense small-object detection by introducing a high-resolution feature layer into the feature fusion process, thereby enhancing the model’ s ability to perceive small-scale grape berries. Secondly, an adaptive spatial prompt module was proposed, in which frequency dynamic convolution was employed to adaptively generate spatial prompts. This module enhanced the representation capability of shallow fine-grained features for complex textures and high-frequency details, thereby strengthening responses to target regions while suppressing background interference. Finally, an adaptive hypergraph structure module was introduced to achieve cross-scale feature enhancement and model the high-order relationships between grape berries and clusters, thereby improving the model’ s discrimination ability for densely occluded grape berries. Experimental results showed that ASP – F – YOLO achieved an mAP0.5 of 95.4% , which was 4.3, 4.6, 5.0, 2.2, 12.8, 14.7, and 29.3 percentage points higher than those of YOLO v8n, YOLO 12n, YOLO 13n, RT DETR, CenterNet, RTMDet, and RetinaNet, respectively. The mAP0.5~0.95 reached 74.1% , representing improvements of 4.4, 5.2, 5.4, 5.8, 8.1, 11.0, and 30.1 percentage points over the above models, respectively. The recall reached 88.0% , representing corresponding improvements of 4.0, 5.1, 5.6, 0.7, 19.5, 22.1, and 37.5 percentage points, respectively. Deployment experiments showed that the model achieved a detection speed of 21.05f/s on the NVIDIA Jetson AGX Orin platform, with a model memory footprint of 7.2MB, demonstrating favorable edge deployment capability. The proposed method can provide effective technical support for dense grape berry detection during the berry thinning stage in complex natural environments.