The detection and segmentation of cucumber fruits are crucial for phenotypic analysis and the management of cucumber growth. However, in complex greenhouse environments, fruits are often occluded by stems and leaves, and their color may be similar to the background, making it difficult for traditional methods to accurately identify fruit boundaries and achieve efficient segmentation. To address this issue, an improved YOLO v8-based method for cucumber fruit segmentation was proposed. This method incorporated deformable convolution network v4 (DCNv4) to enhance the model’s spatial adaptability and utilized the RepNCSPELAN4 module in combination with an additional C2F module to refine feature extraction and fusion, thereby improving the model’s segmentation performance for cucumber fruit images in complex greenhouse environments. Experimental results showed outstanding performance across multiple categories in two experimental settings: a glass greenhouse and a plastic greenhouse. Specifically, in the glass greenhouse scenario, the model achieved a precision of 96.3%, recall of 93.1%, mean average precision (mAP50) of 96.2%, and mAP50-95 of 85.3%. In the plastic greenhouse scenario, the precision was 86.8%, recall was 81.9%, mAP50 was 90.0%, and mAP50-95 was 77.0%. The proposed method demonstrated stronger robustness and generalization in handling boundary issues, multiple occlusions, and multi-scale segmentation, enabling the model to adapt to diverse and complex cultivation environments and accurately segment cucumber fruits. Accurate fruit image segmentation facilitated the acquisition of phenotypic parameters and provides reliable technical support for further phenotypic analysis of cucumber fruits, thereby promoting the application of agricultural phenotyping robots and the intelligent development of agricultural production.