To solve the problem of segmentation for overlapping fruits, an image segmentation algorithm based on disparity map was developed. Firstly, binocular stereo images were obtained by binocular stereo vision system. Then, these images were preprocessed and rectified. Thirdly, stereo matching for rectified images pair to get disparity values for every pixel was made. At last, the disparity map was generated. Distance information in 3-D space was added to algorithm besides color, shape and texture information in 2-D space, so the fruits with different distances in actual space were segmented better in disparity map than in normal image. Experimental results showed that the accuracy of segmentation for disparity map by area-based method was 0.90, and the edge detection error was 5.74%. The proposed method was valid for segmentation of overlapping fruits. 