It is difficult to detect white foreign fibers in cotton by traditional machine vision systems and image segmentation methods, because the color of the targets and background is very close. To solve the problem, an image segmentation algorithm for a machine vision system with an irregular imaging function was presented. Using Gabor operator to extract the orientation feature vectors of an image, combined them into a feature map, thus the contrast between the background and targets was improved by the algorithm. Then a threshold was calculated according to the statistical characteristics of the feature maps. Finally, the white foreign fibers were separated from cotton in the binary image, and the image noises were eliminated by a morphological operation. The experimental results indicated that the algorithm is anti-noise and capable of detecting the targets.