The image data were acquired by using PCI data acquisition card and the color linear CCD camera. And the sampled data have been stored in text files as a data source for the followed study of color-based sorting algorithm. To reduce the disturbance, we attempted to get smooth signals by wavelet filtering method. According to the relevant theory of digital image processing feature extraction and actual image data characteristics, the K-means clustering analysis was applied to sort the image data into three categories: the background category, normal rice category and the bad rice category. A new method of rice color-sorting algorithm was realized based on color linear CCD, and this method is independent of any human experience.