A system with functions of surface defect detection and classification for cherry tomatoes based on machine vision was proposed. The system integrated image acquisition, image processing, image recognition and robotic palletizing, which could achieve image filtering, binarization, edge-detection and location identification. An edge-detection algorithm was proposed based on fractional differential and Sobel operator. The effectiveness of the proposed algorithm was tested by comparing the sorting results of seven different algorithms on the same parcel of cherry tomatoes. Experimental results showed that the algorithm could effectively detect defects and classify the cherry tomatoes. The comprehensive classification accuracy was 98.4% with a good application prospect.