The method of shape feature extraction based on depth image for the classification of tomatoes shape was proposed. Firstly, the shape of tomatoes was separated from the background through the segmentation of image in color space. Secondly, the point cloud of tomatoes was obtained by unitizing a 3-D machine vision measuring device. In order to implement the shape feature extraction of tomatoes in the same scale, the depth values of tomatoes were normalized. The depth map of tomatoes was formed according to the result of segment and the depth information of tomato. Further the depth map was sampled in polar coordinates and the sampling data was re-plotted in Cartesian coordinates. Finally, the depth image was re-plotted in the form of the Fourier transform in the Cartesian coordinates. The generic Fourier descriptor（GFD）was calculated based on depth map. The descriptor was characterized by the invariance of transformation of translation, rotation and scaling. The GFD based on depth image and the general GFD were successively used in the experiment of tomato grading. The result showed that the mean accuracy of the former classification was up to 92% and higher than the latter.