The design of the classifier is an important part for the image recognition system of the stored-grain pests. The simulated annealing algorithm (SAA) was proposed to optimize parameters C and g in the classifier based on support vector machine (SVM), and it was compared with the grid-search optimization. The results indicated that the optimizing efficiency was improved about 3.91 times, and the recognition ratio of the SVM classifier was raised by 5.56%. The nine species of the stored-grain pests in grain-depot were automatically recognized by the classifier based on simulated annealing algorithm and support vector machine, the correct recognition ratio was over 95.56%. The experimental results prove that the method is practical and feasible.  