An optimal feature subset selection method based on improved ant colony algorithm was presented. The initial probability of the feature was related to the ability of classification of the separate feature, which was advantageous to reduce the redundancy and the hunting zone of the optimized algorithm at the same time. Section variation of the feature set avoided local convergence. Experimental results indicated that the proposed algorithm further reduced the search time, got a smaller subset of the optimal feature set of cotton fibers and better classification performance. The classification accuracy rate increased from 84% to 93%.