In the field of hyperspectral detection on heavy metal pollution levels in agricultural soils, the accuracy and stability of hyperspectral inversion model for soil cadmium were seriously affected by the high dimensional and high redundancy characteristics in visible/NIR spectra. In order to solve the above problems, Spearman’s rank correlation analysis-based genetic algorithm by using random forest (SGA-RF) was proposed to select the characteristic wavelength from hyperspectral data. On the first-layer of feature selection stage, Spearman correlation analysis-based feature selection method was applied to remove redundancy between all spectra features and retain the characteristic wavelength which was the most relevant to the cadmium content. On the second-layer of feature selection stage, a new fitness function based on random forest was proposed, which perfectly combined the strong global search ability of genetic algorithm and the high inversion ability of random forest. With the proposed fitness function to evaluate the viability of individuals, the distinguishing ability between similar individuals was improved and a subset of optimal spectra feature set with minimum redundancy and maximum differentiation were obtained. In order to verify the validity of the proposed algorithm, totally 124 representative soil samples collected from the Dagu River Basin were chosen as samples. The optimal feature subset which contained 37 sensitive wavelengths was chosen and used to build soil available cadmium content inversion model, and its performance was compared with that of current feature selection methods. Results indicated that the minimum numbers of wavelength features was selected and meanwhile the prediction performance had lower predictive root mean square error of 0.0601, higher correlation coefficient of 0.9502 and residual predictive deviation of 2.03. As an important step for the quantitative inversion of cadmium concentration by using visible/NIR spectra, the research could provide some theoretical basis for monitoring soil heavy metal pollution.