The shadow is the most common interference factor of remote sensing image in mountainous and hilly area, so shadow removal is helpful to the improvement of accuracy and effectiveness for image interpretation and feature recognition. Shaded vegetation index (SVI) was constructed, and the band regression model was built for the shadow removal. The proposed method was applied in HJ-1 multi-spectral image. The results showed that SVI could increase the differences among water, shaded area and bright area. The threshold method could be used to effectively detect the shadow in the image. The correlation analysis showed that R2 of each band regression models was above 0.80. The comparison of image statistical indicators before and after the shadow removal indicated that, the band mean value increased significantly with the removal of shadow at the near-infrared band influenced by shadow and vegetations. The standard deviations of shadow-removal image were lower than those of the original image, especially at the near-infrared band. The testing results showed SVI had good detection effects for the shadow of HJ-1 image in mountainous and hilly area, and the band regression model method could effectively remove the shadow.