With the rapid development of remote sensing technology, remote sensing image resolution has been greatly improved and the ground targets can be obtained from high resolution remote sensing image. But the traditional visual interpretation has low work efficiency and needs for professional knowledge. Thus using high resolution remote sensing image to construct automatic interpreting model is the key to quickly and efficiently obtain the building of hollow village. Meanwhile, it is important for the hollow village renovation and research. Based on this, a novel automatic building interpretation model of hollow village high resolution images based on bagofwords (BoW) was proposed. Firstly, several existing feature extraction methods were compared, and then based on the BoW and support vector machines (SVM) the automatic interpretation model for the building was constructed. In order to verify the validity of this method, the high resolution remote sensing image of typical hollow village was selected to construct the building sample library. Finally, the model for building interpretation was experimentally studied based on the sample library. The results showed that the classification accuracy (ACC) of this method can reach 0.86. Therefore, the proposed method can be used for the building automatic interpretation, and it had high practical value to hollow village research and renovation.