Accurately obtaining the building information in the hollow village areas is important for hollow village renovation and research. With the rapid development of remote sensing technology, remote sensing image resolution has been greatly improved and the ground targets can be obtained from high-resolution remote sensing image. But the traditional methods based on low-level hand-engineered features or mid-level features have great limitation in complex environment, especially in hollow village areas. So it needs to use high-level features to express. Convolution neural network (CNN) has become one of the important methods of ground object recognition and detection. Based on CNN, a novel automatic building detection method was proposed. Firstly, a multi-scale saliency computation was employed to extract building areas and a sliding windows approach was applied to generate candidate regions. And then a CNN was applied to classify the regions. In order to verify the validity of this method, the high resolution remote sensing image of typical hollow village was selected to construct the building sample library. Finally, the model for building interpretation was experimentally studied based on the sample library. The results showed that multi-scale saliency can effectively get the main target, weaken the impact of other unrelated targets, and reduce data redundancy. The CNN can automatically learn the high level feature, and the classification accuracy (ACC) of this method can reach 97.6%. So the proposed method can be used to detect building and it had high practical value to hollow village research and renovation.