Urban villages (UVs) belong to a special product of China’s rapid urbanization process, which have similar properties to the informal settlements abroad. Specifically, UVs in China usually have a high population density due to the reconstruction of buildings, making it a big challenge in China’s urban and rural sustainable development. Especially under the background of “promoting the newtype urbanization” issued by the government, timely and accurate identification of UVs is of great significance to both urbanrural planning and urban fine management. Researchers usually obtain the spatial information of UVs by field research in traditional studies, which is both laboursome and tedious. Remote sensing, on the other hand, has the merits of synoptic view, dynamic and fast screening of the earth surface, which has been recently applied in the recognition of UVs. Meanwhile, deep learning has shed new light on UVs’identification due to its capability in learning high-level abstract image features, however, it has been rarely documented in the mapping of UVs. Therefore, the objective was to propose a deep learning model for UVs’recognition from very high resolution (VHR) remote sensing images. In specific, the proposed model was a multi-scale dilated convolutional neural network (MD-CNN), which included a series of multi-scale dilated convolutions and a non-local feature extraction module. The former can aggregate multi-level spatial features to adapt to the variability of UVs’shapes and scales, while the latter extracted global semantic features to improve the inter-class divisibility. The experimental results in Beijing City showed that the proposed model achieved good performance with an overall accuracy of 94.27% and a Kappa coefficient of 0.8839, which was better than that of several previous deep learning models such as VGG, ResNet and DenseNet. The research result demonstrated that by using the deep learning model, it was feasible and effective to accurately identify UVs from VHR remote sensing images, which could provide useful geo-spatial distribution of UVs for urban-rural planning.