Accurate data sensing and processing are basic of quantitative decision-making in smart agriculture management. Image sensing provide multi-dimensional information for agriculture detection, such as color, visible-near infrared spectroscopy, 3D and thermal radiation. The traditional way to analyze these images focuses on the characteristics of color, morphology, texture, spectral reflection and so on. The limitations of sample mounts and extracted features always lead to the problems such as insufficient noise reduction and low accuracy of the recognition and detection models, especially for complex background changes and unknown samples. Deep learning (DL), a subset of machine learning approaches, emerged and combined neural networks to extract and represent the high-level features of image. It provided a versatile tool to assimilate and explore distribution and features from heterogeneous data. It could help to build reliable predictions of complex and uncertain phenomena in agriculture. In order to explain the application potential and further direction, the applied sensors, specific models and dataset sources were examined from five areas, including plant recognition and detection, disease and pest identification, remote sensing classification and monitoring, products detection and grading, and animal detection. Finally, several avenues of researches were outlined.