Preventing and managing crop disease and pest has significant impacts on agricultural production. The prerequisite for disease and pest control is accurate detection. Traditional crop disease and pest detection methods rely on human labors and instructions. However, these methods can no longer meet the requirements of scientific research and production, such as detection efficiency, accuracy, and application scenarios. As a main stream of machine learning, deep learning can extract features of objects from large-scale datasets automatically and efficiently, thereby releasing traditional methods from manual feature extraction. Applying deep learning, combined with image processing techniques, to detect crop disease and pest becomes an inevitable trend of precision agriculture in the future. The key techniques in crop disease and pest detection depend on agricultural data. After reviewing the state of the art of key techniques in this domain, including data acquisition, data pre-processing, data augmentation, deep learning network optimization, data visualization, and explainability of results, the challenges of applying these key techniques were detected and summarized. Lastly, potential solutions were explored to highlight the future research lines in this domain, including defining multi-view agricultural datasets, combining transfer learning, adopting new data augmentation methods, and considering visualization and explanation issues.