Aiming to solve the needs of foreign matter detection in walnut production line, a set of walnut impurity detection equipment was designed and built based on the existing universal walnut processing production line, including portable frame, image acquisition system, and constant light source system. The overall size was 470mm×600mm×615mm. Walnuts from Zhejiang Province and impurities, including leaves, stones, paper, screws and fabric were photographed as detection objects by industrial camera above the production line in real time for intuitive image information data. An image segmentation technology combined with deep learning and computer vision, and the fully convolutional network (FCN) algorithm were applied to detect impurities that might occur in walnut production and processing. According to the test, the accuracy for detection and classification of walnut and foreign body was effective, which was 92.75% of training set and 90.35% of testing set. The speed of production line was 1m/s. The recognition speed of detecting was 4.28f/s, which can meet the requirements of real-time detecting of impurities. The biggest error was in the “walnut-background”, where original walnut was predicted to be the background. The main reason was that some features in walnuts (such as cracks and lines) were similar to the background. Focusing on the analysis of foreign body error, it showed that impurities were mis-predicted as the “background” much more than the impurities were mis-predicted as the “walnut”. Two main reasons led to this difference. On the one hand, when labelling manually, the pollutants on the conveyor belt were not judged as foreign bodies. On the other hand, because the size of impurities was generally small and the cardinality of pixel points was insufficient, the influence of false prediction was greater, thus amplifying the error. The reliability of the model was good. Even if the artificial labeling error occurred, walnut was mislabeled as impurities, but the trained model could still distinguish walnut and adjacent impurities well. The method proposed was worthy of further study for the online detection of impurities in automatic production of walnut, and it was of great significance to broaden the market of nut food and improve its economic benefits.