A method based on improved YOLO v4 algorithm was proposed to realize the rapid detection of clods and stones from impurified potatoes after harvest. The YOLO v4 detection model was built on CSPDarknet53 framework. The channel pruning algorithm was used to prune the model to simplify the structure and the computational cost, while under the premise of detection accuracy. Mosaic data enhancement method was used to expand the image data set (8621 images), and the model was fine-tuned to achieve the detection of clods and stones from impurified potatoes. The test results showed that when the pruning rate was 0.8, the number of parameters of the model was reduced by 94.37%, the model size was decreased by 187.35 MB, the inference time was reduced by 24.1%, and the floating-point operations per second was compressed by 54.03%. It was shown that the performance of model can be improved by pruning. In order to verify the performance of the model, the model was compared with Faster R-CNN, Tiny-YOLO v2, YOLO v3, SSD and YOLO v4. The results showed that the mean average precision (mAP) of the model was 96.42%, the detection speed was 78.49 f/s, and the model size was 20.75 MB. The mean average precision was 11.2, 11.5, 5.65 and 10.78 percentage points higher than that of the other four algorithms and 2.1 percentage point lower than that of the YOLO v4 algorithm. The detection speed met the practical needs, and it can be applied to post-harvest potato impurity removal.