Germination and surface damage detection are crucial steps in the commercialization of fresh table potatoes. To address the low accuracy rate of high-pixel image object recognition in the high-throughput grading and sorting process of fresh table potatoes, a method for detecting potato sprouting and surface damage based on improved Faster R-CNN was proposed. Using Faster R-CNN as the baseline network, the feature extraction network in Faster R-CNN was replaced with the residual network (ResNet50), and a feature pyramid network (FPN) integrated with ResNet50 was designed to increase the depth of the neural network. A comparative model assessment and ablation studies were performed to empirically validate the efficacy of the proposed model and its modifications. The findings delineated that the enhanced algorithm demonstrated an average precision rate of 98.89% in identifying potatoes, 97.52% in discerning sprouting events, and 92.94% in recognizing surface defects. When benchmarked against the Faster R-CNN model, the adapted model incurred no additional computational time or memory overhead while manifesting a marginal decline of 0.04 percentage points in potato identification accuracy. Notably, it significantly elevated the average precision in detecting sprouting and surface imperfections by 7.79 percentage points and 34.54 percentage points, respectively. This augmented model was robust in high-resolution imaging environments facilitated by industrial-grade cameras and served as a cornerstone for the methodological advancement of automated grading and sorting processes in the commercial potato industry.