A crop line feature detection method based on semantic segmentation network was proposed to realize stable and reliable visual servo control of plant protection robot. Based on the semantic segmentation network which was termed with ESNet, pixel-wise labeling in farmland images was performed for ribbon regions detection, and least mean squares algorithm was utilized to find out all the crop line feature parameters in real time. Among the derived candidate lines features, a key route line was chosen as the valid navigation path which was responsible for subsequent robot motion control. Kalman filter was subsequently employed to smooth geometrical parameters of the previously specified key route, which effectively suppressed the fluctuation of navigation parameters caused by jolt behavior of plant protection robot generated from uneven ground and measurement noises incorporated in visual images. Afterwards, the sophisticated Ackermann steering kinematic model which was characterized by robot front-wheel steering and rear-wheel differential was introduced. A pure tracking controller was designed in Cartesian coordinate system to realize the servo motion control of plant protection robot. The field experiment conducted in real farmland scenarios verified the effectiveness of the proposed method.