Because the monocular camera lacks of depth information, it was difficult to use this kind of camera to estimate robot pose in the line of fruit tree and measure distance accurately. Under this research background, a method was proposed to calculate yaw angle, lateral offset, and fruit tree position based on instance segmentation neural network. Firstly, based on the Mask R-CNN model, the road and tree trunks were detected, and their masks were extracted. Secondly, to calculate the vanishing point, the boundary equations were identified based on convex hull and Hough transform, and the vanishing point coordinates were calculated by solving the equation. Finally, according to the established poseroad imaging geometric model, the yaw angle, lateral offset and relative position of the fruit tree were calculated. The experimental results showed that the boundary regression accuracy of the improved Mask R-CNN model was 0.564, the segmentation accuracy was 0.559, and the average inference time was 110 ms. Based on the method, the yaw angle estimation error was 2.91%, and the lateral offset error was 4.82%. For fruit tree positioning, the lateral error was 3.80% and the longitudinal error was 2.65%. At various data collection sites, the method could stably extract road and fruit tree masks, calculate vanishing point coordinates and boundary equations, in addition, the yaw angle, lateral displacement and relative position of fruit trees could be estimated more accurately. Under orchard conditions, it could further improve the visual navigation effect and the intelligent level of agricultural equipment.