Dormant pruning is a labor-intensive and time-consuming operation. It is an important part for the refined management of jujube orchard, which can control the tree structures by removing the over-long branches, thus decreasing the limbs density. Automated pruning using a robotic platform could be a better solution. To realize automatic selective pruning for dormant jujube tree, the segmentation of branch and trunk of tree was difficult in complex jujube orchard background. A method based on semantic segmentation network for branch recognition of jujube trees was studied in field. The visual system built by RGB-D camera was used to acquire the point cloud information of jujube trees under different weather conditions, and the background was removed by using the distance threshold for construction of foreground jujube tree datasets. Two kinds of semantic segmentation models, DeepLabV3+ and PSPNet, were utilized to segment branch and trunk of jujube tree and obtain the pruning branch mask, meanwhile the results of segmentation were compared. The mask of pruning branch was binarized, and the noise was removed based on the area of the connected domain of the binary image. The connected domain was labeled after denoising, and the branch skeleton was extracted. Finally, the number of pruning branch was determined, and the linear regression model for the real value and predicted value of the pruning number was established. The results showed that the DeepLabV3+ model based on ResNet-50 (feature extraction network) achieved the best segmentation results, and its average pixel classification accuracy and average intersection-over-union were 89% and 81.85%, respectively. The PA and IoU for trunk and pruning branch were 90.36%, 80.98% and 80.34%, 66.69%, respectively. The mean pixel accuracy for branch and trunk of jujube tree in sunny was 91.97%, which was slightly higher than that in cloudy (91.81%) and night (90.98%) under three typical weather conditions. Meanwhile, the R2 was 0.8699 between predicted values and real value in sunny, which was higher than that of cloudy day (0.8373) and night (0.8120), and the minimum RMSE (1.1618) was obtained.