Real-time detection of orchard environment is an important prerequisite to ensure the accurate operation of orchard spray robot. An improved DeepLab V3+ semantic segmentation model was proposed for multi-category segmentation in orchard scene. For deployment on the orchard spray robot, the lightweight MobileNet V2 network was used to replace the original Xception network to reduce the network parameters, and ReLU6 activation function was applied in atrous spatial pyramid pooling (ASPP) module to reduce the loss of accuracy when deployed in mobile devices. In addition, hybrid dilated convolution (HDC) was combined to replace the void convolution in the original network. The dilated rates in ASPP were prime to each other to reduce the grid effect of dilated convolution. The RGB images of orchard scene were collected by using visual sensor, and eight common targets were selected to make the dataset, such as fruit trees, pedestrians and sky. On this dataset, DeepLab V3+ before and after improvement was trained, verified and tested based on Pytorch. The results showed that the mean pixel accuracy and mean intersection over union of the improved Deeplab V3+ model were 62.81% and 56.64%, respectively, which were 5.52 percentage points and 8.75 percentage points higher than before improvement. Compared with the original model, the parameters were reduced by 88.67%. The segmentation time of a single image was 0.08s, which was 0.09s less than the original model. In particular, the accuracy of tree segmentation reached 95.61%, which was 1.31 percentage points higher than before improvement. This method can provide an effective decision for precision spraying and safe operation of the spraying robot, and it was practical.