In order to improve the data visualization and information level, a kind of greenhouse remote monitoring system was designed and developed, which included inspection robot, mobile communication network, cloud server and remote monitoring center. Three kinds of data transmitted between the greenhouse and the remote monitoring center, including text, image and video. Machine learning and deep learning algorithms were used for human-computer interaction and tomato recognition tasks. On the one hand, administrator face recognition was achieved based on Haar cascade and LBPH algorithms, and the recognition success rate was 90%. Then YOLO v3 and ResNet-50 algorithms were used to recognize the hand and the key points of hand respectively, and the recognition confidence of singlehand and two-hand was 0.98 and 0.96, respectively. Based on the extracted coordinates of the forefinger and the center points of the left and right hand candidate frame, finger interaction and image size scaling were realized. On the other hand, the model framework of Swin Small+Cascade Mask RCNN was used for tomato recognition. Aiming at the problem of limited agricultural data set, the effect of tomato detection before and after applying transfer learning method was compared and analyzed. By using transfer learning method, the experimental results showed that the convergence rate of the model was increased and the loss value was decreased. In terms of semantic segmentation, AP value was used to evaluate the model performance, when IoU was set to be 0, 0.5 and 0.75, test results showed that the mask average precisions were improved by 7.8 percentage points, 6.4 percentage points and 7.2 percentage points, respectively after using transfer learning method.