There are many problems in the current greenhouse and plant factory. It’s an effective solution to work by robots. However, because of the limit of the intelligent algorithm at present, the robot’s works are imprecise.And they can just complete single job in once cruise which cause the inefficient using. Aiming at the problem, an algorithm which based on the motion sensing of Kinect and used in three arms robot was proposed, by using the Kinect to collect the depth image, including the operating personnel and combining the Random forests of statistical learning theory with the mean shift algorithm based on the Gauss kernel function which acquired 20 skeletal joints that can standard the human motion. On this basis, a mapping relation was put forward in innovation of three arms based on the mode switching to achieve that the two arms of the man can freely control three arms of the robot and perform several works in greenhouse. In addition, a way of gesture features segmentation was proposed which based on skeletal tracking technology in Kinect and YCbCr color space, realizing the aim that using the action of the palm to control the robot’s end effectors. Finally, a prototype of three arms robot was built to test the decision algorithm of motion sensing and its accuracy. A discrete value was taken for the angle in shoulder joint to recognize the error data in experiment, so eliminated errors in the shoulder joints. The results showed that the maximum mapping error of joint angle was 1.90° between human and robot. The maximum error of the host computer sending angle and the real angle by robot was 0.80°, which was within the margin of error. Meanwhile, the average time of completing an order of picking and spraying was 13.34s, the picking time was 6.36s and the spraying time was 6.98s. And the performance of the robot can be boosted by training the manipulator. It was indicated that this algorithm had a great practicability and can work accurately.