The research achieved to control the mechanical arm position and pose by using real-time dynamic gesture recognition based on Kinect device. The information of the position controlling was obtained by calculating the position changes of the four hand joint points. The noise of the joints was liable to lead mechanical arm misoperation and the vibration of motion during the control of the mechanical arm. Aiming to avoid the negative impact of the noise in real-time controlling, Kalman filter was adopted to track position and reduce noise. According to the hand point cloud information, the information of the posture controlling was obtained by means of using least squares fitting to get the plane of hand mind. The end of the position and pose of the mechanical arm was controlled by integrating the position and posture information, space coordinate mapping and the resolving of kinematics in real-time. The result of the experiment indicated that the gesture control was easy to operate and mechanical arm responded at high speed. The effect of filter was so remarkable that the motion of the mechanical arm was controlled accurately and smoothly, and no mechanical arm misoperation and others controlling anomaly. Gesture control system could meet the requirement of actually controlling. System could be applied to a variety of human-computer interaction.