Aiming at the problems that the quadrotor plant protection unmanned aerial vehicle (UAV) has poor adaptability in sloping land and imprecise height measurement accuracy during operation, a terrain following flight method based on multi-rate Kalman filtering fusion of stereo vision, barometer and inertial measurement unit (IMU) information was proposed to estimate the height of UAV. Firstly, the ZED2 camera was used to obtain the binocular image of the ground below the UAV, and the point cloud information corresponding to the binocular image was calculated through the parallax principle. After analyzing the relationship between the height of the point cloud, the attitude of the UAV and the best visual detection area, an adaptive algorithm for the visual detection area was proposed to select the ground detection area. The accurate visual ground height was obtained by analyzing the point cloud data of the detection area. Secondly, a multi-rate Kalman filter model was established which fused the visual height, barometer and IMU information to estimate the height above the ground of the UAV. Finally, a two-level control system was composed of an NVIDIA microcomputer and a flight controller. The UAV’s height estimation and terrain following flight performance were verified by flight experiments. The height estimation results of remote-control flight showed that the method proposed can achieve height estimation with an absolute average error of 46.8mm and a standard deviation of 38.2mm under large height changes. The autonomous terrain following flight experiment showed that when the flight height was set to be 2m and the speed was 1m/s, 2m/s and 3m/s, respectively, no matter on the flat ground or the gentle slope of 15°, the absolute average error of height estimation was less than 20mm, and the standard deviation of height estimation was less than 30mm;the absolute average error of height control was less than 30mm, and the standard deviation of height control was less than 30mm. The research result verified the effectiveness of the plant protection UAV in the terrain changing scene, and laid a foundation for the automatic operation of the plant protection UAV in complex terrain.