In order to satisfy the safety and normal operation for intelligent agricultural vehicle, a method of detecting moving obstacles was proposed based on panoramic vision. Compared with the traditional monocular and binocular vision, panoramic vision possessed the advantages of 360° non-blind area detection. Firstly, multi-thread technology was used to acquire multi-vision images. The improved RANSAC-SIFT algorithm was used to extract and match feature points, and then stitch panoramic images. Secondly, improved CLG optical flow algorithm was used to detect moving obstacles based on panoramic images. Compared with the traditional SIFT algorithm , experiments showed that the accuracy of feature points matching was increased by 25.6% and the arithmetic speed was increased by 25.0%. Moving obstacle detection using improved CLG optical flow algorithm could take averagely 1.55 s to detect moving obstacles, and the accuracy was 95.0%.