In order to improve the accuracy of the navigation path and satisfy the safety of driving for autonomous navigation of agricultural vehicles, a method of detecting and tracking multiple moving objects was proposed based on panoramic vision. Panoramic vision possessed the advantages of non blind area detection and the improved algorithm solved the problem of the overlap in multiple moving objects tracking. Firstly, multi-vision images were acquired to stitch panoramic images, the improved kernel function algorithm based on segmented image was used to detect and track the moving object automatically and rapidly. Secondly, the particle filter algorithm based on path prediction was used to track multiple moving objects and solved the overlap problem. Compared with the traditional kernel function algorithm, experiments showed that the memory consumption was reduced by 66.8% and the algorithm speed was increased by 35.63%. Multiple moving objects detection using the particle filter algorithm based on path prediction could take averagely 0.78s to detect moving obstacles, and the success rate of moving objects tracking was increased by 39.5 percentage points under the condition of overlap in multiple moving objects.