In order to improve the accuracy rates and lower the impact on fruit recognition in unstructured environment, a combination of PMD camera and color camera was used to capture multi-source images of orchard scenes, SURF algorithm was used for extracting scale invariant features, Euclidean distance was regarded as a measure for judging the similarity of features, the ratio of distance from the closest neighbor to the distance of the second closest was utilized for initially matching feature vectors, BBF algorithm was devoted to speed up the closest neighbor’s query, a kind of iterative method between picking out outlier points and optimization of model was applied to purify results, the performance of image registration was evaluated according to the MSE, NMI and COEF. The different experimental results show that the amount of information locking to object are enriched by the combination of cameras, the hybrid algorithm is real-time, robust and has ideal precision, which meets the need of orchard test.