A machine vision based real-time localization system for transplanting machine was proposed. The system can realize real-time detection of the plug tray’s position and get the right coordinate for each planting accurately. By off-line calibration using point cloud, equation of the conveyor belt’s plane was determined. With the equation, the plug tray was separated from the background in the point cloud. The 2D mask was got by using the point cloud of the plug tray. Then the contour of the plug tray was detected in the 2D mask, the plug tray’s position was determined and each hole’s coordinate was calculated in its corresponding simple connected area before planting the first graft seedling. After that, considering the movement during each time’s planting caused by vibration, re-localization needed to be done. To do the re-localization, SURF feature points detection was done in the color image before and after last planting. Then the SURF feature points were matched, and the homography matrix was computed based on random sample consensus algorithm, which was used to calculate the coordinate of the hole. The experiment results showed that the system could achieve high precision for localization of the hole in the transplanting plug tray and it was little influenced by the lighting conditions and background color.