A new generation method based on NURBS and optimization method based on genetic algorithm of cutter position for the unequal diameter milling of globoidal cam was proposed. Machining errors caused by the different directions of normal vector along the contact were unavoidable when the cutter diameter was smaller than the roller. The practical surface of tool axis trajectory was a ruled surface, so the theoretical surface of tool axis trajectory was reconstructed by the NURBS ruled surface. In order to find the best tool axis vector to minimize the machining error, an optimization model of machining error was put forward. Furthermore, a real coded genetic algorithm has been discussed to figure out the optimization model. At last, a numerical calculation and simulation example were described to verify the effectiveness of the algorithm proposed.