Through experimental modal analysis and chatter stability lobes simulation, some uncertain factors of machine tool system affecting chatter stability lobes were studied. The research obtained: the cutter flute length, the cutter overhang, the cutter tooth number etc. were main cutter geometry factors affecting the chatter stability；according to actual requirements, cutter overhang should be strictly controlled and cutter parameters should be chosen properly in order to make the system remain high rigidity and cutter tooth number should be chosen reasonably which results in high performance stable milling process. The conclusion obtained has been verified through the industrial example.