As one of the most important performance indexes for electromagnetic actuator with tooth structure, linearity has major impact on the vibration controlling effect. However, traditional fixed tooth electromagnetic actuator has poor linearity. To solve this problem, the analytical model of variable tooth electromagnetic actuator was worked out by analyzing magnetic circuit. Influence rules of several parameters, such as air gap, the difference between tooth width of stator and active cell, coil turns and the radius of the active cell, on actuator performance were analyzed based on the analytical model, indicating that the difference between tooth width of stator and active cell had great influence on the linearity of variable tooth electomagnetic actuator. And the necessity of applying variable tooth to gain good linearity was revealed theoretically and the realization of good linearity needed to be at the expense of decreasing electromagnetic force. To improve the linearity of variable tooth electromagnetic actuator as much as possible and minimize the decrease of electromagnetic force at the same time, genetic algorithm was applied to the optimal design. The results showed different setting for genetic algorithm parameters had negligible effect on optimization results and that compared with the fixed tooth electromagnetic actuator, variable tooth electromagnetic actuator can improve linearity by 93.08% under the condition of electromagnetic force was only dropped by 8.22%. Finally, the simulating calculation for optimized scheme was carried out with the finite model which was verified by the experimental results of the processed prototype, and calculation result proved that the optimized actuator had good linearity in the displacement range of 0~3mm, meanwhile, it validated the analytical model.