To improve the shift performance stability of the direct-drive automated manual transmission (AMT) gearshift system, inverse system method (ISM) was introduced according to the characteristics of 2-DOF electromagnetic actuator. The structure and working principle of the actuator were described, and the analysis of the mathematical model indicated that the parameters of the actuator affected each other. Static characteristics experiments proved that the output characteristics were nonlinear. The reversibility of the system was analyzed and ISM was adopted to build the pseudo-linear system, and the state feedback controller was designed according to the linear system theory. Extended state observer (ESO) was added to weaken the influence of time varying friction force to the stability of the output characteristics. The simulation and experiment results indicate that controlled by the ISM—ESO controller, the gearshift performance is robust to variable parameters. The precision of displacement control is better. The stability of the output characteristics is improved. Based on the inverse system method, the gearshift performance of the direct-drive automated manual transmission is relatively excellent, and it is conducive to improve the competitiveness of AMT.