A robust nonlinear control approach for modern diesel engine operating under low temperature combustion and conventional compressor ignition combustion modes is described. A nonlinear state space model of intake system is established based on the ideal gas equation and the law of mass conversation. The intake pressure, exhaust pressure and composition of inducted flow are chosen as the system outputs, which have key influence on in-cylinder combustion, and the corresponding controller is designed by the method of integral sliding mode. Smooth switching of combustion modes is achieved by tracking the key engine air-path operating variables to control the in-cylinder conditions indirectly. At last, the effects of the controller were evaluated by co-simulations using GT-Power and Matlab/Simulink and rapid control prototype respectively. The result unveiled that compared with the PI algorithm the innovative control algorithm had significant merits such as good tracking performance, fast response, strong anti-interference and robustness and so on.