In order to break through the “NOx-Soot” trade-off curve bound of the diesel engine, a new diesel engine combustion mode in the O2/CO2 environment is proposed. The pressure, temperature fields distribution of the cylinder and the power performance of the engine were simulated by AVL-FIRE software. The influence of various parameters on the engine combustion characteristics were analyzed, so the optimal O2/CO2 mole percent was obtained. Finally, the visualization research of combustion process in cylinder was carried out on the optical engine test bench. The simulation results indicate that when the injection advance angle is unchanged, fuel can’t be ignited or burnt as O2 is below 50%, and the mole percent that 65% O2 and 35% CO2 is the optimum in this scheme;when the injection advance angle is changed, the scheme that 50% O2 and 50% CO2 is the optimum. The visualization tests show that the average temperature in cylinder is lower at the maximum pressure rise rate and the maximum heat release rate, nevertheless the average temperature in cylinder is slightly higher at the peak pressure in the O2/CO2 environment compared to that in the normal intake environment. The diffusion combustion speed during the main combustion period approximately increases by 37.6% in the O2/CO2 environment compared with that in the normal intake environment.