In order to improve the geometric accuracy of high-end CNC machine tools, a machine tool error compensation model based on tensor theory was proposed. Firstly, the error tensor model was established by setting the basic order, the extended order and the error order, which was applicable for error compensation table. Secondly, the error compensation was carried out in three processes, which were setting machine tool information, establishing tensor model and setting detection parameter. Referring to the ISO 230-2 standard, the modeling flow and detection parameters of the error tensor model were presented. Using multiple regression analysis, the linear fitting and residual errors were 2.5μm, while the four polynomial fit residuals were 1μm. Finally, taking CNC turn-milling machine experiment as an example, the error tensor model by step-diagonal measurement was proposed. The analysis of experimental data showed that the pitch error by three order fitting can improve the fitting precision by 59.4%~92.5%, the sum of squared residuals was less than 0.00003;the straightness error needed five order fitting for nonlinear characteristics. The ballbar experiment of roundness error showed that roundness error was increased by 12.2%~27.7%, while only compensated the pitch error. For example, the compensation using error tensor model can be reduced by 22.3% of roundness error at XY plane by three order fitting, which was optimized by 26.4%~95.5% compared with that of the linear interpolation. The experiment showed that the error compensation should be paid more attention to the straightness error. The method of error tensor model can effectively improve the accuracy of machine tools for both pitch error and straightness error.