The accuracy of the Z-axis in the vertical direction of the machine determines the overall performance of the three-dimensional measurement of the micro-nano coordinate measuring machine to a large extent. In order to improve the measurement accuracy of the coordinate measuring machine, it is necessary to study the Z-axis measuring system of the coordinate measuring machine. Firstly, based on the idea of minimizing Abbe error, a Z-axis measurement platform with symmetrical structure of gravity compensator was designed. Secondly, the probe system was composed of scanning electrostatic force microscope. Finally, two separate tests were carried out to verify the rationality and effectiveness of the design. The feasibility of the measurement system with gravity compensator in the static and dynamic was verified separately by the experiment, and then the validity of the scanning electrostatic force microscope probe system compensated with the double height method was verified. The experimental results showed that the novel Z-axis coordinate measuring system had a spatial resolution of 0.6nm with an effective stroke range of 50mm, and had a measurement capability for some non-conductors. The research extended the applicable range of micro-nano coordinate measuring machine and had high application value.