The structure and drive principle of the cycloid ball planetary transmission was introduced. The mathematic models for the tooth surfaces of epitrochoid and hypotrochoid enclosed grooves generated by cone milling cutter were proposed. According to space meshing theory, the meshing equations and the undercut limit functions were deduced. The variational regularity of undercut limit functions with the fundamental design parameters were analyzed through the numerical method. The analytical results showed that the undercutting phenomenon of the tooth surfaces of hypotrochoid and epitrochoid enclosed grooves occurred orderly corresponding to the variation of fundamental design parameters, and the first undercut point always occurred on inside tooth surface of hypotrochoid groove. Consequently, a simpler dimensionless inequality which can avoid the undercut was derived from the proposed analysis. Then, the conclusion was verified by the design examples and the computer simulations. The research results offer a theoretical basis for designing and manufacturing of the cycloid ball planetary transmission. 