The effects of fluctuation of cutting force，the mass eccentricity and the hydrodynamic force of cutting fluid can be taken into consideration in the model of drilling shaft system. To solve the hydrodynamic force of cutting fluid, the variational form of Reynolds equation in hydrodynamic fluid was revised continuously to satisfy certain constraint conditions by variational constraint approach. By means of this approach, a perturbed equation could be obtained directly on the finite element equation. Consequently, nonlinear hydrodynamic force and their Jacobian were calculated simultaneously, and compatible accuracy has been obtained without increasing computational cost. In the stability analysis, the period of the periodic orbit of the nonlinear drilling shaft system was drawn into the governing equation of the system explicitly by changing the time scale, and was taken part into the iteration procedure of the shooting method as a parameter. Then the periodic orbit and its period of the drilling shaft system were determined rapidly. Combining theories with experiment, the local stability and bifurcation behaviors of periodic motions with the change of the drilling shaft design parameter value were obtained by the Floquet theory.