As autonomous mobile machines become more intelligent, the trajectory planning of mobile robots in complex environments often faces the problem of obstacle avoidance failure due to the cluttered and irregular placement of obstacles. Trajectory planning of robot was reduced to optimization problem, that was, the objective function was defined, and then the constraint conditions were set according to the actual planning requirements of the mobile robot, and the appropriate solver was selected. Firstly, the constraint modeling of optimization, including the robot’s kinematics, geometric model, variable extremum constraint and obstacle collision avoidance model were considered. Then an optimization strategy was established to discretize variables by means of interval equipartition of variables, built-in interpolation points and variable description method based on Lagrange polynomial. And for the constraint failure caused by discretization, the variables were discretized by equidistance time and the penalty function was established, so as to realize effective obstacle avoidance. Finally, the stochastic fractal search algorithm was used to solve the above optimization problem. The simulation results showed that the method described can effectively solve the obstacle avoidance problem of mobile robots in complex environments. Also both satisfied the constraints of maximum speed and maximum steering angle. Compared with the existing classical algorithm in the simulation, the experimental results showed that the algorithm described had good robustness in the narrow environment with many obstacles.