The layout model representation method based on space partition method was established to solve the 3-D mixed layout design. Linear octree and raster data model were used in conversion model for corresponding cylinders layout objects and cuboids layout objects respectively. The filled ball group was proposed for the layout model representation method, and on this foundation, the mathematical model of the layout optimization problem was obtained according to the layout design requirements. Then the TSP problem was transformed into the layout design problem by establishing directed layout sequence. Meanwhile, the fitness function was applied to deal with the multi-objective optimization in the course of the layout, and the improved ant colony algorithm was put into use to gain the Pareto optimum. At last, the effectiveness of the layout method for modeling and optimization algorithm was illustrated by the comparison of the layout design schemes.