The deadlock avoidance policy of flexible manufacturing system (FMS) was investigated. In order to achieve deadlock avoidance policy and the most use of resource, the method of designing optimal supervisor based on generalized mutual exclusion constraints was presented, the preliminary constrains in the form of linear inequalities were explored, the legal marking set and boundary illegal marking set were gained by reachability analysis and the mixed integer linear programming was built to obtain generalized mutual exclusion constraints as the optimal supervisor by branch and bound method. The system model of FMS was built with Petri net. Based on the occupation and release of resource in the machining process, research was made on application in robot processing system. Both the proposed method and the ones in other literature were adopted to model the process and make comparisons of control performance. The result showed that the optimal supervisors generated from the algorithm could gain stricter constrains and more simplified model, and achieve deadlock avoidance policy that can forbid illegal markings with a maximally permissive way, which proved the effectiveness of the proposed method.