This article deals with configuration design and layout optimization of a GEO communication satellite. Layout design optimization of satellite objects are performed based on two-stage algorithm. Finite Circle Method (FCM) is used to express geometric constraints mathematically, and provide a smooth design space. For mathematical expression of performance constraints such as electromagnetic and thermal requirements, the distance constraints between objects have been developed. Then a hybrid method is proposed to optimize layout design which is a combination of simulated annealing optimization and quasi-Newton methods. This approach provides global and deterministic optimal solutions. Developed hybrid method validation is performed on a sample test problem. Finally, the proposed algorithm has been implemented for configuration and optimal layout design of a communication satellite. The results show that considered constraints of communication satellite are satisfied.