Higher annual energy production can be obtained by joint optimization which considers active yaw control in the layout design stage. Although accurate representation of a non-centrosymmetric three-dimensional yawed wake is necessary for the joint optimization of a realistic wind farm, it has not been considered. Furthermore, non-convexity in the joint optimization becomes severe because the layout and yaw angles have to be optimized for all wind directions considering non-centrosymmetric three-dimensional yawed wakes, leading to a not globally but locally optimal layout. To tackle the difficulty, a particle-swarm-optimization-based method which is capable of large-scale non-convex joint optimization is developed. In the present method, a farm layout is globally optimized with simultaneous consideration of yaw angles for various wind speeds and directions. The use of random initial particles which consist of the layout and yaw angles of wind turbines prevent from obtaining a locally optimal layout caused by non-convexity of the problem. The improvement in the annual energy production by the present simultaneously optimized layout is demonstrated.