Energy efficiency is a mandatory requirement and integral part of green and sustainable buildings. Energy efficient design optimization is both a design philosophy and a practical technique that has been proposed and used by architects and other professionals for several decades, especially in the past few years. In this review, a set of selection criteria are proposed and 116 works are identified as the core literature. Taking the perspective of architects, analysis is conducted to the core literature to reveal the state of the art of building energy efficient design optimization. The analyzed subjects include the general procedure, the origin and development, the classification, the design objectives and variables, the energy simulation engines, the optimization algorithms, and the applications. The review findings confirm that building energy efficient design optimization is a promising technique to design buildings with higher energy efficiency and better overall performance. However, obstacles still exist and future research is needed.