Building energy optimization (BEO) is an emerging technique for achieving energy-efficient building designs. The performance of optimization algorithms is crucial for achieving effective and efficient BEO techniques. In some cases, optimization algorithms can be ineffective, which results in the failure of the BEO process to identify an optimal design. Thus, it is important to investigate the reasons that cause algorithms to be ineffective in BEO. This study begins with a systematic definition of optimization algorithms' ineffectiveness, describing five ineffective scenarios. Then, a reference building and a representative energy optimization problem are proposed. Four commonly used optimization algorithms, namely, discrete Armijo gradient, Hooke-Jeeves, particle swarm optimization with constriction coefficient (PSOCC) and particle swarm optimization with inertia weight (PSOIW), are tested to determine the circumstances and the causal factors under which they become ineffective. The results shed more light on the performance of algorithms in BEO and can be used to help designers avoid ineffective algorithms.