Metaheuristic algorithms have emerged as the best way of solving complex optimization problems. Consequently, the literature includes a large and growing number of applications of metaheuristics for the size optimization of trusses. Generally speaking, these studies do not focus in the comparison between algorithms. Motivated by this, we present a framework for an unbiased and meaningful comparison between different metaheuristic methods. Based on this framework, a critical evaluation of a number of metaheuristic algorithms is presented, which includes Genetic Algorithms, Particle Swarm Optimization, Artificial Bee Colony, Simulated Annealing and Differential Evolution variants. The differences in performance are highlighted and an explanation for their behavior is attempted. It is found that Differential Evolution is the best optimizer in terms of performance, robustness and scalability. We also demonstrate that, although the methods considered in this study are well-established, often better designs than the ones found in the literature are discovered.