People reliably and automatically make personality inferences from facial appearance despite little evidence for their accuracy. Although such inferences are highly inter-correlated, research has traditionally focused on studying specific traits such as trustworthiness. We advocate an alternative, data-driven approach to identify and model the structure of face evaluation. Initial findings indicate that specific trait inferences can be represented within a 2D space defined by valence/trustworthiness and power/dominance evaluation of faces. Inferences along these dimensions are based on similarity to expressions signaling approach or avoidance behavior and features signaling physical strength, respectively, indicating that trait inferences from faces originate in functionally adaptive mechanisms. We conclude with a discussion of the potential role of the amygdala in face evaluation.