Out-of-order, speculative, superscalar processors are complex. The behavior of multiprocessor systems that use such processors is not well understood and very difficult to predict. We tackle this problem using a powerful simulator, Armadillo, and a novel characterization framework that breaks the instruction pipeline into five meta-stages. The Armadillo simulator models symmetric multiprocessors (SMPs) constructed from highly aggressive superscalar processors on a shared bus, and is able to provide accurate, detailed statistics on numerous aspects of the simulated system, including the amount of time each instruction spends in each of these five meta-stages. We also analyze the fraction of each instruction's lifetime, during which it remains speculative and the amount of time that an instruction spends on the critical path. To demonstrate the effectiveness of this approach, we apply the characterization to applications from the SPLASH-2 benchmark suite. We evaluated the applications' sensitivity to key memory system parameters: bus frequency, bus width, memory latency, and cache latency.