This paper studies and analyzes a multi-pair MIMO two-way relay (TWR) system from a principal-angle perspective. In this system, users are grouped into pairs and two users in each pair exchange information through a common relay. The transmission scheme consists of two phases. In the uplink phase, the relay's receive signal space is divided into two subspaces for each pair: in one subspace, the signals of the two users are near-parallel, and physical-layer network coding (PNC) decoding is applied to retrieve network-coded messages; in the other subspace, the two user signals are near-orthogonal, and complete decoding (CD) is applied to retrieve individual user messages. In the downlink phase, the relay's transmit signal space is split into orthogonal subspaces for each pair, and in each subspace a common message is broadcasted to the two users. Our main contribution is to establish a critical link between the transmission scheme and the concept of principal angle in linear algebra. In particular, principal angle quantizes the degree of orthogonality between the user signals and helps determining how to partition the relay's receive signal space into PNC decoding subspace and CD decoding subspace. Building on that, we discuss the optimal design of user/relay precoders for both uplink and downlink phases to maximize the asymptotic sum-rate of the proposed scheme at high signal-to-noise ratio (SNR). The analysis and numerical results indicate that our scheme performs close to the cut-set outer bound and significantly outperforms the existing schemes in the literature.