In this paper, we study the optimal degrees of freedom (DoF) region for the two-pair MIMO two-way relay channel (TWRC) with asymmetric antenna setting, where two pairs of users exchange information with the help of a common relay. Each user i is equipped with M antennas, for i = 1, 2, 3, 4, and the relay is equipped with N antennas. First, we prove the converse of the DoF region by using the cut-set theorem and the genie-message approach. Then, we present new transmission schemes to achieve the optimal DoF region at different antenna configurations. Due to the asymmetric data exchange, where the two users in each pair can communicate a different number of data streams, we not only need to form the network-coded symbols but also need to process the additional asymmetric data streams at the relay. This is realized through the joint design of relay compression matrix and source precoding matrices. After obtaining the optimal DoF region, we study the optimal sum DoF by solving a linear programing problem. Our DoF analysis reveals that in the asymmetric antenna setting, some antennas at certain source nodes are redundant and cannot contribute to enlarge the DoF region. It is also found that there is no loss of optimality in terms of the sum DoF by enforcing symmetric data exchange within each user pair.