The multiple-input multiple-output (MIMO) Y channel, where three users simultaneously exchange independent messages with each other via a single relay within two time slots, is considered in this paper. We first propose a cooperative network coding protocol, which is called denoise-demodulate-and-forward (DDF), with the design of transmit beamforming and combining schemes to increase network throughput. More importantly, we formulate an optimization problem by using the newly derived bit error rate (BER) expression of adaptive M-ary quadrature amplitude modulation (M-QAM). As the result, the modulation types for both time slots can be chosen to maximize the total throughput of the proposed system under the BER constraint. Performance evaluations show that the proposed scheme can significantly improve the total throughput by comparing to the existing MIMO Y channel and the solution of the optimization problem is validated.