One of the most important methods used to cope with multipath fading effects, which cause the symbol to be received incorrectly in wireless communication systems, is the use of multiple transceiver antenna structures. By combining the multi-input multi-output (MIMO) antenna structure with non-orthogonal multiple access (NOMA), which is a new multiplexing method, the fading effects of the channels are not only reduced but also high data rate transmission is ensured. However, when the maximum likelihood (ML) algorithm that has high performance on coherent detection, is used as a symbol detector in MIMO NOMA systems, the computational complexity of the system increases due to higher-order constellations and antenna sizes. As a result, the implementation of this algorithm will be impractical. In this study, the backtracking search algorithm (BSA) is proposed to reduce the computational complexity of the symbol detection and have a good bit error performance for MIMO-NOMA systems. To emphasize the efficiency of the proposed algorithm, simulations have been made for the system with various antenna sizes. As can be seen from the obtained results, a considerable reduction in complexity has occurred using BSA compared to the ML algorithm, also the bit error performance of the system is increased compared to other algorithms.