In this work, we derive new closed-form expressions for the throughput of non-orthogonal multiple access (NOMA) system with adaptive modulation orders. The system design considers a packet-based transmission where the base station adapts the modulation orders to satisfy the block error rate (BLER) requirement for each user and maximize the throughput by minimizing the maximum signal to noise ratio (SNR) requirements of the users. The optimization problem is formulated as a minmax problem to jointly optimize the modulation orders and SNR thresholds, where the original mixed-integer programming is simplified to integer programming by introducing an auxiliary variable. Compared to the grid search approach, the analytical and simulation results show that the min-max approach can improve the throughput with a significant reduction in the number of transmission modes, the throughput can be improved by up to 2.5 dB at moderate SNRs and by 1 bit/symbol at extremely high SNRs.