Growing interest in measurement invariance (MI)/equivalence for nonachievement measures across heterogeneous populations in an international context also demands an examination of current criteria for establishing scale score comparability. Namely, in international large-scale assessments (ILSAs) operationally, establishing MI using multiple-groups analyses typically relies on the criteria proposed by research that was limited in scope to few groups and relatively small sample sizes. Recent studies that examined situations resembling those of ILSAs found mixed results with respect to the evaluation criteria for testing MI. The current article extended this line of research by first illustrating the current practices in applied research using empirical examples, and second, by evaluating the performance of several fit measures in two simulation studies when the data were assumed to be ordered categorical and multidimensional in nature. Our simulation study results suggested that in some cases, typical or newly proposed recommendations were not suitable in large-group, varied sample size, multidimensional contexts. Thus, we call for a new measure that more appropriately accounts for complex data characteristics on assessments such as ILSAs.