The periodic evaluation of health care services is a primary concern for many institutions. We consider services provided by nursing homes with the aim of ranking a set of these structures with respect to their effect on resident health status. Since the overall health status is not directly observable, and given the longitudinal and multilevel structure of the available data, we rely on latent variable models and, in particular, on a multilevel latent Markov model where residents and nursing homes are the first and the second level units, respectively. The model includes individual covariates to account for resident characteristics. The impact of nursing home membership is modelled through a pair of random effects affecting the initial distribution and the transition probabilities between different levels of health status. Through the prediction of these random effects we obtain a ranking of the nursing homes. Furthermore, the proposed model accounts for nonignorable dropout due to resident death, which typically occurs in these contexts. The motivating dataset is gathered from the Long Term Care Facilities programme, a health care protocol implemented in Umbria (Italy). Our results show that differences in performance between nursing homes are statistically significant.