As an irreparable brain disease, Alzheimer's disease (AD) seriously impairs human thinking and memory. The accurate diagnosis of AD plays an important role in the treatment of patients. Many machine learning methods have been widely used in classification of AD and its early stage. An increasing number of studies have found that multi-modal data provide complementary information for AD prediction problem. In this paper, we propose multi-modal rank minimization with self-paced learning for revealing the latent correlation across different modalities. In the proposed method, we impose low-rank constraint on the regression coefficient matrix, which is composed of regression coefficient vectors of all modalities. Meanwhile, we adaptively evaluate the contribution of each sample to the fusion model by self-paced learning (SPL). Finally, we utilize multiple-kernel learning (MKL) to classify the multi-modal data. Experiments on the Alzheimer's disease Neuroimaging Initiative (ADNI) databases show that the proposed method obtains better classification performance than the state-of-the-art methods.