With the explosive growth of remote sensing big data, large-scale remote sensing image retrieval (RSIR) has become one of the most challenging tasks in data mining, attracting more and more attention from researchers. In recent years, deep learning has achieved fruitful results in RSIR. However, there are three major problems. The first problem is due to the imbalanced categories in RSIR data sets. Accordingly, a novel sample mining method is proposed. This method is based on top- “misplaced” samples (Top- MS) along the top- nearest neighbor decision boundary, rather than setting boundary threshold of sample mining and reducing the impact of artificial and objective factors on network training as usual. The second problem comes from the inconsistency between the loss reduction direction and the optimization direction. In response, we additionally propose a novel result ranking loss (RRL) according to the retrieval results in a batch. RRL is used as a plug-in and combined with other local losses. As a result, the retrieval accuracy improved significantly. For the third problem, processing a large number of high-resolution images will increase training time and cause network freeze. To solve the problem, this article proposes a global optimization model based on feature space and retrieval results (FSRR). More specifically, FSRR consists of local loss based on feature space and RRL based on retrieval results, and can be optimized in an end-to-end manner. Experimental results show that our proposed algorithm achieves the best performance when compared with other state-of-the-art deep RSIR algorithms on two benchmark data sets.