This paper deals with the optimization of a new technique of image compression. After the wavelet transform of an image, blocks of coefficients are further linearly decomposed using a basis selected in a dictionary. This dictionary is known by both the encoder and the decoder. This approach is a generalization of the bandelet transform. This paper investigates the problem of the best basis selection. On each block of wavelet coefficients, this selection is made by minimization of a Lagrangian rate-distortion criterion. Theoretical expressions of the optimal Lagrangian multiplier can be computed based on asymptotic hypotheses. A nearly exhaustive search of the optimal Lagrangian multiplier is done for the compression of high resolution satellite images. This numerical study validates the asymptotic theoretical expressions but as well provides a refined expression of the Lagrangian multiplier. At last, the compression results obtained using those different expressions are compared to the optimal compression results obtained with the exhaustive search.