Convolutional neural networks (CNNs) have been successfully applied in the fields of image classification. The training cost of various large CNN models increases rapidly, which has become a challenging task to balance the recognition accuracy and the convergence speed in the training procedure of CNN models for a particular dataset. In this paper, a KM-Net model with convolutional kernels in the first layer initialized by k-Means clustering algorithm is proposed. Experimental results show that when compared with some other methods on the most widely-used SVHN dataset and ASL dataset, the proposed KM-Net model can not only improve the recognition accuracy, but also accelerate the convergence speed of the training procedure.