Convolutional Neural Networks (CNNs) have emerged as a popular choice of researchers for their robust feature extraction and information mining capability. In the last decades, CNNs have depicted impressive performance on various applications of computer vision tasks like object detection, image segmentation, and image classification. As a consequence, the ear-based recognition system has not gained many benefits from deep learning and CNN-based applications and is still lacking behind due to the availability of sufficient data and varying conditions of captured sample images. In this paper, transfer learning techniques have been applied to the well-known convolutional neural network model VGG16 integrated with the support vector machine(SVM) that acts as a hybrid algorithm for recognizing the person using their ear images. The proposed model is validated on an ear dataset containing a total of 2600 images with variability in terms of pose, rotation, and illumination changes. The proposed model is able to classify the ear images with the highest recognition accuracy of 98.72%. To show the effectiveness of the proposed model, comparative studies of the proposed model with other existing methods have been reported in the literature.