Hand gesture recognition system helps in development of interface system for entering text in human computer interaction. In this paper, we have presented a hand gesture recognition system designed for dataset consisting of numerals and alphabets in lower case. The proposed system detects the hand with the help of skin color and motion information. Hand tracking is done with the help of two-level tracking system using modified Kanade–Lucas–Tomasi (KLT) tracking algorithm. The existing KLT was not able to track the gesture trajectory once the skin detected becomes less in area resulting in decreased number of points. In this paper, traditional KLT has been modified with a new additional feature to overcome this difficulty. In feature extraction process, a feature matrix consisting of 30 features have been created. Among these 30 features, few features like density-1, density-2, and perimeter efficiency have been introduced and are used for calculating efficiency along with some existing features. Inclusion of new features helps in improving the performance and accuracy of the system. Recognition is done using six classifiers including SVM (Support Vector machine), Decision Tree, Naïve Bayes, k-NN (K nearest neighbor), ANN (Artificial neural Network) and ELM (Extreme learning Machine). The experimental results prove that 89.67% of accuracy is achieved for the recognition of dataset containing both numerals and alphabets. Our proposed system is also compared with two existing literatures and it has been observed that better accuracy is exhibited by the proposed system.