Traffic signs play an important role in safety of drivers and regulation of traffic. Traffic sign classification is thus an important problem to solve for the advent of autonomous vehicles. There have been several works that focus on traffic sign classification using various machine learning techniques. While works involving the use of convolutional neural networks with RGB images have shown remarkable results, they require a large amount of training time, and some of these models occupy a huge chunk of memory. Earlier works like HOG-SVM make use of local feature descriptors for classification problem but at the expense of reduced performance. This paper explores the use of hybrid features by combining HOG features and SURF with CNN classifier for traffic sign classification. We propose a unique branching based CNN classifier which achieves an accuracy of 98.48% on GTSRB test set using just 1.5 M trainable parameters.