Traffic Sign Recognition (TSR) has received widespread attention due to increased traffic accidents caused by a failure to recognize road traffic signs. Most Traffic Sign Recognition remains challenging due to the illumination of natural scenes, and the diversity in traffic signs [31]. To solve the problem, we proposed a two-stage classification system to overcome the above-mentioned challenges and classify the traffic sign boards efficiently. During the first phase, colour based histogram equalizations applied to eliminates the illumination factors and then traffic sign boards are classified into many shapes, which includes circular, triangular, hexagonal, and diamond traffic signs boards based on the modified HOG feature descriptors and various machine learning classifiers are compared [12]. In the second phase, circular and triangular traffic signs are further categorized into specific subclasses using circular and triangular signs trained Convolutional Neural Network (CNN) respectively to overcome intra-class variation among the similar sign board. Experimentation was conducted on German Traffic Sign Recognition Benchmark dataset (GTSRB) [30] and the results show 99.96% top-1 accuracy during the first stage of classification. In the second stage, there is accuracy of 99.40% for circular traffic signs and 97.50% for triangular traffic signs.