In this paper, we propose a novel framework for speeding up the test-time of traffic sign recognition, which is named Branch Convolution Neural Network. It is the first time to introduce a branch-output mechanism into a deep Convolution Neural Network. Our model has an accuracy as high as a deep convolution neural network model, while it performs faster at the same condition during test stage. It is a significantly accelerated framework for designing a real-time deep neural network system. We present a detail process to change a regular pre-trained Convolution Neural Network into a Branch Convolution Neural Network: train several simple branch classifiers, bias classifiers and optimize branches. Experiment applied on GTSRB shows that large number of traffic signs are unnecessary to go through all layers in a deep model and they can be separated out in a relative shallow neural network. This framework speeds up the recognition progress, while keeping the accuracy within an extremely minor drop.