Traffic sign detection is a very important part of intelligent assisted driving system. However, with the interference of various target sizes, geometric distortion, occlusion and motion blur, fast and accurate detection on large-size car camera image is extremely hard. To achieve both high efficient and accurate detection, we present a traffic sign detection method within a coarse-to-fine framework, which sequentially detects the targets in grid-level and image-level. We demonstrate that focusing first is a more effective detection strategy for small targets in wide detection space. We propose a target grid prediction network, which is a fully convolutional network for binary classification, to realize rapid coarse localization of the target and effectively guide the clipping and scaling of the target area. With the flexible potential target region extracting strategy, the detecting space can be significantly reduced. At the same time, the correctly extracted local areas for the targets can further facilitate the accurate detection of the subsequent traffic sign detector. In the experiments, our method achieves impressive performance in terms of both efficiency and accuracy. On the challenging Tsinghua-Tencent 100K (TT100K) dataset, our method achieves 20.9 FPS detection speed for 1600 × 1600 images and the F 1 -score of the proposed method for all-scale targets is 91.55.