Understanding pedestrian behavior is a key aspect in the design and testing of a pedestrian pre-collision system. Large-scale naturalistic driving data analysis can provide valuable and objective information on how pedestrians behave in real life. Analyzing pedestrian behavior within large-scale naturalistic driving data requires an efficient pedestrian detection method. However, detecting pedestrians from large-scale naturalistic driving data is challenging due to the high pedestrian appearance variance and fast changing complicated background. In this paper, a categorization-based two-stage pedestrian detection system is proposed to efficiently locate pedestrians within our collected TASI 110-car naturalistic driving dataset. Category information including vehicle status, location, and time is automatically extracted, and efficient category-specific detection algorithms are designed for different scenario categories. The experimental results on the test set show the effectiveness of the proposed approach compared to traditional pedestrian detection methods.