Intelligent video surveillance systems have garnered substantial research attention in recent years within the transportation surveillance field. The systems can assist in identifying activities, interactions, and abnormal behaviors of individuals in traffic. We propose a novel unsupervised learning framework based on a two-layer BNBP-PFA topic model to simultaneously model multiple types of behaviors in crowded and complicated traffic videos. We provide the model’s structure, its inference algorithm, and design a corresponding likelihood function based on an abnormality detection algorithm. Compared to similar existing algorithms, ours readily reveals both the local topic-motion pattern and the global topic-traffic pattern. Comparative experiments on two public traffic video datasets show that our model outperforms the state-of-art algorithms in regards to effective topic discovery and abnormality detection.