Crowd abnormal behavior recognition is essential for intelligent visual surveillance in public places to ensure the safety of the public. This is a challenging work because crowd behaviors are complex which are influenced by various factors. This paper divided these factors into three categories: physical factors, social factors and psychological factors. Then an overview about crowd behavior modeling approaches was given. After that, the paper described and analyzed some influential existing algorithms in crowd abnormal behavior recognition from the view point of behavioral factors they used. Finally, the paper discussed the future research directions in this area and some research proposals were given.