Human face detection along with its localization is a difficult task when the face is presented in the cluttered scene in an unconstrained scenario that might be with arbitrary pose variations, occlusions, random backgrounds and infrared (IR) environment. This paper proposes a novel face detection method which can address some of these issues and challenges quite successfully during face detection in unconstrained as well as infrared environments. It makes use of Fast Successive Mean Quantization Transform (FastSMQT) features for image enhancement and feature representation to deal with illumination and sensor insensitiveness. A split up Sparse Network of Winnows (SNoW) with Winnow updating rule is then exploited to speed up the original SNoW classifier. Finally, the features and classifiers are combined together with skin detection algorithm for face detection in crowd image and head orientation correction of near infrared faces. The proposed face detector is robust in handling pose, occlusion, illumination, blur and low image resolution. The experiment is performed on six challenging and publicly available databases, viz. BIOID, LFW, FDDB, UFI, WIDER FACE and IIT Delhi near infrared. The experimental results depict that the proposed method outperforms traditional as well as some advanced methods in detecting unconstrained and infrared faces under challenging situations.