Pedestrian gender recognition is a very challenging problem, since the viewpoint variations, illumination changes, occlusion, and poor quality are usually encountered in the pedestrian images. To address this problem, an effective HOG-assisted deep feature learning (HDFL) method is proposed in this paper. The key novelty lies in the design of HDFL network to effectively explore both deep-learned feature and weighted histogram of oriented gradient (HOG) feature for the pedestrian gender recognition. Specifically, the deep-learned and weighted HOG feature extraction branches are simultaneously performed on the input pedestrian image. A feature fusion process is subsequently conducted to obtain a more robust and discriminative feature, which is then fed to a softmax classifier for pedestrian gender recognition. Extensive experiments on multiple existing pedestrian image datasets have shown that the proposed HDFL method is able to effectively recognize the pedestrian gender, and consistently outperforms the state-of-the-art methods.