Among different 3D data representations, point cloud stands out for its efficiency and flexibility. Hence, many researchers have been involved in the point cloud analysis recently. Existing approaches for point cloud segmentation task typically suffer from two limitations: 1) They usually treat different neighbor points as equals which cannot characterize the correlation between the center point and its neighborhoods well. Moreover, different parts may have different local structures for a point cloud, but they just learn a single representation space which is not sufficient and stable. 2) They often capture hierarchical information by heuristic sampling approaches which cannot reveal the spatial relationships of points well to learn global features. To overcome these limitations, we propose a novel hierarchical attentive pooling graph network (HAPGN) which utilizes the gated graph attention network (GGAN) and hierarchical graph pooling module (HiGPool) as building blocks for point cloud segmentation. Specifically, GGAN can highlight not only the importance of different neighbor points but also the importance of different representation spaces to enhance the local feature extraction. HiGPool is a novel pooling module that can capture the spatial layouts of points to learn the hierarchical features adequately. Experimental results on the ShapeNet part dataset and S3DIS dataset show that HAPGN can achieve superior performance over the state-of-the-art segmentation approaches. Furthermore, we also combine our proposed HiGPool with some recent approaches for point cloud classification and achieve better results on the ModelNet40 dataset.