This paper presents a novel and robust descriptor, depth-projection-map-based bag of contour fragments, which is applied to extraction of hand shape and structure information from depth maps. Our method projects depth maps onto three orthogonal planes to generate the depth projection maps. Then, the bag of contour fragment descriptors are extracted from the three depth projection maps and concatenated as a final shape representation of the original depth data. A support vector machine with a linear kernel is used as a shape classifier. The proposed description method is evaluated on three public datasets, as well as a new and more challenging dataset for hand gesture recognition. Results demonstrate that the proposed method significantly outperforms the previous methods on all tested datasets for both static digit recognition and letter gesture recognition. For the challenging HUST-ASL dataset, in particular, the proposed method improves on the previous state-of-the-art methods from 40.1% to 64.6%.