A novel direction for efficiently describing face images is proposed by exploring the relationships between both gradient orientations and magnitudes of different local image structures. Presented in this paper are not only a novel feature set called patterns of orientation difference (POD) but also several improvements to our previous algorithm called patterns of oriented edge magnitudes (POEM). The whitened principal component analysis (PCA) dimensionality reduction technique is applied upon both the POEM- and POD-based representations to get more compact and discriminative face descriptors. We then show that the two methods have complementary strength and that by combining the two algorithms, one obtains stronger results than either of them considered separately. By experiments carried out on several common benchmarks, including the FERET database with both frontal and nonfrontal images as well as the very challenging LFW data set, we prove that our approach is more efficient than contemporary ones in terms of both higher performance and lower complexity.