Statistical models have been widely adopted for image set classification owing to their capacity in characterizing the data distribution more flexibly and faithfully. However, these methods typically suffer from the problem that the query image set has weak statistical correlations with the training sets, which leads to larger fluctuations in performance. To address this problem, we propose a semi-supervised fuzzy discriminative learning framework based on Log-Euclidean multivariate Gaussians descriptor to facilitate more robust image set classification. Specifically, by using the semi-supervised setting which definitely has access to the labeled training data and the available unlabeled testing data, we adopt manifold distance metric to construct a “fully trusted” graph and derive two new data dependent probabilistic kernels to strongly reflect the underlying connection relationships between the training and query Gaussian manifold components. The resulted kernel representations are eventually integrated into a kernel fuzzy discriminant framework to enhance the compactness of intra-class Gaussian components and enlarge the margin for inter-class Gaussian components. Thus, more discriminating power of our learning machine is obtained for the classification of the query image set. Extensive experiments on several datasets well demonstrate the effectiveness of the proposed method compared with other image set algorithms.