So-called Small Sample Size Research Articles

A Revised Formation of Trace Ratio LDA for Small Sample Size Problem.

Linear discriminant analysis (LDA) is a classic tool for supervised dimensionality reduction. Because the projected samples can be classified effectively, LDA has been successfully applied in many applications. Among the variants of LDA, trace ratio LDA (TR-LDA) is a classic form due to its explicit meaning. Unfortunately, when the sample size is much smaller than the data dimension, the algorithm for solving TR-LDA does not converge. The so-called small sample size (SSS) problem severely limits the application of TR-LDA. To solve this problem, we propose a revised formation of TR-LDA, which can be applied to datasets with different sizes in a unified form. Then, we present an optimization algorithm to solve the proposed method, explain why it can avoid the SSS problem, and analyze the convergence and computational complexity of the optimization algorithm. Next, based on the introduced theorems, we quantitatively elaborate on when the SSS problem will occur in TR-LDA. Finally, the experimental results on real-world datasets demonstrate the effectiveness of the proposed method.

A General Matrix Function Dimensionality Reduction Framework and Extension for Manifold Learning.

Many dimensionality reduction methods in the manifold learning field have the so-called small-sample-size (SSS) problem. Starting from solving the SSS problem, we first summarize the existing dimensionality reduction methods and construct a unified criterion function of these methods. Then, combining the unified criterion with the matrix function, we propose a general matrix function dimensionality reduction framework. This framework is configurable, that is, one can select suitable functions to construct such a matrix transformation framework, and then a series of new dimensionality reduction methods can be derived from this framework. In this article, we discuss how to choose suitable functions from two aspects: 1) solving the SSS problem and 2) improving pattern classification ability. As an extension, with the inverse hyperbolic tangent function and linear function, we propose a new matrix function dimensionality reduction framework. Compared with the existing methods to solve the SSS problem, these new methods can obtain better pattern classification ability and have less computational complexity. The experimental results on handwritten digit, letters databases, and two face databases show the superiority of the new methods.

Exponential Local Discriminant Embedding and Its Application to Face Recognition

Local discriminant embedding (LDE) has been recently proposed to overcome some limitations of the global linear discriminant analysis method. In the case of a small training data set, however, LDE cannot directly be applied to high-dimensional data. This case is the so-called small-sample-size (SSS) problem. The classical solution to this problem was applying dimensionality reduction on the raw data (e.g., using principal component analysis). In this paper, we introduce a novel discriminant technique called "exponential LDE" (ELDE). The proposed ELDE can be seen as an extension of LDE framework in two directions. First, the proposed framework overcomes the SSS problem without discarding the discriminant information that was contained in the null space of the locality preserving scatter matrices associated with LDE. Second, the proposed ELDE is equivalent to transforming original data into a new space by distance diffusion mapping (similar to kernel-based nonlinear mapping), and then, LDE is applied in such a new space. As a result of diffusion mapping, the margin between samples belonging to different classes is enlarged, which is helpful in improving classification accuracy. The experiments are conducted on five public face databases: Yale, Extended Yale, PF01, Pose, Illumination, and Expression (PIE), and Facial Recognition Technology (FERET). The results show that the performances of the proposed ELDE are better than those of LDE and many state-of-the-art discriminant analysis techniques.

A Discriminant Model for the Pattern Recognition of Linearly Independent Samples

Fisher’s linear discriminant analysis (FLDA) has been used in many pattern recognition applications. However, this method cannot be applied for solving the pattern recognition problems when the within-class scatter matrix is singular, i.e., when the so-called small sample size problem occurs. Many FLDA variants that have been proposed in the past to circumvent this problem either suffer from excessive computational load when patterns have a large dimension or lose some useful discriminant information. In this paper, a new systematic framework for the pattern recognition of datasets with linearly independent samples is developed. Within this framework, a discriminant model, in which the samples of the individual classes of a dataset lie on parallel hyperplanes and project to single distinct points of a discriminant subspace of the underlying input space, is shown to exist. Based on this model, three algorithms that do not encounter the adverse effects of the small sample size (SSS) problem are developed to obtain such a discriminant subspace for a given dataset with linearly independent samples. A kernelized algorithm is also developed for the discriminant analysis of datasets for which the samples are not linearly independent. Simulation results are provided to examine the validity of the proposed discriminant model and to demonstrate the effectiveness, both in terms of complexity and classification accuracy, of the linear and nonlinear algorithms designed based on the proposed model.

Principal component analysis-based techniques and supervised classification schemes for the early detection of Alzheimer's disease

In Alzheimer's disease (AD) diagnosis process, functional brain image modalities such as Single-Photon Emission Computed Tomography (SPECT) and Positron Emission Tomography (PET) have been widely used to guide the clinicians. However, the current evaluation of these images entails a succession of manual reorientations and visual interpretation steps, which attach in some way subjectivity to the diagnostic. In this work, a complete computer aided diagnosis (CAD) system for an automatic evaluation of the neuroimages is presented. Principal component analysis (PCA)-based methods are proposed as feature extraction techniques, enhanced by other linear approaches such as linear discriminant analysis (LDA) or the measure of the Fisher discriminant ratio (FDR) for feature selection. The final features allow to face up the so-called small sample size problem and subsequently they are used for the study of neural networks (NN) and support vector machine (SVM) classifiers. The combination of the presented methods achieved accuracy results of up to 96.7% and 89.52% for SPECT and PET images, respectively, which means a significant improvement over the results obtained by the classical voxels-as-features (VAF) reference approach.

GMM based SPECT image classification for the diagnosis of Alzheimer’s disease

We present a novel classification method of SPECT images based on Gaussian mixture models (GMM) for the diagnosis of Alzheimer’s disease. The aims of the model-based approach for density estimation is to automatically select regions of interest (ROIs) and to effectively reduce the dimensionality of the problem. The resulting Gaussians are constructed according to a maximum likelihood criterion employing the Expectation Maximization (EM) algorithm. By considering only the intensity levels inside the Gaussians, the resulting feature space has a significantly reduced dimensionality with respect to former approaches using the voxel intensities directly as features (VAF). With this feature extraction method one relieves the effects of the so-called small sample size problem and nonlinear classifiers may be used to distinguish between the brain images of normal and Alzheimer patients. Our results show that for various classifiers the GMM-based method yields higher accuracy rates than the classification considering all voxel values.

SUBSPACES VERSUS SUBMANIFOLDS: A COMPARATIVE STUDY IN SMALL SAMPLE SIZE PROBLEM

Automatic face recognition is a challenging problem in the biometrics area, where the dimension of the sample space is typically larger than the number of samples in the training set and consequently the so-called small sample size problem exists. Recently, neuroscientists emphasized the manifold ways of perception, and showed the face images may reside on a nonlinear submanifold hidden in the image space. Many manifold learning methods, such as Isometric feature mapping, Locally Linear Embedding, and Locally Linear Coordination are proposed. These methods achieved the submanifold by collectively analyzing the overlapped local neighborhoods and all claimed to be superior to such subspace methods as Eigenfaces and Fisherfaces in terms of classification accuracy. However, in literature, no systematic comparative study for face recognition is performed among them. In this paper, we carry out a comparative study in face recognition among them, and the study considers theoretical aspects as well as simulations performed using CMU PIE and FERET face databases.

WAVELET-FACE BASED SUBSPACE LDA METHOD TO SOLVE SMALL SAMPLE SIZE PROBLEM IN FACE RECOGNITION

Linear Discriminant Analysis (LDA) is a popular statistical method for both feature extraction and dimensionality reduction in face recognition. The major drawback of LDA is the so-called small sample size (3S) problem. This problem always occurs when the total number of training samples is smaller than the dimension of feature space. Under this situation, the within-class scatter matrix Sw becomes singular and LDA approach cannot be implemented directly. To overcome the 3S problem, this paper proposes a novel wavelet-face based subspace LDA algorithm. Wavelet-face feature extraction and dimensionality reduction are based on two-level D4-filter wavelet transform and discarding the null space of total class scatter matrix St. It is shown that our obtained projection matrix satisfies the uncorrelated constraint conditions. Hence in the sense of statistical uncorrelation, this projection matrix is optimal. The proposed method for face recognition has been evaluated with two public available databases, namely ORL and FERET databases. Comparing with existing LDA-based methods to solve the 3S problem, our method gives the best performance.

Kernel quadratic discriminant analysis for small sample size problem

It is generally believed that quadratic discriminant analysis (QDA) can better fit the data in practical pattern recognition applications compared to linear discriminant analysis (LDA) method. This is due to the fact that QDA relaxes the assumption made by LDA-based methods that the covariance matrix for each class is identical. However, it still assumes that the class conditional distribution is Gaussian which is usually not the case in many real-world applications. In this paper, a novel kernel-based QDA method is proposed to further relax the Gaussian assumption by using the kernel machine technique. The proposed method solves the complex pattern recognition problem by combining the QDA solution and the kernel machine technique, and at the same time, tackles the so-called small sample size problem through a regularized estimation of the covariance matrix. Extensive experimental results indicate that the proposed method is a more sophisticated solution outperforming many traditional kernel-based learning algorithms.

TWO-STEP SINGLE PARAMETER REGULARIZATION FISHER DISCRIMINANT METHOD FOR FACE RECOGNITION

In face recognition tasks, Fisher discriminant analysis (FDA) is one of the promising methods for dimensionality reduction and discriminant feature extraction. The objective of FDA is to find an optimal projection matrix, which maximizes the between-class-distance and simultaneously minimizes within-class-distance. The main limitation of traditional FDA is the so-called Small Sample Size (3S) problem. It induces that the within-class scatter matrix is singular and then the traditional FDA fails to perform directly for pattern classification. To overcome 3S problem, this paper proposes a novel two-step single parameter regularization Fisher discriminant (2SRFD) algorithm for face recognition. The first semi-regularized step is based on a rank lifting theorem. This step adjusts both the projection directions and their corresponding weights. Our previous three-to-one parameter regularized technique is exploited in the second stage, which just changes the weights of projection directions. It is shown that the final regularized within-class scatter matrix approaches the original within-class scatter matrix as the single parameter tends to zero. Also, our method has good computational complexity. The proposed method has been tested and evaluated with three public available databases, namely ORL, CMU PIE and FERET face databases. Comparing with existing state-of-the-art FDA-based methods in solving the S3 problem, the proposed 2SRFD approach gives the best performance.

Discriminant common vectors versus neighbourhood components analysis and Laplacianfaces: A comparative study in small sample size problem

Discriminant common vectors (DCV), neighbourhood components analysis (NCA) and Laplacianfaces (LAP) are three recently proposed methods which can effectively learn linear projection matrices for dimensionality reduction in face recognition, where the dimension of the sample space is typically larger than the number of samples in the training set and consequently the so-called small sample size (SSS) problem exists. The three methods obtained their respective projection matrices based on different objective functions and all claimed to be superior to such methods as Principal component analysis (PCA) and PCA plus Linear discriminant analysis (PCA+LDA) in terms of classification accuracy. However, in literature, no comparative study is carried out among them. In this paper, we carry out a comparative study among them in face recognition (or generally in the SSS problem), and argue that the projection matrix yielded by DCV is the optimal solution to both NCA and LAP in terms of their respective objective functions, whereas neither NCA nor LAP may get their own optimal solutions. In addition, we show that DCV is more efficient than both NCA and LAP for both linear dimensionality reduction and subsequent classification in SSS problem. Finally, experiments are conducted on ORL, AR and YALE face databases to verify our arguments and to present some insights for future study.

Foley–Sammon Optimal Discriminant Vectors Using Kernel Approach

A new nonlinear feature extraction method called kernel Foley-Sammon optimal discriminant vectors (KFSODVs) is presented in this paper. This new method extends the well-known Foley-Sammon optimal discriminant vectors (FSODVs) from linear domain to a nonlinear domain via the kernel trick that has been used in support vector machine (SVM) and other commonly used kernel-based learning algorithms. The proposed method also provides an effective technique to solve the so-called small sample size (SSS) problem which exists in many classification problems such as face recognition. We give the derivation of KFSODV and conduct experiments on both simulated and real data sets to confirm that the KFSODV method is superior to the previous commonly used kernel-based learning algorithms in terms of the performance of discrimination.