Generalized Hebbian Algorithm Research Articles

An effective classification approach for big data with parallel generalized Hebbian algorithm

Advancements in information technology is contributing to the excessive rate of big data generation recently. Big data refers to datasets that are huge in volume and consumes much time and space to process and transmit using the available resources. Big data also covers data with unstructured and structured formats. Many agencies are currently subscribing to research on big data analytics owing to the failure of the existing data processing techniques to handle the rate at which big data is generated. This paper presents an efficient classification and reduction technique for big data based on parallel generalized Hebbian algorithm (GHA) which is one of the commonly used principal component analysis (PCA) neural network (NN) learning algorithms. The new method proposed in this study was compared to the existing methods to demonstrate its capabilities in reducing the dimensionality of big data. The proposed method in this paper is implemented using Spark Radoop platform.

Parallel Generalized Hebbian Algorithm for Large Scale Data Analytics

In order to store and analyse large amounts of data on a parallel cluster, Big Data Systems such as Hadoop and DBMSs require a complex configuration and tuning procedure. This is mostly the result of static partitioning occurring whenever data sets are imported into the file system or transferred into it. Following that, parallel processing is carried out in a distributed fashion, with the objective of achieving balanced parallel execution among nodes. The system is notoriously difficult to configure, particularly in the areas of node synchronisation, data redistribution, and distributed caching in main memory. The extended Hebbian algorithm, abbreviated as GHA, is a linear feedforward neural network model for unsupervised learning that finds the majority of its applications in principle components analysis. Sanger's rule is another name for the GHA that may be found in the academic literature. Its formulation and stability, with the additional feature that it may be used to networks that have more than one output. A unique hardware architecture for principal component analysis is presented here in the form of a paper. The Generalized Hebbian Algorithm (GHA) was chosen as the foundation for the design because to the fact that it is both straightforward and efficient. The architecture may be broken down into three distinct parts: the memory unit, the weight vector updating unit, and the primary computing unit. Within the weight vector updating unit, the computation of various synaptic weight vectors uses the same circuit in order to cut down on the area expenses. This is done in order to save space. The GHA architecture incorporates a versatile multi-computer framework that is based on mpi. Therefore, GHA may be efficiently executed on platforms that utilise either sequential processing or parallel processing. When the data set is studied for a short period of time or when a dynamic number of virtual processors is selected at runtime, we predict that our architecture will be able to profit from parallel processing on the cloud. In this research, a parallel implementation of a variety of machine learning algorithms that are built on top of the MapReduce paradigm is presented with the purpose of improving processing speed and saving time.

An Implicit Form of Krasulina's k-PCA Update without the Orthonormality Constraint

We shed new insights on the two commonly used updates for the online k-PCA problem, namely, Krasulina's and Oja's updates. We show that Krasulina's update corresponds to a projected gradient descent step on the Stiefel manifold of orthonormal k-frames, while Oja's update amounts to a gradient descent step using the unprojected gradient. Following these observations, we derive a more implicit form of Krasulina's k-PCA update, i.e. a version that uses the information of the future gradient as much as possible. Most interestingly, our implicit Krasulina update avoids the costly QR-decomposition step by bypassing the orthonormality constraint. A related update, called the Sanger's rule, can be seen as an explicit approximation of our implicit update. We show that the new update in fact corresponds to an online EM step applied to a probabilistic k-PCA model. The probabilistic view of the update allows us to combine multiple models in a distributed setting. We show experimentally that the implicit Krasulina update yields superior convergence while being significantly faster. We also give strong evidence that the new update can benefit from parallelism and is more stable w.r.t. tuning of the learning rate.

Fuzzy Generalized Hebbian Algorithm for Large-Scale Intrusion Detection System

The huge number of irrelevant and redundant data used in building intrusion detection systems (IDS) is one of the common issues in network intrusion detection systems. This paper proposed the use of Fuzzy Generalized Hebbian Algorithm as a novel data reduction method to overcome this problem of data redundancy in IDS. Two methods for dimensionality reduction (GHA and Fuzzy GHA) were used and compared in this study. This allowed retaining the most relevant traffic data information from the network. Furthermore, the K Nearest Neighbor algorithm was applied for the classification of the test connections into 2 categories (attack or normal). The investigations were carried out on the KDDCUP ‘99 dataset and the results showed the Fuzzy GHA method to perform better than GHA in the detection of both U2R and DoS attacks.

Optimizing Extreme Learning Machine via Generalized Hebbian Learning and Intrinsic Plasticity Learning

Traditional extreme learning machine (ELM) has random weights between input layer and hidden layer, this kind of random feature mapping brings non-discriminative feature space and unstable classification accuracy, which greatly limits the performance of the ELM networks. Therefore, to get the well-pleasing input weights, two biologically inspired, unsupervised learning methods were introduced to optimize the traditional ELM networks, namely the generalized hebbian algorithm (GHA) and intrinsic plasticity learning (IPL). The GHA is able to extract the principal components of the input data of arbitrary size, while the IPL tunes the probability density of the neuron’s output towards a desired distribution such as exponential distribution or weber distribution, thereby maximizing the networks information transmission. With the incorporation of the GHA and IPL approach, the optimized ELM networks generates a discriminative feature space and preserves much more characteristic of the input data, accordingly, achieving a better task performance. Based on the above two unsupervised methods, a simple, yet effective hierarchical feature mapping extreme learning machine (HFMELM) is further proposed. With almost no information loss in the layer-wise feature mapping process, the HFMELM is able to learn the high-level representation of the input data. To evaluate the effectiveness of the proposed methods, extensive experiments on several datasets are presented, the results show that the proposed methods significantly outperform the traditional ELM networks.

Experimental Demonstration of Feature Extraction and Dimensionality Reduction Using Memristor Networks.

Memristors have been considered as a leading candidate for a number of critical applications ranging from nonvolatile memory to non-Von Neumann computing systems. Feature extraction, which aims to transform input data from a high-dimensional space to a space with fewer dimensions, is an important technique widely used in machine learning and pattern recognition applications. Here, we experimentally demonstrate that memristor arrays can be used to perform principal component analysis, one of the most commonly used feature extraction techniques, through online, unsupervised learning. Using Sanger's rule, that is, the generalized Hebbian algorithm, the principal components were obtained as the memristor conductances in the network after training. The network was then used to analyze sensory data from a standard breast cancer screening database with high classification success rate (97.1%).

Improving KPCA Online Extraction by Orthonormalization in the Feature Space.

Recently, some online kernel principal component analysis (KPCA) techniques based on the generalized Hebbian algorithm (GHA) were proposed for use in large data sets, defining kernel components using concise dictionaries automatically extracted from data. This brief proposes two new online KPCA extraction algorithms, exploiting orthogonalized versions of the GHA rule. In both the cases, the orthogonalization of kernel components is achieved by the inclusion of some low complexity additional steps to the kernel Hebbian algorithm, thus not substantially affecting the computational cost of the algorithm. Results show improved convergence speed and accuracy of components extracted by the proposed methods, as compared with the state-of-the-art online KPCA extraction algorithms.

A recursive least square algorithm for online kernel principal component extraction

The online extraction of kernel principal components has gained increased attention, and several algorithms proposed recently explore kernelized versions of the generalized Hebbian algorithm (GHA) [1], a well-known principal component analysis (PCA) extraction rule. Consequently, the convergence speed of such algorithms and the accuracy of the extracted components are highly dependent on a proper choice of the learning rate, a problem dependent factor. This paper proposes a new online fixed-point kernel principal component extraction algorithm, exploring the minimization of a recursive least-square error function, conjugated with an approximated deflation transform using component estimates obtained by the algorithm, implicitly applied upon data. The proposed technique automatically builds a concise dictionary to expand kernel components, involves simple recursive equations to dynamically define a specific learning rate to each component under extraction, and has a linear computational complexity regarding dictionary size. As compared to state-of-art kernel principal component extraction algorithms, results show improved convergence speed and accuracy of the components produced by the proposed method in five open-access databases.

An Efficient VLSI Architecture for Multi-Channel Spike Sorting Using a Generalized Hebbian Algorithm.

A novel VLSI architecture for multi-channel online spike sorting is presented in this paper. In the architecture, the spike detection is based on nonlinear energy operator (NEO), and the feature extraction is carried out by the generalized Hebbian algorithm (GHA). To lower the power consumption and area costs of the circuits, all of the channels share the same core for spike detection and feature extraction operations. Each channel has dedicated buffers for storing the detected spikes and the principal components of that channel. The proposed circuit also contains a clock gating system supplying the clock to only the buffers of channels currently using the computation core to further reduce the power consumption. The architecture has been implemented by an application-specific integrated circuit (ASIC) with 90-nm technology. Comparisons to the existing works show that the proposed architecture has lower power consumption and hardware area costs for real-time multi-channel spike detection and feature extraction.

Model, analysis, and evaluation of the effects of analog VLSI arithmetic on linear subspace-based image recognition

Typical image recognition systems operate in two stages: feature extraction to reduce the dimensionality of the input space, and classification based on the extracted features. Analog Very Large Scale Integration (VLSI) is an attractive technology to achieve compact and low-power implementations of these computationally intensive tasks for portable embedded devices. However, device mismatch limits the resolution of the circuits fabricated with this technology. Traditional layout techniques to reduce the mismatch aim to increase the resolution at the transistor level, without considering the intended application. Relating mismatch parameters to specific effects in the application level would allow designers to apply focalized mismatch compensation techniques according to predefined performance/cost tradeoffs.This paper models, analyzes, and evaluates the effects of mismatched analog arithmetic in both feature extraction and classification circuits. For the feature extraction, we propose analog adaptive linear combiners with on-chip learning for both Least Mean Square (LMS) and Generalized Hebbian Algorithm (GHA). Using mathematical abstractions of analog circuits, we identify mismatch parameters that are naturally compensated during the learning process, and propose cost-effective guidelines to reduce the effect of the rest. For the classification, we derive analog models for the circuits necessary to implement Nearest Neighbor (NN) approach and Radial Basis Function (RBF) networks, and use them to emulate analog classifiers with standard databases of face and hand-writing digits. Formal analysis and experiments show how we can exploit adaptive structures and properties of the input space to compensate the effects of device mismatch at the application level, thus reducing the design overhead of traditional layout techniques. Results are also directly extensible to multiple application domains using linear subspace methods.

Efficient Architecture for Spike Sorting in Reconfigurable Hardware

This paper presents a novel hardware architecture for fast spike sorting. The architecture is able to perform both the feature extraction and clustering in hardware. The generalized Hebbian algorithm (GHA) and fuzzy C-means (FCM) algorithm are used for feature extraction and clustering, respectively. The employment of GHA allows efficient computation of principal components for subsequent clustering operations. The FCM is able to achieve near optimal clustering for spike sorting. Its performance is insensitive to the selection of initial cluster centers. The hardware implementations of GHA and FCM feature low area costs and high throughput. In the GHA architecture, the computation of different weight vectors share the same circuit for lowering the area costs. Moreover, in the FCM hardware implementation, the usual iterative operations for updating the membership matrix and cluster centroid are merged into one single updating process to evade the large storage requirement. To show the effectiveness of the circuit, the proposed architecture is physically implemented by field programmable gate array (FPGA). It is embedded in a System-on-Chip (SOC) platform for performance measurement. Experimental results show that the proposed architecture is an efficient spike sorting design for attaining high classification correct rate and high speed computation.

Compressed Sensing with Generalized Hebbian Algorithm in Video Frame Prediction

Compressed Sensing with Generalized Hebbian Algorithm (GHA) in Video Frame Prediction is proposed in the paper. After analyzing the inter-frame correlation among the images of video sequences, GHA, as neural network algorithm of PCA, is adopted to remove the transform coefficients with lower value according in order to implement video compressed sensing. Furthermore, for statistics of the adjacent frames are similar enough, the algorithm processes superiority in video frame prediction. Simulation results show that, the proposed algorithm can not only improve the reconstructed quality and the visual effects of the video sequence, but also save the sampling resources. Moreover, video frames can be predicted capitally through the application of the algorithm.

FPGA Implementation of Generalized Hebbian Algorithm for Texture Classification

This paper presents a novel hardware architecture for principal component analysis. The architecture is based on the Generalized Hebbian Algorithm (GHA) because of its simplicity and effectiveness. The architecture is separated into three portions: the weight vector updating unit, the principal computation unit and the memory unit. In the weight vector updating unit, the computation of different synaptic weight vectors shares the same circuit for reducing the area costs. To show the effectiveness of the circuit, a texture classification system based on the proposed architecture is physically implemented by Field Programmable Gate Array (FPGA). It is embedded in a System-On-Programmable-Chip (SOPC) platform for performance measurement. Experimental results show that the proposed architecture is an efficient design for attaining both high speed performance and low area costs.

Efficient hardware architecture based on generalized Hebbian algorithm for texture classification

The objective of this paper is to present an efficient hardware architecture for generalized Hebbian algorithm (GHA). In the architecture, the principal component computation and weight vector updating of the GHA are operated in parallel, so that the throughput of the circuit can be significantly enhanced. In addition, the weight vector updating process is separated into a number of stages for lowering area costs and increasing computational speed. To show the effectiveness of the circuit, a texture classification system based on the proposed architecture is designed. It is embedded in a system-on-programmable-chip (SOPC) platform for physical performance measurement. Experimental results show that the proposed architecture is an efficient design for attaining both high speed performance and low area costs.

Global Convergence of GHA Learning Algorithm With Nonzero-Approaching Adaptive Learning Rates

The generalized Hebbian algorithm (GHA) is one of the most widely used principal component analysis (PCA) neural network (NN) learning algorithms. Learning rates of GHA play important roles in convergence of the algorithm for applications. Traditionally, the learning rates of GHA are required to converge to zero so that its convergence can be analyzed by studying the corresponding deterministic continuous-time (DCT) equations. However, the requirement for learning rates to approach zero is not a practical one in applications due to computational roundoff limitations and tracking requirements. In this paper, nonzero-approaching adaptive learning rates are proposed to overcome this problem. These proposed adaptive learning rates converge to some positive constants, which not only speed up the algorithm evolution considerably, but also guarantee global convergence of the GHA algorithm. The convergence is studied in detail by analyzing the corresponding deterministic discrete-time (DDT) equations. Extensive simulations are carried out to illustrate the theory.

Determination of the Number of Principal Directions in a Biologically Plausible PCA Model

Adaptively determining an appropriate number of principal directions for principal component analysis (PCA) neural networks is an important problem to address when one uses PCA neural networks for online feature extraction. In this letter, inspired from biological neural networks, a single-layer neural network model with lateral connections is proposed which uses an improved generalized Hebbian algorithm (GHA) to address this problem. In the proposed model, the number of principal directions can be adaptively determined to approximate the intrinsic dimensionality of the given data set so that the dimensionality of the data set can be reduced to approach the intrinsic dimensionality to any required precision through the network.

Perturbation-Based Eigenvector Updates for On-Line Principal Components Analysis and Canonical Correlation Analysis

Principal components analysis is an important and well-studied subject in statistics and signal processing. Several algorithms for solving this problem exist, and could be mostly grouped into one of the following three approaches: adaptation based on Hebbian updates and deflation, optimization of a second order statistical criterion (like reconstruction error or output variance), and fixed point update rules with deflation. In this study, we propose an alternate approach that avoids deflation and gradient-search techniques. The proposed method is an on-line procedure based on recursively updating the eigenvector and eigenvalue matrices with every new sample such that the estimates approximately track their true values as would be calculated analytically from the current sample estimate of the data covariance matrix. The perturbation technique is theoretically shown to be applicable for recursive canonical correlation analysis, as well. The performance of this algorithm is compared with that of a structurally similar matrix perturbation-based method and also with a few other traditional methods like Sanger's rule and APEX.

Learning principal directions: Integrated-squared-error minimization

A common derivation of principal component analysis (PCA) is based on the minimization of the squared-error between centered data and linear model, corresponding to the reconstruction error. In fact, minimizing the squared-error leads to principal subspace analysis where scaled and rotated principal axes of a set of observed data, are estimated. In this paper, we introduce and investigate an alternative error measure, integrated-squared error (ISE), the minimization of which determines the exact principal axes (without rotational ambiguity) of a set of observed data. We show that exact principal directions emerge from the minimization of ISE. We present a simple EM algorithm, ‘EM-ePCA’, which is similar to EM-PCA [S.T. Roweis, EM algorithms for PCA and SPCA, in: Advances in Neural Information Processing Systems, vol. 10, MIT Press, Cambridge, 1998, pp. 626–632.], but finds exact principal directions without rotational ambiguity. In addition, we revisit the generalized Hebbian algorithm (GHA) and show that it emerges from the ISE minimization in a single-layer linear feedforward neural network.

Recursive Principal Components Analysis Using Eigenvector Matrix Perturbation

Principal components analysis is an important and well-studied subject in statistics and signal processing. The literature has an abundance of algorithms for solving this problem, where most of these algorithms could be grouped into one of the following three approaches: adaptation based on Hebbian updates and deflation, optimization of a second-order statistical criterion (like reconstruction error or output variance), and fixed point update rules with deflation. In this paper, we take a completely different approach that avoids deflation and the optimization of a cost function using gradients. The proposed method updates the eigenvector and eigenvalue matrices simultaneously with every new sample such that the estimates approximately track their true values as would be calculated from the current sample estimate of the data covariance matrix. The performance of this algorithm is compared with that of traditional methods like Sanger's rule and APEX, as well as a structurally similar matrix perturbation-based method.

A neural-network-based method of model reduction for the dynamic simulation ofMEMS

This paper proposes a neuro-network-based method for modelreduction that combines the generalized Hebbian algorithm (GHA) with theGalerkin procedure to perform the dynamic simulation and analysis ofnonlinear microelectromechanical systems (MEMS). An unsupervised neuralnetwork is adopted to find the principal eigenvectors of a correlationmatrix of snapshots. It has been shown that the extensive computer resultsof the principal component analysis using the neural network of GHA canextract an empirical basis from numerical or experimental data, which canbe used to convert the original system into a lumped low-order macromodel.The macromodel can be employed to carry out the dynamic simulation of theoriginal system resulting in a dramatic reduction of computation timewhile not losing flexibility and accuracy. Compared with other existingmodel reduction methods for the dynamic simulation of MEMS, the presentmethod does not need to compute the input correlation matrix in advance.It needs only to find very few required basis functions, which can belearned directly from the input data, and this means that the methodpossesses potential advantages when the measured data are large. Themethod is evaluated to simulate the pull-in dynamics of a doubly-clampedmicrobeam subjected to different input voltage spectra of electrostaticactuation. The efficiency and the flexibility of the proposed method areexamined by comparing the results with those of the fully meshedfinite-difference method.