Faster Training Time Research Articles

Very fast training neural-computation techniques for real measure-correlate-predict wind operations in wind farms

The real operation of a wind farm implies the solution of many different problems related to wind speed at a wind farm location site. Wind speed prediction and wind series reconstruction are the two examples of important problems tackled in wind farm management and prospection. Usually, wind speed prediction and reconstruction of wind series are carried out in wind farms using data from in situ measuring towers, usually named as Measure-Correlate-Predict methods (MCP). MCP processes consist, therefore, in the wind speed prediction or reconstruction from neighbor stations, using different methods. In this paper, we tackle the special case of real MCP operations in wind farms, in which the algorithms to reconstruct or predict the wind series must be extremely fast in order to be useful. We present the application of two state-of-the-art neural networks which have shown a very fast training time, with an excellent performance in terms of accuracy. Specifically, we show the application of Group Method of Data Handling and Extreme Learning Machines in the MCP reconstruction and prediction of wind speed series, in a real wind farm in Spain. A comparison in terms of computation time and accuracy with alternative algorithms in the literature is also carried out. Finally, we show a real implementation of the Group Method of Data Handling (GMDH) and Extreme Learning Machine (ELM) in a software in use for real MCP operations in wind farms.

Detection of Tuberculosis Bacilli in Tissue Slide Images using HMLP Network Trained by Extreme Learning Machine

This paper proposes an automated detection of tuberculosis bacilli in Ziehl-Neelsen-stained tissue slides using image processing and neural network. Image segmentation using CY-based colour filter and k-mean clustering procedure is used to separate objects of interest from the background. A number of geometrical features are then extracted from the segmented images. A recent training algorithm called Extreme Learning Machine (ELM) is modified to train a hybrid multilayered perceptron network (HMLP) for the classification task. The results indicate that the performance of HMLP-ELM network is comparable to the previously proposed methods and offers a fast training time with no designing parameter required. Ill. 6, bibl. 15, tabl. 1 (in English; abstracts in English and Lithuanian).DOI: http://dx.doi.org/10.5755/j01.eee.120.4.1456

Channel-Robust Classifiers

A key assumption underlying traditional supervised learning algorithms is that labeled examples used to train a classifier are drawn i.i.d. from the same distribution as test samples. This assumption is violated when classifying a test sample whose statistics differ from the training samples because the test signal is the output of a noisy linear time-invariant system, e.g., from channel propagation or filtering. We assume that the channel impulse response is unknown, but can be modeled as a random channel with finite first and second-order statistics that can be estimated from sample impulse responses. We present two kernels, the expected and projected RBF kernels, that account for the stochastic channel. Compared to the strategy of virtual examples, an SVM trained with the proposed kernels requires dramatically less training time, and may perform better in practice. We also extend the joint quadratic discriminant analysis (joint QDA) classifier, which also accounts for a stochastic channel, to a local version that reduces model bias. Results show the proposed methods achieve state-of-the-art performance and significantly faster training times.

AN EVALUATION OF MULTIPLE FEED-FORWARD NETWORKS ON GPUs

The Graphics Processing Unit (GPU) originally designed for rendering graphics and which is difficult to program for other tasks, has since evolved into a device suitable for general-purpose computations. As a result graphics hardware has become progressively more attractive yielding unprecedented performance at a relatively low cost. Thus, it is the ideal candidate to accelerate a wide variety of data parallel tasks in many fields such as in Machine Learning (ML). As problems become more and more demanding, parallel implementations of learning algorithms are crucial for a useful application. In particular, the implementation of Neural Networks (NNs) in GPUs can significantly reduce the long training times during the learning process. In this paper we present a GPU parallel implementation of the Back-Propagation (BP) and Multiple Back-Propagation (MBP) algorithms, and describe the GPU kernels needed for this task. The results obtained on well-known benchmarks show faster training times and improved performances as compared to the implementation in traditional hardware, due to maximized floating-point throughput and memory bandwidth. Moreover, a preliminary GPU based Autonomous Training System (ATS) is developed which aims at automatically finding high-quality NNs-based solutions for a given problem.

Classifying network protocols: a ‘two-way’ flow approach

The identification and classification of network traffic and protocols is a vital step in many quality of service and security systems. Traffic classification strategies must evolve, alongside the protocols utilising the Internet, to overcome the use of ephemeral or masquerading port numbers and transport layer encryption. This research expands the concept of using machine learning on the initial statistics of flow of packets to determine its underlying protocol. Recognising the need for efficient training/retraining of a classifier and the requirement for fast classification, the authors investigate a new application of k-means clustering referred to as 'two-way' classification. The 'two-way' classification uniquely analyses a bidirectional flow as two unidirectional flows and is shown, through experiments on real network traffic, to improve classification accuracy by as much as 18% when measured against similar proposals. It achieves this accuracy while generating fewer clusters, that is, fewer comparisons are needed to classify a flow. A 'two-way' classification offers a new way to improve accuracy and efficiency of machine learning statistical classifiers while still maintaining the fast training times associated with the k-means.

Condensed Vector Machines: Learning Fast Machine for Large Data

Scalability is one of the main challenges for kernel-based methods and support vector machines (SVMs). The quadratic demand in memory for storing kernel matrices makes it impossible for training on million-size data. Sophisticated decomposition algorithms have been proposed to efficiently train SVMs using only important examples, which ideally are the final support vectors (SVs). However, the ability of the decomposition method is limited to large-scale applications where the number of SVs is still too large for a computer's capacity. From another perspective, the large number of SVs slows down SVMs in the testing phase, making it impractical for many applications. In this paper, we introduce the integration of a vector combination scheme to simplify the SVM solution into an incremental working set selection for SVM training. The main objective of the integration is to maintain a minimal number of final SVs, bringing a minimum resource demand and faster training time. Consequently, the learning machines are more compact and run faster thanks to the small number of vectors included in their solution. Experimental results on large benchmark datasets shows that the proposed condensed SVMs achieve both training and testing efficiency while maintaining a generalization ability equivalent to that of normal SVMs.

OPELM and OPKNN in long-term prediction of time series using projected input data

Long-term time series prediction is a difficult task. This is due to accumulation of errors and inherent uncertainties of a long-term prediction, which leads to deteriorated estimates of the future instances. In order to make accurate predictions, this paper presents a methodology that uses input processing before building the model. Input processing is a necessary step due to the curse of dimensionality, where the aim is to reduce the number of input variables or features. In the paper, we consider the combination of the delta test and the genetic algorithm to obtain two aspects of reduction: scaling and projection. After input processing, two fast models are used to make the predictions: optimally pruned extreme learning machine and optimally pruned k-nearest neighbors. Both models have fast training times, which makes them suitable choice for direct strategy for long-term prediction. The methodology is tested on three different data sets: two time series competition data sets and one financial data set.

Features extraction for speech emotion

In this paper the speech emotion verification using two most popular methods in speech processing and analysis based on the Mel-Frequency Cepstral Coefficient (MFCC) and the Gaussian Mixture Model (GMM) were proposed and analyzed. In both cases, features for the speech emotion were extracted using the Short Time Fourier Transform (STFT) and Short Time Histogram (STH) for MFCC and GMM respectively. The performance of the speech emotion verification is measured based on three neural network (NN) and fuzzy neural network (FNN) architectures; namely: Multi Layer Perceptron (MLP), Adaptive Neuro Fuzzy Inference System (ANFIS) and Generic Self-organizing Fuzzy Neural Network (GenSoFNN). Results obtained from the experiments using real audio clips from movies and television sitcoms show the potential of using the proposed features extraction methods for real time application due to its reasonable accuracy and fast training time. This may lead us to the practical usage if the emotion verifier can be embedded in real time applications especially for personal digital assistance (PDA) or smart-phones.

Memory-Based Machine Translation and Language Modeling

Memory-Based Machine Translation and Language Modeling We describe a freely available open source memory-based machine translation system, mbmt. Its translation model is a fast approximate memory-based classifier, trained to map trigrams of source-language words onto trigrams of target-language words. In a second decoding step, the predicted trigrams are rearranged according to their overlap, and candidate output sequences are ranked according to a memory-based language model. We report on the scaling abilities of the memory-based approach, observing fast training and testing times, and linear scaling behavior in speed and memory costs. The system is released as an open source software package1, for which we provide a first reference guide.

An Improved Fast Training Algorithm for RBF Networks Using Symmetry-Based Fuzzy C-Means Clustering

In fuzzy C-means (FCM) clustering, each data point belongs to a cluster to a degree specified by a membership grade. FCM partitions a collection of vectors in c fuzzy groups and finds a cluster center in each group such that the dissimilarity measure is minimized. This paper presents a training algorithm for the radial basis function (RBF) network using symmetry-based Fuzzy C-means (SFCM) clustering method which is the modified version of FCM clustering method based on point symmetry distance measure. The training algorithm which uses SFCM clustering method to train the network has a number of advantages such as faster training time, more accurate predictions and reduced network architecture compared to the standard RBF networks. The proposed training algorithm has been implemented in the RBF networks created by the newrb function of MATLAB which uses gradient based iterative method as learning strategy, therefore the new network will undergo a hybrid learning process. The networks called Symmetry-based Fuzzy C-means Clustering–Radial Basis Function Network (SFCM/RBF) has been tested against the standard RBF network and the networks called standard Fuzzy C-means Clustering (FCM)-RBF network (FCM/RBF) in forecasting. The experimental models has been tested on three real world application problems, particularly in Air pollutant problem, Biochemical Oxygen Demand (BOD) problem, and Phytoplankton problem. Keywords: Fuzzy c-means clustering; SFCM; Radial basis function network; point symmetry distance; forecasting.

Development and Comparison of Backpropagation and Generalized Regression Neural Network Models to Predict Diurnal and Seasonal Gas and PM10 Concentrations and Emissions from Swine Buildings

The quantification of diurnal and seasonal gas (NH3, H2S, and CO2) and PM10 concentrations and emission rates (GPCER) from livestock production facilities is indispensable for the development of science-based setback determination methods and evaluation of improved downwind community air quality resulting from the implementation of gas pollution control. The purpose of this study was to employ backpropagation neural network (BPNN) and generalized regression neural network (GRNN) techniques to model GPCER generated and emitted from swine deep-pit finishing buildings as affected by time of day, season, ventilation rates, animal growth cycles, in-house manure storage levels, and weather conditions. The statistical results revealed that the BPNN and GRNN models were successfully developed to forecast hourly GPCER with very high coefficients of determination (R2) from 81.15% to 99.46% and very low values of systemic performance indexes. These good results indicated that the artificial neural network (ANN) technologies were capable of accurately modeling source air quality within and from the animal operations. It was also found that the process of constructing, training, and simulating the BPNN models was very complex. Some trial-and-error methods combined with a thorough understanding of theoretical backpropagation were required in order to obtain satisfying predictive results. The GRNN, based on nonlinear regression theory, can approximate any arbitrary function between input and output vectors and has a fast training time, great stability, and relatively easy network parameter settings during the training stage in comparison to the BPNN method. Thus, the GRNN was characterized as a preferred solution for its use in air quality modeling.

Real Time Machine Learning Based Car Detection in Images With Fast Training

Our primary interest is to build fast and reliable object recognizers in images based on small training sets. This is important in cases where the training set needs to be built mostly manually, as in the case that we studied, the recognition of the Honda Accord 2004 from rear views. We describe a novel variant of the AdaBoost based learning algorithm, which builds a strong classifier by incremental addition of weak classifiers (WCs) that minimize the combined error of the already selected WCs. Each WC is trained only once, and examples do not change their weights. We describe a set of appropriate feature types for the considered recognition problem, including a redness measure and dominant edge orientations. The existing edge orientation bin division was improved by shifting so that all horizontal (vertical, respectively) edges belong to the same bin. We propose to pre-eliminate features whose best threshold value is near the trivial position at the minimum or maximum of all threshold values. This is a novel method that has reduced the training set WC quantity to less than 10% of its original number, greatly speeding up training time, and showing no negative impact on the quality of the final classifier. We also modified the AdaBoost based learning machine. Our experiments indicate that the set of features used by Viola and Jones and others for face recognition was inefficient for our problem, recognizing cars accurately and in real time with fast training. Our training method has resulted in finding a very accurate classifier containing merely 30 WCs after about 1 h of training. Compared to existing literature, we have overall achieved the design of a real time object detection machine with the least number of examples, the least number of WCs, the fastest training time, and with competitive detection and false positive rates.

ADAPTIVE LVQ CLASSIFIER FOR INVARIANT FACE RECOGNITION

Visual communication plays an important role in human communication and interaction. In order to interact socially, we must be able to process faces in a variety of ways. This paper addresses the issue of recognition using a compact architecture of a learning vector quantization classifier that learns the correlation of patterns and identifies human faces. The suggested neural network structure is shown to exhibit robustness in achieving better classification results with both good generalization performance and a fast training time on a variety of test problems using standard and variable databases. Sources of variability include facial expression, gender, individual appearance, tilt, lighting conditions, and occluding objects (hair, spectacles, etc). Empirical results yield a peak accuracy rate of 100% (99.63% average) on the face recognition task for a random test set of 90 face images of 30 subjects from the Kuwait University databases on the network that is trained with another set of 60 faces of the same subjects. The computer execution time is about 0.04 s per face image on a Pentium II, 233 MHz, 32 MB, and Win98 PC.

A PARAMETER-BASED COMBINED CLASSIFIER FOR INVARIANT FACIAL EXPRESSION AND GENDER RECOGNITION

In this paper an learning vector quantization network architecture based on varying parameters and eliminating blind neurons28 is developed that learns the correlation of gender patterns and recognizes facial expressions of human faces. The network is developed to classify the 1 — neutral, 2 — smile or happiness, 3 — anger and 4 — scream or fear expressions. A peak accuracy rate of (a) 100% on the expression recognition task and (b) 100% on the gender recognition task for random training and test samples is achieved. The computer execution time for the recognition is about 0.004 seconds per face image for the latex expression classification task and 0.02 for the gender recognition task on an IBM PC. It is demonstrated that the proposed architecture is capable of achieving better recognition results with both good generalization performance and a fast training-time on a variety of test problems. The developed system showed potential for real life application domains.

An Adaptive Combined Classifier System for Invariant Face Recognition

Khuwaja, G. A., An Adaptive Combined Classifier System for Invariant Face Recognition, Digital Signal Processing 12 (2002) 21–46 In classification tasks it may be wise to combine observations from different sources. In this paper, to obtain classification systems with both good generalization performance and efficiency in space and time, a learning vector quantization learning method based on combinations of weak classifiers is proposed. The weak classifiers are generated using automatic elimination of redundant hidden layer neurons of the network on both the entire face images and the extracted features: forehead, right eye, left eye, nose, mouth, and chin. The neuron elimination is based on the killing of blind neurons, which are redundant. The classifiers are then combined through majority voting on the decisions available from input classifiers. It is demonstrated that the proposed system is capable of achieving better classification results with both good generalization performance and a fast training time on a variety of test problems using a large and variable database. The selection of stable and representative sets of features that efficiently discriminate between faces in a huge database is discussed.

Time series prediction using support vector machines, the orthogonal and the regularized orthogonal least-squares algorithms

Generalization properties of support vector machines, orthogonal least squares and zero-order regularized orthogonal least squares algorithms are studied using simulation. For high signal-to-noise ratios (40 dB), mixed results are obtained, but for a low signal-to-noise ratio, the prediction performance of support vector machines is better than the orthogonal least squares algorithm in the examples considered. However, the latter can usually give a parsimonious model with very fast training and testing time. Two new algorithms are therefore proposed that combine the orthogonal least squares algorithm with support vector machines to give a parsimonious model with good prediction accuracy in the low signal-to-noise ratio case.

A Batch Learning Vector Quantization Algorithm for Nearest Neighbour Classification

We introduce a batch learning algorithm to design the set of prototypes of 1 nearest-neighbour classifiers. Like Kohonen's LVQ algorithms, this procedure tends to perform vector quantization over a probability density function that has zero points at Bayes borders. Although it differs significantly from their online counterparts since: (1) its statistical goal is clearer and better defined; and (2) it converges superlinearly due to its use of the very fast Newton's optimization method. Experiments results using artificial data confirm faster training time and better classification performance than Kohonen's LVQ algorithms.

Combinations of weak classifiers

To obtain classification systems with both good generalization performance and efficiency in space and time, we propose a learning method based on combinations of weak classifiers, where weak classifiers are linear classifiers (perceptrons) which can do a little better than making random guesses. A randomized algorithm is proposed to find the weak classifiers. They are then combined through a majority vote. As demonstrated through systematic experiments, the method developed is able to obtain combinations of weak classifiers with good generalization performance and a fast training time on a variety of test problems and real applications. Theoretical analysis on one of the test problems investigated in our experiments provides insights on when and why the proposed method works. In particular, when the strength of weak classifiers is properly chosen, combinations of weak classifiers can achieve a good generalization performance with polynomial space- and time-complexity.