Proposed Training Algorithm Research Articles

Nonlinear Channel Equalization Using Wavelet Neural Network Trained Using PSO

Wireless communication channels are increasingly being pushed into the paradigm nonlinearity and inter symbol interference (ISI) because of demand for high speed data through portable and power efficient hand held devices. ISI and nonlinearity cause severe degradation in received signal resulting in poor quality of service. In this paper, channel equalizers are designed for mitigating channel nonlinearity and ISI. We propose an improved method of training wavelet neural network-based equalizer using particle swarm optimization (PSO). Our approach consists of optimizing the translation, dilation and other weights of hidden layer to achieve improvement in bit error rate (BER) performance. Superior performance of the proposed training algorithm is established by comparing the BER with established equalization schemes in literature.

Bi-Real Net: Binarizing Deep Network Towards Real-Network Performance

In this paper, we study 1-bit convolutional neural networks (CNNs), of which both the weights and activations are binary. While being efficient, the lacking of a representational capability and the training difficulty impede 1-bit CNNs from performing as well as real-valued networks. To this end, we propose Bi-Real net with a novel training algorithm to tackle these two challenges. To enhance the representational capability, we propagate the real-valued activations generated by each 1-bit convolution via a parameter-free shortcut. To address the training difficulty, we propose a training algorithm using a tighter approximation to the derivative of the sign function, a magnitude-aware binarization for weight updating, a better initialization method, and a two-step scheme for training a deep network. Experiments on ImageNet show that an 18-layer Bi-Real net with the proposed training algorithm achieves 56.4% top-1 classification accuracy, which is 10% higher than the state-of-the-arts (e.g., XNOR-Net), with a greater memory saving and a lower computational cost. Bi-Real net is also the first to scale up 1-bit CNNs to an ultra-deep network with 152 layers, and achieves 64.5% top-1 accuracy on ImageNet. A 50-layer Bi-Real net shows comparable performance to a real-valued network on the depth estimation task with merely a 0.3% accuracy gap.

An improved weight-constrained neural network training algorithm

In this work, we propose an improved weight-constrained neural network training algorithm, named iWCNN. The proposed algorithm exploits the numerical efficiency of the L-BFGS matrices together with a gradient-projection strategy for handling the bounds on the weights. Additionally, an attractive property of iWCNN is that it utilizes a new scaling factor for defining the initial Hessian approximation used in the L-BFGS formula. Since the L-BFGS Hessian approximation is defined utilizing a small number of correction vector pairs, our motivation is to further exploit them in order to increase the efficiency of the training algorithm and the convergence rate of the minimization process. The preliminary numerical experiments provide empirical evidence that the proposed training algorithm accelerates the training process.

Stuck-at-Fault Tolerant Schemes for Memristor Crossbar Array-Based Neural Networks

In this study, a circuit technique and training algorithm that minimizes the effect of stuck-at-faults (SAFs) within a memristor crossbar array of neural networks (NNs) are presented. To improve the network performance in the presence of SAFs, a conventional transimpedance amplifier, which is used for summing the currents that flow through the memristors, is modified to ensure that the amplifier output is within the appropriate operating range. Further improvement in the network performance is achieved by using the proposed training algorithm, which utilizes the locations and values of faulty memristors for network training. A feedforward NN employing 32×32 memristor crossbar arrays is implemented to verify the performance improvement in the NNs using the proposed circuit technique and training algorithm.

An adaptive nonmonotone active set – weight constrained – neural network training algorithm

In this work, a new direction for improving the classification accuracy of artificial neural networks is proposed by bounding the weights of the network, during the training process. Furthermore, a new adaptive nonmonotone active set – weight constrained – neural network training algorithm is proposed in order to demonstrate the efficacy and efficiency of our approach. The proposed training algorithm consists of two phases: a gradient projection phase which utilizes an adaptive nonmonotone line search and an unconstrained optimization phase which exploits the box structure of the bounds. Also, a set of switching criteria is defined for efficiently switching between the two phases. Our preliminary numerical experiments illustrate that the classification efficiency of the proposed algorithm outperforms classical neural network training algorithms, providing empirical evidence that it provides more stable, efficient and reliable learning.

Vocoder-free text-to-speech synthesis incorporating generative adversarial networks using low-/multi-frequency STFT amplitude spectra

This paper proposes novel training algorithms for vocoder-free text-to-speech (TTS) synthesis based on generative adversarial networks (GANs) that compensate for short-term Fourier transform (STFT) amplitude spectra in low/multi frequency resolution. Vocoder-free TTS using STFT amplitude spectra can avoid degradation of synthetic speech quality caused by the vocoder-based parameterization used in conventional TTS. Our previous work for the vocoder-based TTS proposed a method for incorporating the GAN-based distribution compensation into acoustic model training to improve synthetic speech quality. This paper extends the algorithm to the vocoder-free TTS and propose a GAN-based training algorithm using low-frequency-resolution amplitude spectra to overcome the difficulty in modeling complicated distribution of the high-dimensional spectra. In the proposed algorithm, amplitude spectra are transformed into low-frequency-resolution amplitude spectra by applying an average pooling function along with a frequency axis; then the GAN-based distribution compensation is performed in the low-frequency-resolution domain. Because the low-frequency-resolution amplitude spectra approximately emulate filter banks, the proposed algorithm is expected to improve synthetic speech quality by reducing differences in spectral envelopes of natural and synthetic speech. Furthermore, various frequency scales that are related to human speech perception (e.g., mel and inverse mel frequency scales) can be introduced to the proposed training algorithm by applying an frequency warping function to amplitude spectra. This paper also proposes a GAN-based training algorithm using multi-frequency-resolution amplitude spectra that uses both low- and original-frequency-resolution amplitude spectra to reduce the differences in not only spectral envelopes but also fine structures. Experimental results demonstrate that (1) GANs using low-frequency-resolution amplitude spectra improve speech quality and work robustly against the settings of the frequency resolution and hyperparameters, (2) in comparison among low-, original-, and multi-frequency-resolution amplitude spectra, the use of low-frequency-resolution ones work best improve the synthetic speech quality, and (3) the use of the inverse mel frequency scale for obtaining low-frequency-resolution amplitude spectra further improves synthetic speech quality.

A MODEL AND TRAINING ALGORITHM OF SMALL-SIZED OBJECT DETECTION SYSTEM FOR A COMPACT AERIAL DRONE

Context. Lightweight model and effective training algorithm of on-board object detection system for a compact drone are developed. The object of research is the process of small object detection on aerial images under computational resource constraint and uncertainty caused by small amount of labeled training data. The subject of the research are the model and training algorithm for detecting small objects on aerial imagery. Objective. Goal of the research is developing efficient model and training algorithm of object detection system for a compact aerial drone under conditions of restricted computing resources and the limited volume of the labeled training set. Methods. The four-stage training algorithm of the object detector is proposed. At the first stage, selecting the type of deep convolutional neural network and the number of low-level layers that is pre-trained on the ImageNet dataset for reusing takes place. The second stage involves unsupervised training of high-level convolutional sparse coding layers using modification of growing neural gas to automatically determine the required number of neurons and provide optimal distributions of the neurons over the data. Its application makes it possible to utilize the unlabeled training datasets for the adaptation of the high-level feature description to the domain application area. At the third stage, there are reduction of output feture map using principan component analysis and building of decision rules. In this case, output feature map is formed by concatenation of feature maps from different level of deep network using upscaling upper maps to uniform shape for each channel. This approach provides more contextual information for efficient recognition of small objects on aerial images. In order to build classifier of output feature map pixels is proposed to use boosted information-extreme learning. Besides that, the extreme learning machine is used to build of regression model for predict bounding box of detected object. Last stage involves fine-tuning of high-level layers of deep network using simulated annealing metaheuristic algorithm in order to approximating the global optimum of complex criterion of training efficiency of detection model. Results. The results obtained on open datasets testify to the suitability of the model and training algorithm for practical usage. The proposed training algorithm utilize 500 unlabeled and 200 labeled training samples to provide 96% correctly detection of objects on the images of the test dataset. Conclusions. Scientific novelty of the paper is a new model and training algorithm for object detection, which enable to achieve high confidence of recognition of small objects on aerial images under computational resource constraint and limited volume of the labeled training set. Practical significance of the paper results consists in the developed model and training algorithm made it possible to reduce requirements for size of labeled training set and computation resources of on-board detection system of aerial drone in training and inference modes.

New fast multi‐user beam training scheme based on compressed sensing theory for millimetre‐wave communication

Compensating for the considerable propagation loss, millimetre-wave communication systems adopt large antenna arrays and beamforming technologies to obtain high antenna gains. The beamforming training is rather time-consuming especially in multi-user cases. To reduce the training overhead, a new fast multi-user training scheme based on Compressed Sensing theory is proposed. This scheme includes a multi-user training algorithm and a corresponding semi-fixed semi-random codebook design strategy. The base station (BS) simultaneously broadcasts the training pilots to all user equipments (UEs). Then all UEs independently conduct the proposed training algorithm from channel responses. Finally, all UEs send back the training results to the BS. Compared to the exhaustive search algorithm's training overhead ξ = K r K t , the cost of the proposed scheme is only O ( ξ ) = log ⁡ ( K r K t ) , where K r and K t denote the number of beam patterns in the receiver and the transmitter. Simulation results show that the proposed scheme achieves excellent performance in high SNR (signal-to-noise ratio) cases with only 3 − 5 % performance difference from the theoretical bound. And in contrast with other Compressed Sensing schemes with random codebooks, the proposed scheme reduces the sensitivity to transmission SNR and improves approximately 3 dB performance in low SNR cases.

Distance Weighted K-Means Algorithm for Center Selection in Training Radial Basis Function Networks

The accuracies rates of the neural networks mainly depend on the selection of the correct data centers. The K-means algorithm is a widely used clustering algorithm in various disciplines for centers selection. However, the method is known for its sensitivity to initial centers selection. It suffers not only from a high dependency on the algorithm's initial centers selection but, also from data points. The performance of K-means has been enhanced from different perspectives, including centroid initialization problem over the years. Unfortunately, the solution does not provide a good trade-off between quality and efficiency of the centers produces by the algorithm. To solve this problem, a new method to find the initial centers and improve the sensitivity to the initial centers of K-means algorithm is proposed. This paper presented a training algorithm for the radial basis function network (RBFN) using improved K-means (KM) algorithm, which is the modified version of KM algorithm based on distance-weighted adjustment for each centers, known as distance-weighted K-means (DWKM) algorithm. The proposed training algorithm, which uses DWKM algorithm select centers for training RBFN obtained better accuracy in predictions and reduced network architecture compared to the standard RBFN. The proposed training algorithm was implemented in MATLAB environment; hence, the new network was undergoing a hybrid learning process. The network called DWKM-RBFN was tested against the standard RBFN in predictions. The experimental models were tested on four literatures nonlinear function and four real-world application problems, particularly in Air pollutant problem, Biochemical Oxygen Demand (BOD) problem, Phytoplankton problem, and forex pair EURUSD. The results are compared to proposed method for root mean square error (RMSE) in radial basis function network (RBFN). The proposed method yielded a promising result with an average improvement percentage more than 50 percent in RMSE.

An Improved Radial Basis Function Networks in Networks Weights Adjustment for Training Real-World Nonlinear Datasets

In neural networks, the accuracies of its networks are mainly relying on two important factors which are the centers and the networks weight. The gradient descent algorithm is a widely used weight adjustment algorithm in most of neural networks training algorithm. However, the method is known for its weakness for easily trap in local minima. It suffers from a random weight generated for the networks during initial stage of training at input layer to hidden layer networks. The performance of radial basis function networks (RBFN) has been improved from different perspectives, including centroid initialization problem to weight correction stage over the years. Unfortunately, the solution does not provide a good trade-off between quality and efficiency of the weight produces by the algorithm. To solve this problem, an improved gradient descent algorithm for finding initial weight and improve the overall networks weight is proposed. This improved version algorithm is incorporated into RBFN training algorithm for updating weight. Hence, this paper presented an improved RBFN in term of algorithm for improving the weight adjustment in RBFN during training process. The proposed training algorithm, which uses improved gradient descent algorithm for weight adjustment for training RBFN, obtained significant improvement in predictions compared to the standard RBFN. The proposed training algorithm was implemented in MATLAB environment. The proposed improved network called IRBFN was tested against the standard RBFN in predictions. The experimental models were tested on four literatures nonlinear function and four real-world application problems, particularly in Air pollutant problem, Biochemical Oxygen Demand (BOD) problem, Phytoplankton problem, and forex pair EURUSD. The results are compared to IRBFN for root mean square error (RMSE) values with standard RBFN. The IRBFN yielded a promising result with an average improvement percentage more than 40 percent in RMSE.

Convolutional neural network based on an extreme learning machine for image classification

Over the last decade, substantial advances have been made in various computer vision technologies and many of them are based on convolutional neural network (CNN) architecture. Typically, CNN is trained by a stochastic gradient descent algorithm using Back-Propagation (BP) but the training process is adversely affected by slow convergence and the need for extensive parameter tuning. In this paper, we propose a new CNN architecture and training algorithm based on an Extreme Learning Machine (ELM) to overcome these drawbacks. The proposed training algorithm is a layer-wise training method for CNN, and uses an alternating strategy of random convolutional filters and semi-supervised filters to combine the advantages of both approaches. On each semi-supervised layer, the CNN efficiently solves a convex optimization problem based on nonlinear random projection. It is faster and requires less human effort than BP-based training. We experimentally validated the proposed method using a well-known character and object recognition benchmark. In our experiment, the performance of our method is comparable to approaches based on deep features and has higher accuracy than other unsupervised feature-learning methods.

Weak-lensing shear measurement with machine learning

Cosmic shear, that is weak gravitational lensing by the large-scale matter structure of the Universe, is a primary cosmological probe for several present and upcoming surveys investigating dark matter and dark energy, such as Euclid or WFIRST. The probe requires an extremely accurate measurement of the shapes of millions of galaxies based on imaging data. Crucially, the shear measurement must address and compensate for a range of interwoven nuisance effects related to the instrument optics and detector, noise in the images, unknown galaxy morphologies, colors, blending of sources, and selection effects. This paper explores the use of supervised machine learning as a tool to solve this inverse problem. We present a simple architecture that learns to regress shear point estimates and weights via shallow artificial neural networks. The networks are trained on simulations of the forward observing process, and take combinations of moments of the galaxy images as inputs. A challenging peculiarity of the shear measurement task, in terms of machine learning applications, is the combination of the noisiness of the input features and the requirements on the statistical accuracy of the inverse regression. To address this issue, the proposed training algorithm minimizes bias over multiple realizations of individual source galaxies, reducing the sensitivity to properties of the overall sample of source galaxies. Importantly, an observational selection function of these source galaxies can be straightforwardly taken into account via the weights. We first introduce key aspects of our approach using toy-model simulations, and then demonstrate its potential on images mimicking Euclid data. Finally, we analyze images from the GREAT3 challenge, obtaining competitively low multiplicative and additive shear biases despite the use of a simple training set. We conclude that the further development of suited machine learning approaches is of high interest to meet the stringent requirements on the shear measurement in current and future surveys. We make a demonstration implementation of our technique publicly available.

Convolutional ensembles for Arabic Handwritten Character and Digit Recognition.

A learning algorithm is proposed for the task of Arabic Handwritten Character and Digit recognition. The architecture consists on an ensemble of different Convolutional Neural Networks. The proposed training algorithm uses a combination of adaptive gradient descent on the first epochs and regular stochastic gradient descent in the last epochs, to facilitate convergence. Different validation strategies are tested, namely Monte Carlo Cross-Validation and K-fold Cross Validation. Hyper-parameter tuning was done by using the MADbase digits dataset. State of the art validation and testing classification accuracies were achieved, with average values of 99.74% and 99.47% respectively. The same algorithm was then trained and tested with the AHCD character dataset, also yielding state of the art validation and testing classification accuracies: 98.60% and 98.42% respectively.

Technology Aware Training in Memristive Neuromorphic Systems for Nonideal Synaptic Crossbars

The advances in the field of machine learning using neuromorphic systems have paved the pathway for extensive research on possibilities of hardware implementations of neural networks. Various memristive technologies such as oxide-based devices, spintronics and phase change materials have been explored to implement the core functional units of neuromorphic systems, namely the synaptic network, and the neuronal functionality, in a fast and energy efficient manner. However, various non-idealities in the crossbar implementations of the synaptic arrays can significantly degrade performance of neural networks and hence, impose restrictions on feasible crossbar sizes. In this work, we build mathematical models of various non-idealities that occur in crossbar implementations such as source resistance, neuron resistance and chip-to-chip device variations and analyze their impact on the classification accuracy of a fully connected network (FCN) and convolutional neural network (CNN) trained with standard training algorithm. We show that a network trained under ideal conditions can suffer accuracy degradation as large as 59.84% for FCNs and 62.4% for CNNs when implemented on non-ideal crossbars for relevant non-ideality ranges. This severely constrains the sizes for crossbars. As a solution, we propose a technology aware training algorithm which incorporates the mathematical models of the non-idealities in the standard training algorithm. We demonstrate that our proposed methodology achieves significant recovery of testing accuracy within 1.9% of the ideal accuracy for FCNs and 1.5% for CNNs. We further show that our proposed training algorithm can potentially allow the use of significantly larger crossbar arrays of sizes 784$\times$500 for FCNs and 4096$\times$512 for CNNs with a minor or no trade-off in accuracy

End‐to‐end training algorithm for conceptor‐based neural networks

Since a conceptor, achieving direction-selective damping of high-dimensional network signals, usually takes the form of a projection matrix and is deduced analytically, a conceptor-based neural network is thought to be untrainable with backpropagation and gradient-descent algorithms from end to end. It limits the application of conceptors. To address this issue, an algorithm is proposed to train conceptor-based neural networks from end to end with gradient-descent algorithms. To the best of the authors’ knowledge, it is the first work of such an end-to-end training algorithm. To develop this algorithm, a softmax-like loss function involved with conceptors is constructed empirically. Based on this loss function, corresponding gradients are deduced by using backpropagation method so that it is possible to train conceptor neural networks from end to end with a gradient-descent algorithm. Several experiments are conducted to show the feasibility and effectiveness of the proposed training algorithm.

Smooth group L1/2 regularization for input layer of feedforward neural networks

Abstract A smooth group regularization method is proposed to identify and remove the redundant input nodes of feedforward neural networks, or equivalently the redundant dimensions of the input data of a given data set. This is achieved by introducing a smooth group L1/2 regularizer with respect to the input nodes into the error function to drive some weight vectors of the input nodes to zero. The main advantage of the method is that it can remove not only the redundant nodes, but also some redundant weights of the surviving nodes. As a comparison, the L1 regularization (Lasso) is mainly designed for removing the redundant weights, and it is not very good at removing the redundant nodes. And the group Lasso can remove the redundant nodes, but not any weight of the surviving nodes. Another advantage of the proposed method is that it uses a smooth function to replace the non-smooth absolute value function in the common L1/2 regularizer, and thus it reduces the oscillation caused by the non-smoothness and enables us to prove the convergence properties of the proposed training algorithm. Numerical simulations are performed to illustrate the efficiency of the algorithm.

Differential evolution training algorithm for dendrite morphological neural networks

Dendrite morphological neural networks are emerging as an attractive alternative for pattern classification, providing competitive results with other classification methods. A key problem in the design of these neural networks is the election of the number of their dendrites. Most training methods are heuristics that do not optimize the learning parameters. Therefore, we propose a new training algorithm for classification tasks based on an optimization approach: differential evolution. We show that the besought method increases classification performance and also optimizes the number of dendrites. For generating the initial population of hyper-boxes, we adopt two techniques: one based on the division of an initial hyper-box, and the other on an initial clustering using the so-called k-means++. Both alternatives were tested on four synthetic and 11 real databases as benchmarks overcoming the state-of-the-art morphological neuron training methods as well as the radial basis networks. The proposed training algorithm achieved a favorable average accuracy compared with the well-known multilayer perceptrons and support vector machines. In addition, a real-life problem was solved by this method to recognize geometric figures using a Nao robot.

Heart abnormality detection by using artificial neural network

The detection of heart abnormality activity of a patient is important when an abnormal symptom occurred. This project is aimed to detect the disease by using Hybrid Multilayer Perceptron (HMLP) network. Several data from the ECG signal is extracted to be set as input parameters. In order to get the best result, the proposed training algorithm of HMLP network is by using Modified Recursive Prediction Error (MRPE). The result obtained is then compared with other training algorithm method to determine the best performance. The HMLP network is expected to give better performance compared with other network classifiers.

Sequential Labeling With Structural SVM Under Nondecomposable Losses.

Sequential labeling addresses the classification of sequential data, which are widespread in fields as diverse as computer vision, finance, and genomics. The model traditionally used for sequential labeling is the hidden Markov model (HMM), where the sequence of class labels to be predicted is encoded as a Markov chain. In recent years, HMMs have benefited from minimum-loss training approaches, such as the structural support vector machine (SSVM), which, in many cases, has reported higher classification accuracy. However, the loss functions available for training are restricted to decomposable cases, such as the 0-1 loss and the Hamming loss. In many practical cases, other loss functions, such as those based on the $F_{1}$ measure, the precision/recall break-even point, and the average precision (AP), can describe desirable performance more effectively. For this reason, in this paper, we propose a training algorithm for SSVM that can minimize any loss based on the classification contingency table, and we present a training algorithm that minimizes an AP loss. Experimental results over a set of diverse and challenging data sets (TUM Kitchen, CMU Multimodal Activity, and Ozone Level Detection) show that the proposed training algorithms achieve significant improvements of the $F_{1}$ measure and AP compared with the conventional SSVM, and their performance is in line with or above that of other state-of-the-art sequential labeling approaches.

A fast training method for memristor crossbar based multi-layer neural networks

Memristor crossbar arrays carry out multiply–add operations in parallel in the analog domain which is the dominant operation in a neural network application. On-chip training of memristor neural network systems have the significant advantage of being able to get around device variability and faults. This paper presents a novel technique for on-chip training of multi-layer neural networks implemented using a single crossbar per layer and two memristors per synapse. Using two memristors per synapse provides double the synaptic weight precision when compared to a design that uses only one memristor per synapse. Proposed system utilizes a novel variant of the back-propagation (BP) algorithm to reduce both circuit area and training time. During training, all the memristors in a crossbar are updated in four steps in parallel. We evaluated the training of the proposed system with some nonlinearly separable datasets through detailed SPICE simulations which take crossbar wire resistance and sneak-paths into consideration. The proposed training algorithm trained the nonlinearly separable functions with a slight loss in accuracy compared to training with the traditional BP algorithm.