Training Time Of Neural Network Research Articles

An Efficient, Fast and Accurate Online Signature Verification Using Blended Feature Vector and Deep Learning

Online signature verification is one of the biometric authentication techniques. Online signature verification reduces the human error that may occur due to physical signature verification. Many researchers have provided a method of online signature verification still there is a scope for improvement. In this paper, a novel blended feature vector (BFV) is formed by combining two feature vectors. The first of these feature vectors is formed using a raw online signature database and the other feature vector is formed of the features extracted from the grayscale images obtained from the signature database. The algorithm makes use of the probability distribution of the features to provide a feature vector that results in an enhanced verification system. Two online signature databases, namely SVC2004 and ATVS-SSig are used. BFV is used to train the proposed mixed sequence deep neural network (MS-DNN). The use of a blended feature vector in the training of MS-DNN, instead of a whole online signature database, improves the training time of deep neural network to a great extent. This reduction in training time might be useful for new user registration. The performance parameters considered are validation efficiency and equal error rate. Results are compared with the existing state of technology and the comparison shows that the use of a blended feature vector as a classification vector improves the validation efficiency (99.5%) and also there is an improvement in the verification success rate in terms of equal error rate (EER 1.5%) as compared to the existing research.

Modeling nonlinear dynamic objects using pre-trained time delay neural networks

The work is devoted to resolving the contradiction between the accuracy of modelling nonlinear dynamics and the speed of model construction under conditions of limited computing resources. The purposeof the work is to reduce the time for building time delay neural networks while ensuring a given accuracy in the tasks of identifying nonlinear dynamic objects with continuous characteristics. This goal is achieved by developing a method for pre-training neural networks that reflect the basic characteristics of the subject area. The scientific novelty of the work is the development of a method for identifying nonlinear dynamic objectsin the form of time delay neural networks based on the use of a set of basic pre-trained neural networks that reflect the typical properties of the subject area. In contrast to the traditional approach to pre-training, the developed method allows building models of lower complexity.A formal criterion is proposed for determining the moment of termination of the neural network pre-training, the use of which allows avoiding retraining of the base model and ensuring a significant reduction in the model training time on the target data set. The practical utility of the work lies in the development of an algorithm for the method of pre-training time delay neural networks in the tasks of identifying nonlinear dynamic objects with continuous characteristics, which allows to significantlyreduce the training time of neural networks without losing model accuracy. The value of this study is todetermine the area of effective use of the proposed method, namely, when the general and target datasets do not have significant differences and the target dataset is of sufficient size to reflect the properties of the research object.

Deep Spiking Residual Shrinkage Network for Bearing Fault Diagnosis.

Bearing fault diagnosis of electrical equipment has been a popular research area in recent years because there are often some faults during continuous operation in production due to the harsh working environment. However, the traditional fault signal processing methods rely on highly expert experience, and some parameters are difficult to be optimized by machine-learning methods. Thus, the satisfactory recognition accuracy of fault diagnosis cannot be achieved in the above methods. In this article, a new model based on the spiking neural network (SNN) is proposed, which is called deep the spiking residual shrinkage network (DSRSN) for bearing fault diagnosis. In the model, attention mechanisms and soft thresholding are introduced to improve the recognition rate under a high-level noise background. The higher recognition accuracy is obtained in the proposed model which is tested on the fault signal dataset under different noise intensities. Meanwhile, the training time is about treble as fast as the training time of the artificial neural network, which is reflecting the high efficiency of SNN.

Circumventing Stragglers and Staleness in Distributed CNN using LSTM

INTRODUCTION: Using neural networks for these inherently distributed applications is challenging and time-consuming. There is a crucial need for a framework that supports a distributed deep neural network to yield accurate results at an accelerated time.
 METHODS: In the proposed framework, any experienced novice user can utilize and execute the neural network models in a distributed manner with the automated hyperparameter tuning feature. In addition, the proposed framework is provided in AWS Sage maker for scaling the distribution and achieving exascale FLOPS. We benchmarked the framework performance by applying it to a medical dataset. 
 RESULTS: The maximum performance is achieved with a speedup of 6.59 in 5 nodes. The model encourages expert/ novice neural network users to apply neural network models in the distributed platform and get enhanced results with accelerated training time. There has been a lot of research on how to improve the training time of Convolutional Neural Networks (CNNs) using distributed models, with a particular emphasis on automating the hyperparameter tweaking process. The study shows that training times may be decreased across the board by not just manually tweaking hyperparameters, but also by using L2 regularization, a dropout layer, and ConvLSTM for automatic hyperparameter modification.
 CONCLUSION: The proposed method improved the training speed for model-parallel setups by 1.4% and increased the speed for parallel data by 2.206%. Data-parallel execution achieved a high accuracy of 93.3825%, whereas model-parallel execution achieved a top accuracy of 89.59%.

Hybrid optical convolutional neural network with convolution kernels trained in the spatial domain

Although many hybrid convolutional neural networks using optical convolution processors have been demonstrated for high energy efficiency and fast image processing speed, the need for accurate modeling of the optical convolution hardware and large Fourier kernel sizes result in a long and energy-intensive training process. Here, we demonstrate a hybrid convolutional neural network based on an optimized optical convolution processor—the system uses kernels trained in the spatial domain and compensates the optical path mismatch that arises from the reflection of digital micromirror devices for convolution computation. The spatial-domain convolution kernels are Fourier transformed and then binarized before being programmed into a digital micromirror device in the optical convolution processor. This method minimizes overhead from the optical convolution process in the training of the hybrid convolutional neural network, and it can improve the energy efficiency and reduce the training time of the hybrid optical convolutional neural network. For convolution computation using digital micromirror devices with grayscale images, bit-plane-wise convolution is also presented. We tested the hybrid neural network on MNIST, Fashion-MNIST, and CIFAR-10 color datasets and obtained classification accuracies of 98.7%, 85.8%, and 57.8%, respectively.

Event-based diffractive neural network chip for dynamic action recognition

Dynamic action recognition has promising applications in human–computer interaction, information encryption, and high-speed image processing. However, it is challenging for a high-dynamic scene due to the huge amount of data and the computation costs. Here, we have proposed an event-based diffractive neural network (EBDNN) chip to recognize human dynamic actions. An event camera with high dynamic range and high temporal resolution is used for dynamic action capture. Based on the approach of compressing dimensions, a series of static images can be obtained by superimposing the event frames of the intervals, which serve as the train dataset of a diffractive neural network with photon wave properties. Compared to a frame camera with the same temporal resolution, the required data volume is only 0.0048% of that, which greatly shortens the training time of the diffractive neural network on the computer and saves computing resources. Notably, the diffractive neural network was trained to perform image classification tasks at the speed of light with almost no energy consumption, and the diffractive devices, which includes 400 × 400 neurons, is printed based on the two-photon polymerization technology. In addition, the experimental accuracy was as high as 99.17%. Our work opens up a new direction for photoelectrically integrated intelligent device, paving the way for the development of real-time monitoring systems and other machine vision tasks, such as autonomous driving and video surveillance.

A competitive learning scheme for deep neural network pattern classifier training

To reduce the computational complexity of training a deep neural network architecture using large data sets of 3D scenes, a competitive learning scheme was devised. The proposed algorithm pits a neural network learning algorithm, in this case the standard Adam optimiser, against an evolutionary algorithm that is used to select the most difficult training examples. The overall scheme is similar to a predator–prey system, where the predator (the neural network) strives to optimise its ability to capture (identify) the prey (the training patterns), and the evolutionary procedure selects the prey that so far evaded capture. As a consequence of the evolutionary process, the neural network is presented only a fraction of the training examples, and the computational complexity of the learning procedure is reduced. Experimental evidence showed that the proposed scheme allows reducing the deep neural network training time on different model sets, sometimes significantly, without affecting the recognition accuracy. The proposed predator–prey scheme is fairly independent of the ANN type and training algorithm employed, and has the potential to be beneficial to a wide range of deep learning applications, where practical implementations are often hindered by the time complexity of the training process.

PHY: A performance-driven hybrid communication compression method for distributed training

Distributed training is needed to shorten the training time of deep neural networks. However, the communication overhead often hurts performance efficiency, especially in a distributed computing environment with limited network bandwidth. Hence, gradient compression techniques have been proposed to reduce communication time. But, compression also has the risk of causing lower model accuracy and longer training time due to compression loss and compression time. As a result, compression may not consistently achieve desired results, and there are limited discussions on when and which compression should be used. To address this problem, we propose a performance-driven hybrid compression solution. We make three main contributions. (1) We describe a hybrid compression strategy that chooses the compression method for individual model gradients. (2) We build an offline performance estimator and an online loss monitor to ensure the compression decision can minimize training time without sacrificing mode accuracy. (3) Our implementation can be imported to existing deep learning frameworks and applicable to a wide range of compression methods. Up to 3.6x training performance speedup was observed compared to other state-of-the-art methods.

A Self-Supervised Tree-Structured Framework for Fine-Grained Classification

In computer vision, fine-grained classification has become an important issue in recognizing objects with slight visual differences. Usually, it is challenging to generate good performance when solving fine-grained classification problems using traditional convolutional neural networks. To improve the accuracy and training time of convolutional neural networks in solving fine-grained classification problems, this paper proposes a tree-structured framework by eliminating the effect of differences between clusters. The contributions of the proposed method include the following three aspects: (1) a self-supervised method that automatically creates a classification tree, eliminating the need for manual labeling; (2) a machine-learning matcher which determines the cluster to which an item belongs, minimizing the impact of inter-cluster variations on classification; and (3) a pruning criterion which filters the tree-structured classifier, retaining only the models with superior classification performance. The experimental evaluation of the proposed tree-structured framework demonstrates its effectiveness in reducing training time and improving the accuracy of fine-grained classification across various datasets in comparison with conventional convolutional neural network models. Specifically, for the CUB 200 2011, FGVC aircraft, and Stanford car datasets, the proposed method achieves a reduction in training time of 32.91%, 35.87%, and 14.48%, and improves the accuracy of fine-grained classification by 1.17%, 2.01%, and 0.59%, respectively.

Selecting and Composing Learning Rate Policies for Deep Neural Networks

The choice of learning rate (LR) functions and policies has evolved from a simple fixed LR to the decaying LR and the cyclic LR, aiming to improve the accuracy and reduce the training time of Deep Neural Networks (DNNs). This article presents a systematic approach to selecting and composing an LR policy for effective DNN training to meet desired target accuracy and reduce training time within the pre-defined training iterations. It makes three original contributions. First, we develop an LR tuning mechanism for auto-verification of a given LR policy with respect to the desired accuracy goal under the pre-defined training time constraint. Second, we develop an LR policy recommendation system (LRBench) to select and compose good LR policies from the same and/or different LR functions through dynamic tuning, and avoid bad choices, for a given learning task, DNN model, and dataset. Third, we extend LRBench by supporting different DNN optimizers and show the significant mutual impact of different LR policies and different optimizers. Evaluated using popular benchmark datasets and different DNN models (LeNet, CNN3, ResNet), we show that our approach can effectively deliver high DNN test accuracy, outperform the existing recommended default LR policies, and reduce the DNN training time by 1.6-6.7× to meet a targeted model accuracy.

Predicting Model Training Time to Optimize Distributed Machine Learning Applications

Despite major advances in recent years, the field of Machine Learning continues to face research and technical challenges. Mostly, these stem from big data and streaming data, which require models to be frequently updated or re-trained, at the expense of significant computational resources. One solution is the use of distributed learning algorithms, which can learn in a distributed manner, from distributed datasets. In this paper, we describe CEDEs—a distributed learning system in which models are heterogeneous distributed Ensembles, i.e., complex models constituted by different base models, trained with different and distributed subsets of data. Specifically, we address the issue of predicting the training time of a given model, given its characteristics and the characteristics of the data. Given that the creation of an Ensemble may imply the training of hundreds of base models, information about the predicted duration of each of these individual tasks is paramount for an efficient management of the cluster’s computational resources and for minimizing makespan, i.e., the time it takes to train the whole Ensemble. Results show that the proposed approach is able to predict the training time of Decision Trees with an average error of 0.103 s, and the training time of Neural Networks with an average error of 21.263 s. We also show how results depend significantly on the hyperparameters of the model and on the characteristics of the input data.

Transfer Learning for the Behavior Prediction of Microwave Structures

Microwave structure behavior prediction is an important research topic in radio frequency (RF) design. In recent years, deep-learning-based techniques have been widely implemented to study microwaves, and they are envisaged to revolutionize this arduous and time-consuming work. However, empirical data collection and neural network training are two significant challenges of applying deep learning techniques to practical RF modeling and design problems. To this end, this letter investigates a transfer-learning-based approach to improve the accuracy and efficiency of predicting microwave structure behaviors. Through experimental comparisons, we validate that the proposed approach can reduce the amount of data required for training while shortening the neural network training time for the behavior prediction of microwave structures.

Data augmentation to improve performance of neural networks for failure management in optical networks

Despite the increased exploration of machine learning (ML) techniques for the realization of autonomous optical networks, less attention has been paid to data quality, which is critical for ML performance. Failure management in optical networks using ML is constrained by the fact that some failures may occur more frequently than others, resulting in highly imbalanced datasets for the training of ML models. To address this limitation, a variational-autoencoder-based data augmentation technique is investigated in this paper, which can be used during data preprocessing to improve data quality. The synthetic data generated by the variational autoencoder are utilized to reduce imbalance in an experimental dataset used for training of neural networks (NNs) for failure management in optical networks. First, it is shown that, with a modified training dataset, the training time of NNs can be reduced. Reductions of up to 37.1% and 60.6% are achieved for failure detection and cause identification, respectively. Second, it is shown that improvement in the quality of the training dataset can reduce the computational complexity of NNs during the inference phase. As determined analytically, almost 68% reduction in computational complexity is achieved for the NN used for failure cause identification. Finally, data augmentation is shown to achieve improvement in classification accuracy. This work demonstrates improvement of up to 7.32%.

Two-level group convolution

Group convolution has been widely used in order to reduce the computation time of convolution, which takes most of the training time of convolutional neural networks. However, it is well known that a large number of groups significantly reduce the performance of group convolution. In this paper, we propose a new convolution methodology called “two-level” group convolution that is robust with respect to the increase of the number of groups and suitable for multi-GPU parallel computation. We first observe that the group convolution can be interpreted as a one-level block Jacobi approximation of the standard convolution, which is a popular notion in the field of numerical analysis. In numerical analysis, there have been numerous studies on the two-level method that introduces an intergroup structure that resolves the performance degradation issue without disturbing parallel computation. Motivated by these, we introduce a coarse-level structure which promotes intergroup communication without being a bottleneck in the group convolution. We show that all the additional work induced by the coarse-level structure can be efficiently processed in a distributed memory system. Numerical results that verify the robustness of the proposed method with respect to the number of groups are presented. Moreover, we compare the proposed method to various approaches for group convolution in order to highlight the superiority of the proposed method in terms of execution time, memory efficiency, and performance.

A Deep Convolutional Neural Network Stacked Ensemble for Malware Threat Classification in Internet of Things

Malicious attacks to software applications are on the rise as more people use Internet of things (IoT) devices and high-speed internet. When a software system crash happens caused by malicious action, a malware imaging method can examine the application. In this study, we present a novel malware classification method that captures suspected operations in a variety of discrete size image features, allowing us to identify such IoT device malware families. To decrease deep neural network training time, essential local and global image features are selected using a combined local and global feature descriptor (LBP-GLCM). The classification performance of the proposed deep learning model is improved by combining the predictions of weak learners (CNNs) and using them as knowledge input to a multi-layer perceptron meta learner. This is a neural network ensemble with stacked generalization that is used to improve network generalization ability. The public dataset used for performance evaluation contains 5472 samples from 11 different malware families. In order to compare the proposed methodology to current malware detection systems, we developed a baseline experiment. The proposed approach improved malware classification results to 98.5% accuracy and 98.4% accuracy when using [Formula: see text] and [Formula: see text] image sizes, respectively. Overall, the results showed that the stacked generalization ensemble with multi-step extracting features is a more effective method for classification performance and response time.

Application of Improved VMD-LSTM Model in Sports Artificial Intelligence.

In recent years, with the rapid development of a new generation of artificial intelligence technology, how to deeply apply artificial intelligence technology to physical education and break through the limitations of time-space scenarios and knowledge transfer methods in traditional models has become a key issue in intelligent physical education in the era of artificial intelligence. In order to realize the online monitoring of wearable devices with artificial intelligence in sports and overcome the problem of low recognition accuracy of electrocardiogram, blood oxygen, and respiratory signals in many cases, this paper proposes a combination of variational modal decomposition based on the maximum envelope kurtosis method. Long-short-term neural network (VMD-LSTM) monitoring method for wearable sports equipment. Through experimental analysis and verification, the current signal of the VMD model shows a trend of fluctuating from large to stable and then to large with motion, while the training accuracy of LSTM after the 150th iteration is 94.09%, which shows that the coupling model VMD LSTM can better predict the direction of sports artificial intelligence. In addition, although the training time of the BP neural network is shorter than that of the LSTM model, there is a large gap between the recognition effect and the LSTM, and there are also large differences between different neural network structures. This shows that the VMD-LSTM model has broad application prospects in such models.

Economic Forecasting Model Based on Chaos Simulated Annealing Neural Network

In view of the prototype of the traditional chaotic network model, a neural network model with continuously updated chaotic noise is innovatively improved. This model has the advantages of two network models. On this basis, a logic graph is proposed, which can replace the chaotic noise generated by the running process of the model function. Models with these advantages can innovatively solve problems such as constrained optimization with dimensions larger than three, high discreteness, and weak linear relationship between convex and concave. Simulation results can be confirmed that the algorithm of this model is very close to the predicted value. This neural network model is an improved model with strong applicability and can be applied to optimization problems of economic systems or other industrial systems. In order to effectively alleviate the predictive control problem of nonlinear research objects, we propose a control method based on chaotic neural network in this study. Taking the economic model as the nonlinear object, establishing the basic structure of the chaotic neural network model, reconstructing the time and space structure, and obtaining the optimal solution of time continuation and embedding dimension through information entropy and pseudonearest neighbor, then the chaotic properties of nonlinear objects and the topology of chaotic neural networks are determined. In the simulation, test samples and experimental samples are established, and the predicted and true values are compared. The prediction results of the model established by this method in this study prove the effectiveness of the method. The training time of the chaotic neural network will not fall into a local minimum, thereby reducing the training time and ensuring the accuracy of the prediction.

Multi-Timescale Wavelet Scattering With Genetic Algorithm Feature Selection for Acoustic Scene Classification

In this paper, we apply a genetic algorithm (GA) for feature selection, wrapper approach, on wavelet scattering (WS) second-order coefficients to reduce the large frequency dimension (>500). The evaluation demonstrates the capability of GA to reduce the dimension space by approximately 30% while ensuring a minimum performance drop. Furthermore, the reduced WS directly impacts the training time of the convolutional neural network, by reducing the computational time by 20% to 32%. The paper extends its scopes to explore GA for feature selection on multiple timescales of WS: 46ms, 92ms, 185ms, and 371ms. Incorporating multiple timescales has improved classification performance (by ~ around 2.5%) as an acoustic representation usually contains information at different time scales. However, it can increase computational cost due to the larger frequency dimension of 1851. With the application of GA for feature selection, the frequency dimension is reduced by 50%, saving around 40% computational time, thus increasing the classification performance by 3% compared to a vanilla WS. Lastly, the entire implementations are evaluated using the Detection and Classification of Acoustic Scenes and Events (DCASE) 2020 dataset, and the proposed multiple timescales model achieves 73.32% of classification accuracy.

A neural network regularization method to address variance inflation in autoencoders

There exist various machine learning techniques which can be used to reduce the dimensionality of original data while minimizing the information loss. Principal component analysis (PCA) is one of the most well known such techniques, which transforms the original correlated variables into uncorrelated variables called principal components. Although PCA is known to preserve the total variance of the original data during the transformation, there are some cases with a potential of variance inflation, where the total variance of principal components becomes much larger than that of original variables. It is important to prevent variance inflation, as it can negatively affect the performance of other application systems (e.g. process monitoring systems) which are designed on the basis of principal component with inflated variances. Variance inflation also has a high potential to occur during the training of autoencoder, a special type of neural network performing nonlinear version of PCA. Although there are several neural network regularization methods available to alleviate the problem of variance inflation, none of them is tailored to do such task. To this end, in this work, an alternative neural network regularization method is proposed, which can strongly regulate the total variance in the feature space. Using the Tennessee Eastman process as an illustrative example, the proposed regularization method is compared with the existing ones in terms of neural network overfitting, variance inflation, and training time.

Improved iSAM Based on Flexible Re-Linearization Threshold and Error Learning Model for AUV in Large Scale Areas

This paper proposed an improved incremental smoothing and mapping (iSAM) which combines flexible re-linearization threshold with error learning model to improve the efficiency and accuracy of navigation for autonomous underwater vehicle (AUV). The flexible threshold of the proposed method, which can avoid periodic re-linearization of iSAM, is controlled by adaptive threshold. Through the use of flexible re-linearization, the proposed method can reduce the running time and carry out re-linearization timely. Simultaneously, for the first time, the proposed method takes advantages of the fast training time of hidden-layer neural networks and Gaussian Process Regression which is more suitable for non-linearity to get error learning model for iSAM. The proposed method can achieve better accuracy with only a rough model and can be easily transplanted to other systems without cumbersome calculations of the precise model. Our algorithm has been demonstrated by a range of simulated and real datasets, especially in practical application it outperforms current available iSAM algorithms in efficiency and accuracy. The RMSE of proposed method increases by 21.7% and efficiency also has been improved by 56% than iSAM2 in practical application for AUV.