Greedy Training Research Articles

From communities to interpretable network and word embedding: an unified approach

Abstract Modelling information from complex systems such as humans social interaction or words co-occurrences in our languages can help to understand how these systems are organized and function. Such systems can be modelled by networks, and network theory provides a useful set of methods to analyze them. Among these methods, graph embedding is a powerful tool to summarize the interactions and topology of a network in a vectorized feature space. When used in input of machine learning algorithms, embedding vectors help with common graph problems such as link prediction, graph matching, etc In Natural Language Processing (NLP), such a vectorization process is also employed. Word embedding has the goal of representing the sense of words, extracting it from large text corpora. Despite differences in the structure of information in input of embedding algorithms, many graph embedding approaches are adapted and inspired from methods in NLP. Limits of these methods are observed in both domains. Most of these methods require long and resource greedy training. Another downside to most methods is that they are black-box, from which understanding how the information is structured is rather complex. Interpretability of a model allows understanding how the vector space is structured without the need for external information, and thus can be audited more easily. With both these limitations in mind, we propose a novel framework to efficiently embed network vertices in an interpretable vector space. Our Lower Dimension Bipartite Framework (LDBGF) leverages the bipartite projection of a network using cliques to reduce dimensionality. Along with LDBGF, we introduce two implementations of this framework that rely on communities instead of cliques: SINr-NR and SINr-MF. We show that SINr-MF can perform well on classical graphs and SINr-NR can produce high-quality graph and word embeddings that are interpretable and stable across runs.

Damage detection of jacket platforms through improved stacked autoencoder and softmax classifier

The jacket platform is a common structural form for offshore operations, and its health status determines the safety of the marine development process. In addressing the issue of damage detection in jacket platforms, a damage detection and prediction model based on stacked autoencoder (SAE) and softmax classifier is proposed. The model first constructs a SAE model using a layer-wise greedy training strategy to extract features from input signals. Subsequently, the extracted features are input into the softmax classifier for detecting damage types in the jacket platform. The model also utilizes a risk coefficient to forecast potential faults on the jacket platform. Experimental results on the jacket platform model demonstrate that the proposed method achieves an overall accuracy of 96.7% in damage detection in both isolated noise and marine environments, which is an improvement of approximately 5% compared to the autoencoder (AE) model. Additionally, in the damage prediction experiments, the model achieves a precision of 98.7% and a recall of 77.5% for potential damage.

Decouple Graph Neural Networks: Train Multiple Simple GNNs Simultaneously Instead of One.

Graph neural networks (GNN) suffer from severe inefficiency due to the exponential growth of node dependency with the increase of layers. It extremely limits the application of stochastic optimization algorithms so that the training of GNN is usually time-consuming. To address this problem, we propose to decouple a multi-layer GNN as multiple simple modules for more efficient training, which is comprised of classical forward training (FT) and designed backward training (BT). Under the proposed framework, each module can be trained efficiently in FT by stochastic algorithms without distortion of graph information owing to its simplicity. To avoid the only unidirectional information delivery of FT and sufficiently train shallow modules with the deeper ones, we develop a backward training mechanism that makes the former modules perceive the latter modules, inspired by the classical backward propagation algorithm. The backward training introduces the reversed information delivery into the decoupled modules as well as the forward information delivery. To investigate how the decoupling and greedy training affect the representational capacity, we theoretically prove that the error produced by linear modules will not accumulate on unsupervised tasks in most cases. The theoretical and experimental results show that the proposed framework is highly efficient with reasonable performance, which may deserve more investigation.

Accelerated solutions of convection‐dominated partial differential equations using implicit feature tracking and empirical quadrature

SummaryThis work introduces an empirical quadrature‐based hyperreduction procedure and greedy training algorithm to effectively reduce the computational cost of solving convection‐dominated problems with limited training. The proposed approach circumvents the slowly decaying ‐width limitation of linear model reduction techniques applied to convection‐dominated problems by using a nonlinear approximation manifold systematically defined by composing a low‐dimensional affine space with bijections of the underlying domain. The reduced‐order model is defined as the solution of a residual minimization problem over the nonlinear manifold. An online‐efficient method is obtained by using empirical quadrature to approximate the optimality system such that it can be solved with mesh‐independent operations. The proposed reduced‐order model is trained using a greedy procedure to systematically sample the parameter domain. The effectiveness of the proposed approach is demonstrated on two shock‐dominated computational fluid dynamics benchmarks.

Discrete-time nonlinear feedback linearization via physics-informed machine learning

We present a physics-informed machine learning (PIML) scheme for the feedback linearization of nonlinear discrete-time dynamical systems. The PIML finds the nonlinear transformation law, thus ensuring stability via pole placement, in one step. In order to facilitate convergence in the presence of steep gradients in the nonlinear transformation law, we address a greedy training procedure. We assess the performance of the proposed PIML approach via a benchmark nonlinear discrete map for which the feedback linearization transformation law can be derived analytically; the example is characterized by steep gradients, due to the presence of singularities, in the domain of interest. We show that the proposed PIML outperforms, in terms of numerical approximation accuracy, the traditional numerical implementation, which involves the construction –and the solution in terms of the coefficients of a power-series expansion–of a system of homological equations as well as the implementation of the PIML in the entire domain, thus highlighting the importance of continuation techniques in the training procedure of PIML schemes.

A novel transformer-based ECG dimensionality reduction stacked auto-encoders for arrhythmia beat detection.

Electrocardiogram (ECG) is a powerful tool for studying cardiac activity and diagnosing various cardiovascular diseases, including arrhythmia. While machine learning and deep learning algorithms have been applied to ECG interpretation, there is still room for improvement. For instance, the commonly used Recurrent Neural Networks (RNNs), reply on its previous state to update and is therefore ineffective for parallel computing. RNN also struggles to efficiently address the issue of long-distancereliance. To reduce computational complexity by dimensionality reduction of ECG signals we constructed a Stacked Auto-encoders model using Transformer for ECG-based arrhythmia detection. And overcome the challenges of long-term dependencies and limited parallelizability in traditional RNNs when applied to ECG signalprocessing. In this paper, a Transformer-Based ECG Dimensionality Reduction Stacked Auto-encoders model is proposed for ECG-based arrhythmia detection. The transformer is used to encode ECG signals into a feature matrix, which is then dimensionally reduced using unsupervised greedy training through the four linear layers. This resulted in a low-dimensional representation of ECG features, which are subsequently classified using support vector machines (SVM) to minimizeoverfitting. The proposed method is benchmarked on the MIT-BIH Arrhythmia database. In the 10-fold cross validation of beat-based arrhythmia detection, the average accuracy, sensitivity, specificity and F1 score of the proposed method are 99.83%, 98.84%, 99.84% and 99.13%, respectively, for the record-based arrhythmia detection which refers to the approach where the training and testing sets use ECG data from independent recorded patients are 88.10%, 49.79%, 91.56% and 39.95%, respectively. Compared to other existing ECG-based arrhythmia detection methods, our proposed approach exhibits improved detection accuracy and stronger generalization for arrhythmia beats. Additionally, the use of the record-based data division method makes our approach more suitable for clinicalpractice.

Greedy training algorithms for neural networks and applications to PDEs

Recently, neural networks have been widely applied for solving partial differential equations (PDEs). Although such methods have been proven remarkably successful on practical engineering problems, they have not been shown, theoretically or empirically, to converge to the underlying PDE solution with arbitrarily high accuracy. The primary difficulty lies in solving the highly non-convex optimization problems resulting from the neural network discretization, which are difficult to treat both theoretically and practically. It is our goal in this work to take a step toward remedying this. For this purpose, we develop a novel greedy training algorithm for shallow neural networks. Our method is applicable to both the variational formulation of the PDE and also to the residual minimization formulation pioneered by physics informed neural networks (PINNs). We analyze the method and obtain a priori error bounds when solving PDEs from the function class defined by shallow networks, which rigorously establishes the convergence of the method as the network size increases. Finally, we test the algorithm on several benchmark examples, including high dimensional PDEs, to confirm the theoretical convergence rate. Although the method is expensive relative to traditional approaches such as finite element methods, we view this work as a proof of concept for neural network-based methods, which shows that numerical methods based upon neural networks can be shown to rigorously converge.

Anomaly Detection Algorithm Based on Broad Learning System and Support Vector Domain Description

Deep neural network-based autoencoders can effectively extract high-level abstract features with outstanding generalization performance but suffer from sparsity of extracted features, insufficient robustness, greedy training of each layer, and a lack of global optimization. In this study, the broad learning system (BLS) is improved to obtain a new model for data reconstruction. Support Vector Domain Description (SVDD) is one of the best-known one-class-classification methods used to solve problems where the proportion of sample categories of data is extremely unbalanced. The SVDD is sensitive to penalty parameters C, which represents the trade-off between sphere volume and the number of target data outside the sphere. The training process only considers normal samples, which leads to a low recall rate and weak generalization performance. To address these issues, we propose a BLS-based weighted SVDD algorithm (BLSW_SVDD), which introduces reconstruction error weights and a small number of anomalous samples when training the SVDD model, thus improving the robustness of the model. To evaluate the performance of BLSW_SVDD model, comparison experiments were conducted on the UCI dataset, and the experimental results showed that in terms of accuracy and F1 values, the algorithm has better performance advantages than the traditional and improved SVDD algorithms.

Defect Prediction Technology of Aerospace Software Based on Deep Neural Network and Process Measurement

In order to ensure high reliability, the efficiency of traditional aerospace software testing is often low. With the rapid development of machine learning, its powerful data feature extraction ability has great potential in improving the efficiency of aerospace software testing. Therefore, this paper proposed a software defect prediction method based on deep neural network and process measurement. Based on the NASA data set and combined with the software process data, the software defect measurement set is constructed. 35 measurement elements are used as the original input, and multiple single-layer automatic coding networks are superimposed to form the deep neural network model of software defect. The model is finally trained by the layer-by-layer greedy training method to realize software defect prediction. Experimental verification shows that the prediction method has a good prediction effect on aerospace software defects, and the accuracy rate reached 90%, which can greatly improve the efficiency and effect of aerospace software testing.

Learning to Decode Protograph LDPC Codes

The recent development of deep learning methods provides a new approach to optimize the belief propagation (BP) decoding of linear codes. However, the limitation of existing works is that the scale of neural networks increases rapidly with the codelength, thus they can only support short to moderate codelengths. From the point view of practicality, we propose a high-performance neural min-sum (MS) decoding method that makes full use of the lifting structure of protograph low-density parity-check (LDPC) codes. By this means, the size of the parameter array of each layer in the neural decoder only equals the number of edge-types for arbitrary codelengths. In particular, for protograph LDPC codes, the proposed neural MS decoder is constructed in a special way such that identical parameters are shared by a bundle of edges derived from the same edge-type. To reduce the complexity and overcome the vanishing gradient problem in training the proposed neural MS decoder, an iteration-by-iteration (i.e., layer-by-layer in neural networks) greedy training method is proposed. With this, the proposed neural MS decoder tends to be optimized with faster convergence, which is aligned with the early termination mechanism widely used in practice. To further enhance the generalization ability of the proposed neural MS decoder, a codelength/rate compatible training method is proposed, which randomly selects samples from a set of codes lifted from the same base code. As a theoretical performance evaluation tool, a trajectory-based extrinsic information transfer (T-EXIT) chart is developed for various decoders. Both T-EXIT and simulation results show that the optimized MS decoding can provide faster convergence and up to 1dB gain compared with the plain MS decoding and its variants with only slightly increased complexity. In addition, it can even outperform the sum-product algorithm for some short codes.

Research on Movie Recommendation Algorithm Based on Stack De-noising Auto-encoder

Focused on the problems that the randomness of the noise-adding operation in the de-noising auto-encoder (DAE), and the data matrix does not consider the impact of trusted users on the deep preferences of target users, this paper proposes a recommendation algorithm based on stack de-noising auto-encoder (SDAE) which integrates the preferences of trusted users. Firstly, the score vector is used as the input of the auto-encoder, and the mask vector is designed to train the potential preference of the target user. Secondly, the deep preference of the target user and the trusted user is obtained by the weighted fusion of the features of the two hidden layers of the auto-encoder. Thirdly, in order to reduce the impact of noise on the prediction accuracy, the cascaded auto-encoder model is constructed and trained according to the greedy training method layer by layer training. Finally, SDAE model is compared with other models on different data sets. The experimental results show that SDAE model has better recommendation performance.

Efficient Training of Deep Classifiers for Wireless Source Identification Using Test SNR Estimates

We study efficient deep learning training algorithms that process received wireless signals, if a test Signal to Noise Ratio (SNR) estimate is available. We focus on two tasks that facilitate source identification: 1- Identifying the modulation type, 2- Identifying the wireless technology and channel in the 2.4 GHz ISM band. For benchmarking, we rely on recent literature on testing deep learning algorithms against two well-known datasets. We first demonstrate that using training data corresponding only to the test SNR value leads to dramatic reductions in training time while incurring a small loss in average test accuracy, as it improves the accuracy for low SNR values. Further, we show that an erroneous test SNR estimate with a small positive offset is better for training than another having the same error magnitude with a negative offset. Secondly, we introduce a greedy training SNR Boosting algorithm that leads to uniform improvement in accuracy across all tested SNR values, while using a small subset of training SNR values at each test SNR. Finally, we demonstrate the potential of bootstrap aggregating (Bagging) based on training SNR values to improve generalization at low test SNR values with scarcity of training data.

Cloud service security evaluation of smart grid using deep belief network

In this paper, we analyse key security problems in the scalable platform architecture of cloud service of smart grid. We also establish evaluating process according to the lack of evaluation mechanism. They evaluate security risks in 5 respects which are policy and organisational risks, general technical risks, SaaS risks, PaaS risks and IaaS risks. The evaluation model based on deep belief network (DBN) is proposed which is composed of multiple RBMs and a BP neural network. The RBMs are trained by greedy training algorithm, and then BP algorithm is used to fine-tuning. After case verification, it is found that the various errors (including mean absolute error (MAE), mean relative error (MRE), mean square error (MSE), maximum error and minimum error) of DBN model are the smallest compared BP and AE model. It avoids the problem that multilayer neural network is trapped in the local optimum.

Unsupervised Learning on Resistive Memory Array Based Spiking Neural Networks.

Spiking Neural Networks (SNNs) offer great potential to promote both the performance and efficiency of real-world computing systems, considering the biological plausibility of SNNs. The emerging analog Resistive Random Access Memory (RRAM) devices have drawn increasing interest as potential neuromorphic hardware for implementing practical SNNs. In this article, we propose a novel training approach (called greedy training) for SNNs by diluting spike events on the temporal dimension with necessary controls on input encoding phase switching, endowing SNNs with the ability to cooperate with the inevitable conductance variations of RRAM devices. The SNNs could utilize Spike-Timing-Dependent Plasticity (STDP) as the unsupervised learning rule, and this plasticity has been observed on our one-transistor-one-resistor (1T1R) RRAM devices under voltage pulses with designed waveforms. We have also conducted handwritten digit recognition task simulations on MNIST dataset. The results show that the unsupervised SNNs trained by the proposed method could mitigate the requirement for the number of gradual levels of RRAM devices, and also have immunity to both cycle-to-cycle and device-to-device RRAM conductance variations. Unsupervised SNNs trained by the proposed methods could cooperate with real RRAM devices with non-ideal behaviors better, promising high feasibility of RRAM array based neuromorphic systems for online training.

A $Q$ -Learning-Based Proactive Caching Strategy for Non-Safety Related Services in Vehicular Networks

Content caching has brought huge potential for the provisioning of non-safety related infotainment services in future vehicular networks. Assisted by multiaccess edge computing, roadside units (RSUs) could become cache-capable and offer fast caching services to moving vehicles for content providers. On the other hand, deep learning makes it possible to accurately estimate the behavior of vehicles, which enables effective proactive caching strategies. However, caching services considering both the mobility of vehicles and storage could incur increased latency and considerable cost due to the cache size needed in RSUs. In this paper, we model such a problem using Markov decision processes, and propose a heuristic ${Q}$ -learning solution together with vehicle movement predictions based on a long short-term memory network. The optimal caching strategy which minimizes the latency of caching services can be derived by our heuristic ${\varepsilon }_{{n}}$ -greedy training processes. Numerical results demonstrate that our proposed strategy can achieve better performance compared with several baselines under different prediction accuracies.

Intelligent fault diagnosis method of planetary gearboxes based on convolution neural network and discrete wavelet transform

Considering the planetary gearbox vibration signals show highly non-stationary and non-linear behavior because of wind turbines (WTs) often working under time-varying running conditions, we propose an effective and reliable method based on convolution neural network (CNN) and discrete wavelet transformation (DWT) to identify the fault conditions of planetary gearboxes. Firstly, the discrete wavelet transformation is used with the goal of presenting more salient and comprehensive time-frequency distributed representation. Secondly, the deep hierarchical structure of CNN constructed by the alternating convolution layers and subsample layers is trained using a forward transmitting rule of greedy training layer by layer and translates the low-level features of input to the high-level features in order to identify the internal characteristic. Finally, a top classifier Softmax is added at uppermost layer of CNN and the backpropagation process is conducted to fine-tune the parameters of CNN, establishing the mapping relation among the feature space and the fault space. Thus, feature extraction process and fault recognition are incorporated into a general-purpose learning procedure. The experimental results of the fault diagnosis for the planetary gearbox demonstrated the effectiveness and feasibility of the proposed method.

An LP empirical quadrature procedure for reduced basis treatment of parametrized nonlinear PDEs

We present a model reduction formulation for parametrized nonlinear partial differential equations (PDEs). Our approach builds on two ingredients: reduced basis (RB) spaces which provide rapidly convergent approximations to the parametric manifold; sparse empirical quadrature rules which provide rapid evaluation of the nonlinear residual and output forms associated with the RB spaces. We identify both the RB spaces and the sparse quadrature rules in the offline stage through a greedy training procedure over the parameter domain; the procedure requires the dual norm of the finite element (FE) residual at many training points in the parameter domain, but only very few FE solutions—the snapshots retained in the RB space. The quadrature rules are identified by a linear program (LP) empirical quadrature procedure (EQP) which (i) admits efficient solution by a simplex method, and (ii) directly controls the solution error induced by the approximate quadrature. We demonstrate the formulation for a parametrized neo-Hookean beam: the dimension of the approximation space and the number of quadrature points are both reduced by two orders of magnitude relative to FE treatment, with commensurate savings in computational cost.

Online multi-label dependency topic models for text classification

Multi-label text classification is an increasingly important field as large amounts of text data are available and extracting relevant information is important in many application contexts. Probabilistic generative models are the basis of a number of popular text mining methods such as Naive Bayes or Latent Dirichlet Allocation. However, Bayesian models for multi-label text classification often are overly complicated to account for label dependencies and skewed label frequencies while at the same time preventing overfitting. To solve this problem we employ the same technique that contributed to the success of deep learning in recent years: greedy layer-wise training. Applying this technique in the supervised setting prevents overfitting and leads to better classification accuracy. The intuition behind this approach is to learn the labels first and subsequently add a more abstract layer to represent dependencies among the labels. This allows using a relatively simple hierarchical topic model which can easily be adapted to the online setting. We show that our method successfully models dependencies online for large-scale multi-label datasets with many labels and improves over the baseline method not modeling dependencies. The same strategy, layer-wise greedy training, also makes the batch variant competitive with existing more complex multi-label topic models.

Intelligent fault diagnosis of rolling bearing using hierarchical convolutional network based health state classification

Rolling bearing tips are often the most susceptible to electro-mechanical system failure due to high-speed and complex working conditions, and recent studies on diagnosing bearing health using vibration data have developed an assortment of feature extraction and fault classification methods. Due to the strong non-linear and non-stationary characteristics, an effective and reliable deep learning method based on a convolutional neural network (CNN) is investigated in this paper making use of cognitive computing theory, which introduces the advantages of image recognition and visual perception to bearing fault diagnosis by simulating the cognition process of the cerebral cortex. The novel feature representation method for bearing data is first discussed using supervised deep learning with the goal of identifying more robust and salient feature representations to reduce information loss. Next, the deep hierarchical structure is trained in a robust manner that is established using a transmitting rule of greedy training layer by layer. Convolution computation, rectified linear units, and sub-sampling are applied for weight replication and reducing the number of parameters that need to be learned to improve the general feed-forward back propagation training. The CNN model could thus reduce learning computation requirements in the temporal dimension, and an invariance level of working condition fluctuation and ambient noise is provided by identifying the elementary features of bearings. A top classifier followed by a back propagation process is used for fault classification. Contrast experiments and analyses have been undertaken to delineate the effectiveness of the CNN model for fault classification of rolling bearings.

Fault diagnosis of rotary machinery components using a stacked denoising autoencoder-based health state identification

Effective fault diagnosis has long been a research topic in the prognosis and health management of rotary machinery engineered systems due to the benefits such as safety guarantees, reliability improvements, and economical efficiency. This paper investigates an effective and reliable deep learning method known as stacked denoising autoencoder (SDA), which is shown to be suitable for certain health state identifications for signals containing ambient noise and working condition fluctuations. SDA has become a popular approach to achieve the promised advantages of deep architecture-based robust feature representations. In this paper, the SDA-based fault diagnosis method contains three successive steps: health states are first divided into training and testing groups for the SDA model, a deep hierarchical structure is then established with a transmitting rule of greedy training, layer by layer, where sparsity representation and data destruction are applied to obtain high-order characteristics with better robustness in the iteration learning. Validation data are finally employed to confirm the fault diagnosis results of the SDA, where existing health state identification methods are used for comparison. Rotating machinery datasets are employed to demonstrate the effectiveness of the proposed method.