Train Acceleration Research Articles

Distributed classification based on distances between probability distributions in feature space

We consider a distributed framework where training and test samples drawn from the same distribution are available, with the training instances spread across disjoint nodes. In this setting, a novel learning algorithm based on combining with different weights the outputs of classifiers trained at each node is proposed. The weights depend on the distributional distance between each node and the test set in the feature space. Two different weighting approaches are introduced, which are referred to as per-Node Weighting (pNW) and per-Instance Weighting (pIW). While pNW assigns the same weight to all test instances at each node, pIW allows distinct weights for test instances differently represented at the node. By construction, our approach is particularly useful to deal with unbalanced nodes. Our methods require no communication between nodes, allowing for data privacy, independence of the kind of trained classifier at each node and maximum training speedup. In fact, our methods do not require retraining of the node’s classifiers if available. Although a range of different combination rules are considered to ensemble the single classifiers, theoretical support for the optimality of using the sum rule is provided. Our experiments illustrate all of these properties and show that pIW produces the highest classification accuracies compared with pNW and the standard unweighted approaches.

Effective and Efficient Batch Normalization Using a Few Uncorrelated Data for Statistics Estimation.

Deep neural networks (DNNs) thrive in recent years, wherein batch normalization (BN) plays an indispensable role. However, it has been observed that BN is costly due to the huge reduction and elementwise operations that are hard to be executed in parallel, which heavily reduces the training speed. To address this issue, in this article, we propose a methodology to alleviate the BN's cost by using only a few sampled or generated data for mean and variance estimation at each iteration. The key challenge to reach this goal is how to achieve a satisfactory balance between normalization effectiveness and execution efficiency. We identify that the effectiveness expects less data correlation in sampling while the efficiency expects more regular execution patterns. To this end, we design two categories of approach: sampling or creating a few uncorrelated data for statistics' estimation with certain strategy constraints. The former includes "batch sampling (BS)" that randomly selects a few samples from each batch and "feature sampling (FS)" that randomly selects a small patch from each feature map of all samples, and the latter is "virtual data set normalization (VDN)" that generates a few synthetic random samples to directly create uncorrelated data for statistics' estimation. Accordingly, multiway strategies are designed to reduce the data correlation for accurate estimation and optimize the execution pattern for running acceleration in the meantime. The proposed methods are comprehensively evaluated on various DNN models, where the loss of model accuracy and the convergence rate are negligible. Without the support of any specialized libraries, 1.98× BN layer acceleration and 23.2% overall training speedup can be practically achieved on modern GPUs. Furthermore, our methods demonstrate powerful performance when solving the well-known "micro-BN" problem in the case of a tiny batch size. This article provides a promising solution for the efficient training of high-performance DNNs.

On extreme learning machines in sequential and time series prediction: A non-iterative and approximate training algorithm for recurrent neural networks

Recurrent neural networks (RNN) are a type of artificial neural networks (ANN) that have been successfully applied to many problems in artificial intelligence. However, they are expensive to train since the number of learned weights grows exponentially with the number of hidden neurons. Non-iterative training algorithms have been proposed to reduce the training time, mainly on feedforward ANN. In this work, the application of non-iterative randomized training algorithms to various RNN architectures, including Elman RNN, fully connected RNN, and long short-term memory (LSTM), are investigated. The mathematical formulation and theoretical computational complexity of the proposed algorithms are presented. Finally, their performance is empirically compared to other iterative RNN training algorithms on time series prediction and sequential decision-making problems. Non-iteratively-trained RNN architectures showed promising results as significant training speedup of up to 99%, and improved repeatability were achieved compared to backpropagation-trained RNN. Although the decrease in prediction accuracy was found to be statistically significant based on Friedman and ANOVA testing, some applications like real-time embedded systems can tolerate and make use of that.

Tera-scale coordinate descent on GPUs

In this work we propose an asynchronous, GPU-based implementation of the widely-used stochastic coordinate descent algorithm for convex optimization. We define the class of problems that can be solved using this method, and show that it includes many popular machine learning applications. For three such applications, we demonstrate at least a 10× training speed-up relative to a state-of-the-art implementation that uses all available resources of a modern CPU. In order to train on very large datasets that do not fit inside the memory of a single GPU, we then consider techniques for distributed learning. We show that while such techniques do not necessarily allow one to achieve further speed-up, they do allow one to train on datasets that would otherwise not fit into memory. We thus propose a distributed learning system that uses the synchronous CoCoA framework to distribute the global optimization across GPUs, and our novel asynchronous algorithm to solve the corresponding local optimizations within each GPU. We benchmark such a system using a 200 GB dataset that consists of 1 billion training examples. We show by scaling out across 16 GPUs, we can train an SVM model to a high degree of accuracy in around 1 min: a 15× speed-up in training time compared to a state-of-the-art CPU-based implementation that uses 640 threads distributed across 8 CPUs.

Unified Partial Configuration Model Framework for Fast Partially Occluded Object Detection in High-Resolution Remote Sensing Images

Partially occluded object detection (POOD) has been an important task for both civil and military applications that use high-resolution remote sensing images (HR-RSIs). This topic is very challenging due to the limited object evidence for detection. Recent partial configuration model (PCM) based methods deal with occlusion yet suffer from the problems of massive manual annotation, separate parameter learning, and low training and detection efficiency. To tackle this, a unified PCM framework (UniPCM) is proposed in this paper. The proposed UniPCM adopts a part sharing mechanism which directly shares the root and part filters of a deformable part-based model (DPM) among different partial configurations. It largely reduces the convolution overhead during both training and detection. In UniPCM, a novel DPM deformation deviation method is proposed for spatial interrelationship estimation of PCM, and a unified weights learning method is presented to simultaneously obtain the weights of elements within each partial configuration and the weights between partial configurations. Experiments on three HR-RSI datasets show that the proposed UniPCM method achieves a much higher training and detection efficiency for POOD compared with state-of-the-art PCM-based methods, while maintaining a comparable detection accuracy. UniPCM obtains a training speedup of maximal 10× and 2.5× for airplane and ship, and a detection speedup of maximal 7.2×, 4.1× and 2.5× on three test sets, respectively.

Efficient Scheduling in Training Deep Convolutional Networks at Large Scale

The deep convolutional network is one of the most successful machine learning models in recent years. However, training large deep networks is a time consuming process. Due to a large number of parameters in these networks, the efficiency of data parallel methods is usually limited by the communication speed of networks. In this paper, we introduce two new algorithms to speedup training large deep networks with multiple machines: 1) propose a new scheduling algorithm to reduce communication delay in gradient transmission and 2) present a new collective algorithm based on reverse-reduce tree to reduce link contentions. We implement our algorithms on a well-known library Caffe and obtain near linearly scaling performance on commodity Ethernet networks.

Automatic and Fast PCM Generation for Occluded Object Detection in High-Resolution Remote Sensing Images

Partial configuration model (PCM) is an occluded object detection method in high-resolution remote sensing images (HR-RSIs) based on the deformable part-based model (DPM). However, it needs extra category predefinition, considerable part-level annotation, and repeated multimodel training. In this letter, an automatic and fast PCM generation method is proposed based on a novel part sharing mechanism. We propose to share parts from one trained DPM model (tDPM) among different models of partial configurations (PCs) to address the above problems. PCs are first designed according to part anchors of tDPM. The model is then generated through corresponding parts selection, root coverage cropping, and elements reweighing. This method avoids the need for manual category predefinition and part-level annotation, while largely reducing the computation of PCM training. Experimental results on three HR-RSI data sets show that the proposed method obtains a training speedup of $6.7\times $ and $2\times $ for each PC of airplane and ship categories, while achieving a comparable accuracy compared with PCM.

Energy efficient parallel neuromorphic architectures with approximate arithmetic on FPGA

In this paper, we present the parallel neuromorphic processor architectures for spiking neural networks on FPGA. The proposed architectures address several critical issues pertaining to efficient parallelization of the update of membrane potentials, on-chip storage of synaptic weights and integration of approximate arithmetic units. The trade-offs between throughput, hardware cost and power overheads for different configurations are thoroughly investigated. Notably, for the application of handwritten digit recognition, a promising training speedup of 13.5x and a recognition speedup of 25.8x are achieved by a parallel implementation whose degree of parallelism is 32. In spite of the 120MHz operating frequency, the 32-way parallel hardware design demonstrates a 59.4x training speedup over the single-thread software program running on a 2.2GHz general purpose CPU. Equally importantly, by leveraging the built-in resilience of the neuromorphic architecture we demonstrate the energy benefit resulted from the use of approximate arithmetic computation. Up to 20% improvement in energy consumption is achieved by integrating approximate multipliers into the system while maintaining almost the same level of recognition rate achieved using standard multipliers. To the best of our knowledge, it is the first time that the approximate computing and parallel processing are applied to FPGA based spiking neural networks. The influence of the parallel processing on the benefits of approximate computing is also discussed in detail.

Slip and Slide Detection and Adaptive Information Sharing Algorithms for High-Speed Train Navigation Systems

The position and velocity information of high-speed trains (HSTs) are essential to passenger safety, operational efficiency, and maintenance, for which an accurate navigation system is required. In this paper, we propose a two-stage federated Kalman filter (TS-FKF) for an HST navigation system that uses multi-sensors, such as tachometer, inertial navigation system, differential GPS, and RFID, with a feedback scheme. However, the FKF with a feedback scheme often shows severe performance degradation in the presence of undetected large sensor errors. Tachometers often have large slip or slide errors during the train's acceleration, deceleration, and moving along a curved railway, and there are significant performance differences between different sensors. To make the proposed system robust to these errors, we propose a slip and slide detection algorithm for the tachometer and an adaptive information-sharing algorithm to deal with a large tachometer error and performance difference between sensors. We provide theoretical analysis and simulation results to demonstrate the performance of the proposed navigation system with the proposed algorithms.

State-Clustering Based Multiple Deep Neural Networks Modeling Approach for Speech Recognition

The hybrid deep neural network (DNN) and hidden Markov model (HMM) has recently achieved dramatic performance gains in automatic speech recognition (ASR). The DNN-based acoustic model is very powerful but its learning process is extremely time-consuming. In this paper, we propose a novel DNN-based acoustic modeling framework for speech recognition, where the posterior probabilities of HMM states are computed from multiple DNNs (mDNN), instead of a single large DNN, for the purpose of parallel training towards faster turnaround. In the proposed mDNN method all tied HMM states are first grouped into several disjoint clusters based on data-driven methods. Next, several hierarchically structured DNNs are trained separately in parallel for these clusters using multiple computing units (e.g. GPUs). In decoding, the posterior probabilities of HMM states can be calculated by combining outputs from multiple DNNs. In this work, we have shown that the training procedure of the mDNN under popular criteria, including both frame-level cross-entropy and sequence-level discriminative training, can be parallelized efficiently to yield significant speedup. The training speedup is mainly attributed to the fact that multiple DNNs are parallelized over multiple GPUs and each DNN is smaller in size and trained by only a subset of training data. We have evaluated the proposed mDNN method on a 64-hour Mandarin transcription task and the 320-hour Switchboard task. Compared to the conventional DNN, a 4-cluster mDNN model with similar size can yield comparable recognition performance in Switchboard (only about 2% performance degradation) with a greater than 7 times speed improvement in CE training and a 2.9 times improvement in sequence training, when 4 GPUs are used.

Study on the critical diameter of the subway tunnel based on the pressure variation

When the subway train operates at a speed higher than 100 km/h, the corresponding aerodynamic issue becomes severe. To meet the future requirement for the speedup of subway trains, a research on the critical diameters of the subway tunnel for trains operating at 120 and 140 km/h has been performed based on passengers’ aural discomfort caused by rail tunnel pressure variation. A three-dimensional computational fluid dynamic approach has been adopted for analysis. Meanwhile, trains with different airtight indices are considered and the pressure variations inside and outside the trains are both under investigation. Based on the corresponding criteria for different airtight indices, critical tunnel diameters for trains running at different speeds have been determined. This study would aid in the tunnel section design for future high-speed subway trains.

Efficient approximations of robust soft learning vector quantization for non-vectorial data

Due to its intuitive learning algorithms and classification behavior, learning vector quantization (LVQ) enjoys a wide popularity in diverse application domains. In recent years, the classical heuristic schemes have been accompanied by variants which can be motivated by a statistical framework such as robust soft LVQ (RSLVQ). In its original form, LVQ and RSLVQ can be applied to vectorial data only, making it unsuitable for complex data sets described in terms of pairwise relations only. In this contribution, we address kernel RSLVQ which extends its applicability to data which are described by a general Gram matrix. While leading to state of the art results, this extension has the drawback that models are no longer sparse, and quadratic training complexity is encountered due to the dependency of the method on the full Gram matrix. In this contribution, we investigate the performance of a speed-up of training by means of low rank approximations of the Gram matrix, and we investigate how sparse models can be enforced in this context. It turns out that an efficient Nyström approximation can be used if data are intrinsically low dimensional, a property which can be efficiently checked by sampling the variance of the approximation prior to training. Further, all models enable sparse approximations of comparable quality as the full models using simple geometric approximation schemes only. We demonstrate the behavior of these approximations in a couple of benchmarks.

Learning Visual Object Classifiers with only Positive Images

In this paper, a novel approach is proposed for learning visual object classifiers with only positive images. Both positive and negative examples are collected from the positive images, and the false negative examples are filtered out by the Ramp Loss function, which has a strong ability of suppressing the influence of outliers. Meanwhile, the Latent Structural Support Vector Machines are adopted to estimate the best bounding boxes due to the unavoidable annotation errors. In learning the sub-problems, an extremely fast learning algorithm, CCCP-SGD, is proposed to optimize the Ramp Loss-based SVMs based on the Stochastic Gradient Descent algorithm. Experiments demonstrate that the CCCP-SGD algorithm can reduce the computing time both in the training and predicting phases, especially with a significant speedup in training, whilst it can also yield robust generalization performance when dataset is noisy. Besides, the visual object classifiers learnt by our approach with only positive images almost have the same performance as the classifiers learned with both positive and negative examples in object detection tasks.

THE DIAGNOSIS MODEL BUILDING ON THE BASIS OF NEGATIVE SELECTION PARADIGM USING THE PRINCIPLE OF DETECTOR ASKING

A new negative selection model using masked detectors with training method has been developed. Various stopping criteria for the training method have been investigated. The stopping criterion is proposed. It allows to terminate training procedure, to speedup training process and to reduce computational resources.

360369 A SIMPLE IDENTIFICATION METHOD FOR MEMBER VIBRATION PROPERTIES OF RAILWAY VIADUCTS(Condition Monitoring,Technical Session)

For train speed-up, resonance response of structures and these environmental performance of it such as ground vibration and noise are critical issues from the viewpoints of safety and serviceability, respectively. To adequately tackle these issues, the vibration properties of structures must be thoroughly understood on an each member. In this study, we estimated the vibration mode shape and the natural frequency of RC rigid frame railway viaducts based on results of simultaneous multipoint measurement and proposed an estimation method for member vibration.

A highly efficient implementation of a backpropagation learning algorithm using matrix ISA

BackPropagation (BP) is the most famous learning algorithm for Artificial Neural Networks (ANN). BP has received intensive research efforts to exploit its parallelism in order to reduce the training time for complex problems. A modified version of BP based on matrix–matrix multiplication was proposed for parallel processing. In this paper, we present the implementation of Matrix BackPropagation (MBP) using scalar, vector, and matrix Instruction Set Architectures (ISAs). Besides this, we show that the performance of the MBP is improved by switching from scalar ISA to vector ISA. It is further improved by switching from vector ISA to matrix ISA. On a practical application, speech recognition, the speedup of training a neural network using unrolling scalar ISA over scalar ISA is 1.83. On eight parallel lanes, the speedups of using vector, unrolling vector, and matrix ISAs are respectively 10.33, 11.88, and 15.36, where the maximum theoretical speedup is 16. The results obtained show that the use of matrix ISA gives a performance close to optimal, because of reusing the loaded data, decreasing the loop overhead, and overlapping the memory operations with arithmetic operations.

A highly efficient implementation of back propagation algorithm using matrix instruction set architecture

Back Propagation (BP) training algorithm has received intensive research efforts to exploit its parallelism in order to reduce the training time for complex problems. A modified version of BP based on matrix-matrix multiplication was proposed for parallel processing. This paper discusses the implementation of Matrix Back Propagation (MBP) using scalar, vector, and matrix instruction set architecture (ISA). Besides, it shows that the performance of the MBP is improved by switching form scalar to vector ISA and form vector to matrix ISA. On a practical application, speech recognition, the speedup of training a neural network using unrolling scalar over scalar ISA is 1.83. On eight parallel lanes, the speedup of using vector, unrolling vector, and matrix ISA are respectively 10.33, 11.88, and 15.36, where the maximum theoretical speedup is 16. Our results show that the use of matrix ISA gives a performance close to the optimal because of reusing the loaded data, decreasing the loop overhead, and overlapping the memory operations by arithmetic operations.

Improving Performance of Type T Overhead Rigid Conductor Lines

Because of their simple structure and low-maintenance characteristics, railways in Japan have often adopted overhead rigid conductor lines for current collection systems applicable to underground track sections or tunnels. With regard to the copper Type T rigid conductor line adopted on some JR commercial lines, undulating wear appeared at intervals of about 60 mm on contact wire sliding surfaces. Such wear would cause impaired train speedup and thus increase maintenance costs. We investigated the causes of the wears, and proposed measures to improve the quality of overhead rigid conductor lines.

The critical loading for lateral buckling of continuous welded rail

The most significant differences between continuous welded rails (CWRs) and general split-type connectors are axial compression in the longitudinal direction, buckling stability and other issues generated under the influence of thermal effect. Under thermal effect, a dynamical behavior similar to that of a beam fixed on two sides occurs in the central locked area of the welded rail, as there is axial compression but no possibility of sliding. Continuous welded rails do not contract or expand, and are supported by the dynamical system made up of ballasts and rail clips. The rail-support system mentioned above has the features of non-uniform material distribution and uncertainty of construction quality. Due to these facts, the dynamics method based on the linear elastic hypothesis cannot correctly evaluate the rail's buckling conditions. This study is aimed at applying Finite Difference Method (FDM) and Monte Carlo Random Normal Variables Method to the analysis of welded rail's buckling behavior during the train's acceleration and deceleration, under thermal effect and uncertain factors of ballast and rail clips. The analysis result showed that buckling occurs under the combined effect of thermal effect and the train's deceleration force co-effect and the variance ratio of ballast and rail clips is over 0.85, or under the combined effect of thermal effect and the train's acceleration force when the variance ratio is over 0.88.

Impact Factors of Concrete Girders Coping with Train Speed-up

The dynamic response of railway bridges is a problem of the resonant response of girders that occurs because bridges are excited at a constant period by running trains. In general, this is referred to as "the speed effect of multiple-axle moving loads." In this study, vehicle running tests were performed to examine 56 bridges of various structural types and span lengths. The impact factor was measured to be about 1.0 on four girders, and the damping ratio as low as 0.02. Based on the results of these running tests and numerical analysis, a revision of the railway design standard was proposed, to cope with the speed-up of trains.