Parallel Training Research Articles

Parallel Training: An ACP-Based Training Framework for Iterative Learning in Uncertain Driving Spaces

The traffic environment and driving behaviors are of great complexity and uncertainty in our physical world. Therefore, training in the digital world with low cost and diverse complexities become popular for autonomous driving in recent years. However, the current training methods tend to be limited to static data sets and deterministic models that do not sufficiently take into account the uncertainty and diversity prevalent in real traffic scenarios. These approaches also limit more possibilities for the comprehensive development and optimization of vision systems. In this paper, we develop a parallel training method based on artificial systems, computational experiments, and parallel execution (ACP) for the intelligent optimization and learning of the aforementioned agents in uncertain driving spaces. Parallel training creates a virtual driving space following the instruction of the ACP approach and conducts large-scale rehearsal experiments for possible scenarios. By enhancing the diversity of virtual scenarios, intelligent vehicles are trained to respond and adapt to the diverse uncertainties in the physical real-world driving space. Specifically, parallel training first proposes a standard operating procedure for intelligent driving systems, namely the projection-emergence-convergence-operation (PECO) loop. Digital quadruplets for parallel training, i.e., physical, descriptive, predictive, and prescriptive coaches, are also proposed. With the guidance of parallel training, virtual and real-world driving spaces are set up in parallel and interact frequently. They are closely linked and unified in opposition to each other, ultimately building a parallel driving system that fulfills safety, security, sustainability, sensitivity, service, and smartness (6S).

Wireless Distributed Learning: A New Hybrid Split and Federated Learning Approach

Cellular-connected unmanned aerial vehicle (UAV) with flexible deployment is foreseen to be a major part of the sixth generation (6G) networks. The UAVs connected to the base station (BS), as aerial users (UEs), could exploit machine learning (ML) algorithms to provide a wide range of advanced applications, like object detection and video tracking. Conventionally, the ML model training is performed at the BS, known as centralized learning (CL), which causes high communication overhead due to the transmission of large datasets, and potential concerns about UE privacy. To address this, distributed learning algorithms, including federated learning (FL) and split learning (SL), were proposed to train the ML models in a distributed manner via only sharing model parameters. FL requires higher computational resource on the UE side than SL, while SL has larger communication overhead when the local dataset is large. To effectively train an ML model considering the diversity of UEs with different computational capabilities and channel conditions, we first propose a novel distributed learning architecture, a hybrid split and federated learning (HSFL) algorithm by reaping the parallel model training mechanism of FL and the model splitting structure of SL. We then provide its convergence analysis under non-independent and identically distributed (non-IID) data with random UE selection scheme. By conducting experiments on training two ML models, Net and AlexNet, in wireless UAV networks, our results demonstrate that the HSFL algorithm achieves higher learning accuracy than FL and less communication overhead than SL under IID and non-IID data, and the learning accuracy of HSFL algorithm increases with the increasing number of the split training UEs. We further propose a Multi-Arm Bandit (MAB) based best channel (BC) and best 2-norm (BN2) (MAB-BC-BN2) UE selection scheme to select the UEs with better wireless channel quality and larger local model updates for model training in each round. Numerical results demonstrate it achieves higher learning accuracy than BC, MAB-BC and MAB-BN2 UE selection scheme under non-IID, Dirichlet-nonIID and Dirichlet-Imbalanced data.

Traffic Prediction With Missing Data: A Multi-Task Learning Approach

Traffic speed prediction based on real-world traffic data is a classical problem in intelligent transportation systems (ITS). Most existing traffic speed prediction models are proposed based on the hypothesis that traffic data are complete or have rare missing values. However, such data collected in real-world scenarios are often incomplete due to various human and natural factors. Although this problem can be solved by first estimating the missing values with an imputation model and then applying a prediction model, the former potentially breaks critical latent features and further leads to the error accumulation issues. To tackle this problem, we propose a graph-based spatio-temporal autoencoder that follows an encoder-decoder structure for spatio-temporal traffic speed prediction with missing values. Specifically, we regard the imputation and prediction as two parallel tasks and train them sequentially to eliminate the negative impact of imputation on raw data for prediction and accelerate the model training process. Furthermore, we utilize graph convolutional layers with a self-adaptive adjacency matrix for spatial dependencies modeling and apply gated recurrent units for temporal learning. To evaluate the proposed model, we conduct comprehensive case studies on two real-world traffic datasets with two different missing patterns and a wide and practical missing rate range from 20% to 80%. Experimental results demonstrate that the model consistently outperforms the state-of-the-art traffic prediction with missing values methods and achieves steady performance in the investigated missing scenarios and prediction horizons.

Chinese students’ motivation to adopt distance learning of the Russian language

Introduction. Currently, restrictions related to the coronavirus pandemic have not yet been fully lifted in the People's Republic of China and students studying in Russia periodically switch to distance learning. The purpose of the article is to study the specifics of students' motivation to use distance technologies for teaching the Russian language using internationally recognized methods of measurement and statistical data analysis. Materials and Methods. The Instructional Materials Motivation Survey (IMMS) measuring materials for studying the specifics of the educational motivation of students of the Ural Federal University (hereinafter referred to as UrFU) from the PRC for the use of distance technologies for teaching the Russian language were created for the AttentionRelevance-Confidence-Satisfaction (ARCS) structural model. We studied the relationship between the personal data of students (gender, age, year of study, educational program, additional language training and the results of measuring educational motivation (IMMS) for the use of distance technologies for teaching the Russian language and its structural components: attention (ATT), relevance (REL), Confidence (CON), Satisfaction (SAT).The study involved 673 UrFU students from the PRC, who had experience of long-term distance learning under restrictions due to the coronavirus pandemic. Statistical processing of the measurement results was carried out using the structural equation modeling method Structural Equation Modeling (SEM). A confirmatory and exploratory factorial analysis of the obtained results was carried out using the SPSS software tool, including the AMOS module. Results. The developed IMMS questionnaire was validated by confirmatory factor analysis of IMMS measurement results using the AMOS module of the SPSS software tool. The method of exploratory factor analysis revealed statistically significant relationships between the indicators of students' educational motivation for using distance technologies for teaching the Russian language (IMMS) and its structural components with the personal data of the respondents. The gender of the student and the presence of parallel additional language training were found to have the strongest influence on learning motivation (girls are more motivated than boys). The gender of the student and the presence of parallel additional language training (regression coefficient r = 0.61) have the strongest influence on the structural component of learning motivation relevance (REL). Conclusion. The results of the study showed a high level of motivation for distance learning of the Russian language among UrFU students from China which is due to the effective design of educational materials. It is shown that when developing motivating teaching materials for distance learning, it is necessary to take into account the gender composition of trainees and the availability of parallel additional training in the subject.

Toward Large-Scale Graph-Based Traffic Forecasting: A Data-Driven Network Partitioning Approach

Network partitioning is recognized as an effective auxiliary approach for solving transportation tasks on large-scale traffic networks in a domain-decomposition (DD) manner. Most of the existing related partitioning algorithms are explicitly designed to traffic management problems and merely focus on the implied topology of the networks. In this article, toward the practical problems that happened to traffic forecasting (TF) tasks, we propose a network-partitioning-based DD framework to improve graph convolutional network (GCN)-based predictors’ performance on large-scale transportation networks. Particularly, we devise a data-driven network-partitioning approach, namely, speed-matching-partitioning (SMP), which employs not only the topological features but also the traffic speed observations of traffic networks for partitioning. Additionally, we propose a data-parallel training strategy that feeds partitioned subnetworks into independent predictors for parallel training. The proposed approach is tested by comprehensive case studies on three real-world data sets to evaluate its effectiveness. The results indicate that the proposed approach can help improve GCN-based predictors’ accuracy and training efficiency on both small and relatively large traffic data sets. Furthermore, we investigate the model sensitivity to the selection of graph representations and framework parameters, and the learning efficiency of the data-parallel training strategy.

Taking care of who cares: Evaluation of a training program for home palliative care professionals.

To evaluate an enrichment training program targeted at home palliative care professionals in terms of its effects and participants' satisfaction. The program had 2 main aims: give voice to professionals' emotional fatigue and promote their personal resources. One hundred twenty-three home palliative care professionals participated in 12 parallel training courses; each course consisted of four 3-hour meetings led by 2 trainers and involved about 10-15 participants. The program adopted the method and tools typical of the enrichment approach, with the insertion of an art therapy exercise in the central meetings. The topics addressed were the following: emotional awareness in care relationship; the recognition of the needs of the patient, the family, and the professional himself; the inevitability of the death of the patient; and the challenges and resources of the multidisciplinary care team. At the first (T1) and last (T2) meetings, participants filled in a self-report questionnaire assessing work emotional fatigue, empowerment, generativity, and satisfaction with the course. Participants were highly satisfied with the course. They reported a higher level of work emotional fatigue and a higher perception of personal resources, in terms of empowerment (both individual-oriented and relationship-oriented) and generativity at the end of the program than before. Results confirm the need to provide home palliative care professionals with trainings in which they can express, share, and deal with personal and professional needs. This course gave voice to professionals' work emotional fatigue and promoted their personal resources, while enhancing collaboration in the multidisciplinary team.

Coordinated voltage regulation of high renewable-penetrated distribution networks: An evolutionary curriculum-based deep reinforcement learning approach

With the increasing penetration of renewable energy in active distribution networks (ADNs), voltage regulation problem is becoming more and more challenging. In this article, we focus on providing a scalable data-driven approach to ensure the voltage security of ADNs with high penetration of PVs. To this end, we first formulate an optimization problem for real-time voltage control considering source-load-storage collaboration while minimizing the total active power curtailment of PVs. Then, we reformulate the above problem as a Markov game and propose a novel voltage regulation algorithm based on evolutionary curriculum-based multi-agent deep reinforcement learning (EC-MADRL) to solve it. The key idea of the proposed algorithm is to adopt a multi-stage parallel training framework based on attention multi-agent deep deterministic policy gradient algorithm (Attention-MADDPG) and use an evolutionary mechanism to select voltage regulation models with high fitnesses in the previous stage, which means that good agents with best adaptability could be utilized for learning in the environment with a larger number of PVs. Simulation results show the effectiveness and scalability of the proposed algorithm.

Aktivisme Warga dan Negara dalam Kerusakan Lingkungan di Lebak Banten

Environmental damage caused by the negative impact of industrial activities in the Bayah District area which causes public complaints about health. This triggers the emergence of community activism gathered in JMPB with its role which always strives for environmental sustainability and demands industry accountability by reporting directly on forms of pollution to the government so that the government pays more attention to the impact of industry on the environment. The purpose of this study is to find out how the emergence of social movements in Bayah, JMPB’s efforts in reducing environmental damage, and the role of the government in supervising and evaluating industry in Bayah. This research uses a qualitative method with a case study approach. The results showed that the emergence of social movements in Bayah was caused by the presence of PT Cemindo Gemilang which produces cement and has an impact on the environment and public health. The role of JMPB as an indirect actor is conducting protests, environmental socialization, parallel training, reforestation, and reposting forms of pollution to the authorities. Meanwhile, the role of DLH Lebak Regency as a direct actor who has an important role in environmental supervision and can provide sanctions to the industry in accordance with indications of environmental damage that occurs.

A bidirectional DNN partition mechanism for efficient pipeline parallel training in cloud

Recently, deep neural networks (DNNs) have shown great promise in many fields while their parameter sizes are rapidly expanding. To break through the computation and memory limitation of a single machine, pipeline model parallelism is proposed for large-scale DNN training by fully utilizing the computation and storage power of the distributed cluster. Cloud data centers can also provide sufficient computing, storage and bandwidth resources. However, most existing approaches apply layer-wise partitioning, which is difficult to obtain an even model partition result because of the large computational overhead discrepancy between DNN layers, resulting in degraded efficiency. To tackle this issue, we propose “Bi-Partition”, a novel partitioning method based on bidirectional partitioning for forward propagation (FP) and backward propagation (BP), which improves the efficiency of the pipeline model parallelism system. By deliberated designing distinct cut positions for FP and BP of DNN training, workers in the pipeline get nearly equal computational loads, and the balanced pipeline fully utilizes the computing resources. Experiments on various DNN models and datasets validate the efficiency of our mechanism, e.g., the training efficiency achieving up to 1.9times faster than the state-of-the-art method PipeDream.

Exposure to exposure: A protocol for leveraging exposure principles during training to address therapist-level barriers to exposure implementation.

Exposure therapy is a highly effective but underutilized treatment for anxiety disorders. A primary contributor to its underutilization is therapist-level negative beliefs about its safety and tolerability for patients. Given functional similarities between anxious beliefs among patients and negative beliefs among therapists, the present protocol describes how exposure principles can be leveraged during training to target and reduce therapist negative beliefs. The study will take place in two phases. First, is a case-series analysis to fine-tune training procedures that is already complete, and the second is an ongoing randomized trial that tests the novel exposure to exposure (E2E) training condition against a passive didactic approach. A precision implementation framework will be applied to evaluate the mechanism(s) by which training influences aspects of therapist delivery following training. It is hypothesized that the E2E training condition will produce greater reductions in therapists' negative beliefs about exposure during training relative to the didactic condition, and that greater reduction in negative beliefs will be associated with higher quality exposure delivery as measured by coding of videotaped delivery with actual patients. Implementation challenges encountered to date are discussed along with recommendations for future training interventions. Considerations for expansion of the E2E training approach are also discussed within the context of parallel treatment and training processes that may be tested in future training trials.

Cooperative modular reinforcement learning for large discrete action space problem

Deep reinforcement learning (DRL) has achieved remarkable results on high-dimension state tasks. However, it suffers in hard convergence and low sample efficiency when solving large discrete action space problems. To meet these challenges, we develop a cooperative modular reinforcement learning (CMRL) method to distributedly solve the problems with a large discrete action space. A general yet effective task decomposition method is proposed to decompose the complex decision task in a large action space into multiple decision sub-tasks in small action subsets, using a rule-based action division method. The CMRL method consisting of multiple Critic networks is proposed to settle the multiple sub-tasks, where each Critic network learns a decomposed value function to obtain the local optimal action in a sub-task. The global optimal action is cooperatively chosen by all local optimal actions. Moreover, we propose a new parallel training mechanism, which trains multiple Critic networks with different models and multi-data in parallel. Mathematical properties are proposed to analyze the rationality and superiority of CMRL. Four different simulation experiments are conducted to verify the generality and effectiveness of CMRL for large action space problems. The results show that CMRL has superior performance on training efficiency compared with classical and latest DRL methods while maintaining the accuracy of the solution.

Hybrid robust convolutional autoencoder for unsupervised anomaly detection of machine tools under noises

• A new FDD loss function to suppress the noises is designed. • Construct the PCDF module to enhance the robustness of the network. • The unsupervised anomaly detection of machine tools under noises. • The proposed HRCAE performs effective in the CNC machine tool. Anomaly detection of machine tools plays a vital role in the machinery industry to sustain efficient operation and avoid catastrophic failures. Compared to traditional machine learning and signal processing methods, deep learning has greater adaptive capability and end-to-end convenience. However, challenges still exist in recent research in anomaly detection of machine tools based on deep learning despite the marvelous endeavors so far, such as the necessity of labeled data for model training and insufficient consideration of noise effects. During machine operation, labeled data is often difficult to obtain; the collected data contains varying degrees of noise disturbances. To address the above challenges, this paper develops a hybrid robust convolutional autoencoder (HRCAE) for unsupervised anomaly detection of machine tools under noises. A parallel convolutional distribution fitting (PCDF) module is constructed, which can effectively fuse multi-sensor information and enhance network robustness by training in parallel to better fit the data distribution with unsupervised learning. A fused directional distance (FDD) loss function is designed to comprehensively consider the distance and angle differences among the data, which can effectively suppress the influence of noises and further improve the model robustness. The proposed method is validated by real computer numerical control (CNC) machine tool data, obtaining better performance of unsupervised anomaly detection under different noises compared to other popular unsupervised improved autoencoder methods.

Efficient Knowledge Graph Embedding Training Framework with Multiple GPUs

When training a large-scale knowledge graph embedding (KGE) model with multiple graphics processing units (GPUs), the partition-based method is necessary for parallel training. However, existing partition-based training methods suffer from low GPU utilization and high input/output (10) overhead between the memory and disk. For a high 10 overhead between the disk and memory problem, we optimized the twice partitioning with fine-grained GPU scheduling to reduce the 10 overhead between the CPU memory and disk. For low GPU utilization caused by the GPU load imbalance problem, we proposed balanced partitioning and dynamic scheduling methods to accelerate the training speed in different cases. With the above methods, we proposed fine-grained partitioning KGE, an efficient KGE training framework with multiple GPUs. We conducted experiments on some benchmarks of the knowledge graph, and the results show that our method achieves speedup compared to existing framework on the training of KGE.

Multi-Job Intelligent Scheduling With Cross-Device Federated Learning

Recent years have witnessed a large amount of decentralized data in various (edge) devices of end-users, while the decentralized data aggregation remains complicated for machine learning jobs because of regulations and laws. As a practical approach to handling decentralized data, Federated Learning (FL) enables collaborative global machine learning model training without sharing sensitive raw data. The servers schedule devices to jobs within the training process of FL. In contrast, device scheduling with multiple jobs in FL remains a critical and open problem. In this article, we propose a novel multi-job FL framework, which enables the training process of multiple jobs in parallel. The multi-job FL framework is composed of a system model and a scheduling method. The system model enables a parallel training process of multiple jobs, with a cost model based on the data fairness and the training time of diverse devices during the parallel training process. We propose a novel intelligent scheduling approach based on multiple scheduling methods, including an original reinforcement learning-based scheduling method and an original Bayesian optimization-based scheduling method, which corresponds to a small cost while scheduling devices to multiple jobs. We conduct extensive experimentation with diverse jobs and datasets. The experimental results reveal that our proposed approaches significantly outperform baseline approaches in terms of training time (up to 12.73 times faster) and accuracy (up to 46.4% higher).

Real-Time Risk Assessment Detection for Weak People by Parallel Training Logical Execution of a Supervised Learning System Based on an IoT Wearable MEMS Accelerometer.

Activity monitoring has become a necessary demand for weak people to guarantee their safety. The paper proposed a Parallel Training Logical Execution (PTLE) system using machine learning (ML) models on a microelectromechanical system (MEMS) accelerometer to detect coughs, falls, and other normal activities. When there are many categories, the ML prediction can be confused between these activities with each other. The PTLE system trains several models in parallel with more specific activity classes in each dataset. The shared tasks between parallel models relieve the complexity for a single one. There are six additional parameters for accelerometer characteristics, which were calculated from three axes accelerations as input features to improve the ML's consciousness. Once all models were trained, the system was ready to receive the input accelerations and activated the logical flow to manage link operation between these ML models for output predictions. Random Forest (RF) had the highest potential among the ML classification algorithms after the validation. In the experiment, the comparison between the PTLE model and the regular ML model were carried out with real-time data from an M5stickC wearable device on the user's chest to the trained models on PC. The result showed the advancement of the proposed method in term of precision, recall, F1-score with an overall accuracy of 98% in the real-time test. The accelerations from the wearable device were sent to ML models via Wi-Fi with Message Queue Telemetry Transport (MQTT) broker, and the activity predictions were transferred to the cloud for the family members or doctor care based on Internet of Things (IoT) communication.

Features detection of Al alloy porosity during GTAW process based on arc spectrum and improved porosity-focus decision tree

As one of the common internal defects in the aluminum welding process, porosity defect is still challenging for real-time control due to the complexity and diversity of the welding process. Arc spectrum contains large amount of welding process information and is effective for the detection of internal defects. This paper proposed a real-time porosity defect detection method for aluminum alloy in gas tungsten arc welding (GTAW) based on the improved gradient boosting decision tree and arc spectrum. The line and continuum spectrum are separated, and the arc blackbody radiation energy and electron temperature are extracted based on the new spectral separation algorithm. Considering the long-tail distribution characteristic of the training samples, a porosity-focus loss function and a parallel training structure were proposed. More targeted line spectrum features were extracted. The defect recall rate increased by 8.6 %, and the area under the receiver operating characteristic curve (AUC) increased by 5 %, without affecting the detection accuracy and response speed. The confidence level of the prediction results was improved. The model shows better robustness through testing experiments in the face of complex welding conditions and is of great importance to the internal quality monitoring of GTAW.

Characterization of anomalous diffusion through convolutional transformers

The results of the Anomalous Diffusion Challenge (AnDi Challenge) (Muñoz-Gil G et al 2021 Nat. Commun. 12 6253) have shown that machine learning methods can outperform classical statistical methodology at the characterization of anomalous diffusion in both the inference of the anomalous diffusion exponent α associated with each trajectory (Task 1), and the determination of the underlying diffusive regime which produced such trajectories (Task 2). Furthermore, of the five teams that finished in the top three across both tasks of the AnDi Challenge, three of those teams used recurrent neural networks (RNNs). While RNNs, like the long short-term memory network, are effective at learning long-term dependencies in sequential data, their key disadvantage is that they must be trained sequentially. In order to facilitate training with larger data sets, by training in parallel, we propose a new transformer based neural network architecture for the characterization of anomalous diffusion. Our new architecture, the Convolutional Transformer (ConvTransformer) uses a bi-layered convolutional neural network to extract features from our diffusive trajectories that can be thought of as being words in a sentence. These features are then fed to two transformer encoding blocks that perform either regression (Task 1 1D) or classification (Task 2 1D). To our knowledge, this is the first time transformers have been used for characterizing anomalous diffusion. Moreover, this may be the first time that a transformer encoding block has been used with a convolutional neural network and without the need for a transformer decoding block or positional encoding. Apart from being able to train in parallel, we show that the ConvTransformer is able to outperform the previous state of the art at determining the underlying diffusive regime (Task 2 1D) in short trajectories (length 10–50 steps), which are the most important for experimental researchers.

A Communication Efficient ADMM-based Distributed Algorithm Using Two-Dimensional Torus Grouping AllReduce

Large-scale distributed training mainly consists of sub-model parallel training and parameter synchronization. With the expansion of training workers, the efficiency of parameter synchronization will be affected. To tackle this problem, we first propose 2D-TGA, a grouping AllReduce method based on the two-dimensional torus topology. This method synchronizes the model parameters by grouping and makes full use of bandwidth. Secondly, we propose a distributed algorithm, 2D-TGA-ADMM, which combines the 2D-TGA with the alternating direction method of multipliers (ADMM). It focuses on sub-model training and reduces the wait time among workers in the synchronization process. Finally, experimental results on the Tianhe-2 supercomputing platform show that compared with the {mathtt {MPI_Allreduce}}, the 2D-TGA could shorten the synchronization wait time by 33%.

ОПРЕДЕЛЕНИЕ ПРОПУСКНОЙ СПОСОБНОСТИ ПОЛИГОНА ЖЕЛЕЗНОЙ ДОРОГИ ПРИ РАЗЛИЧНОЙ КОНФИГУРАЦИИ СЕТИ

The transition to testing facility technologies for controlling train traffic on the railway network of the Russian Federation requires new principles for calculating classical parameters and characteristics of railway transportation service. This article describes the algorithm developed by the authors for calculating the throughput capacity of testing facilities with an extensive infrastructure, availability of multiple parallel train routes and non-linear connections between the initial point and final destination of a train flow.

Parallel and Memory-Efficient Distributed Edge Learning in B5G IoT Networks

Nowadays we are witnessing rapid development of the Internet of Things (IoT), machine learning, and cellular network technologies. They are key components to promote wireless networks beyond 5G (B5G). The plenty of data generated from numerous IoT devices, such as smart sensors and mobile devices, can be utilised to train intelligent models. But it still remains a challenge to efficiently utilise IoT networks and edge in B5G to conduct model training. In this paper, we propose a parallel training method which uses operators as scheduling units during training task assignment. Besides, we discuss a pebble-game-based memory-efficient optimisation in training. Experiments based on various real world network architectures show the flexibility of our proposed method and good performance compared with state of the art.