Training Bottleneck Research Articles

Efficient Training of Graph Neural Networks on Large Graphs

Graph Neural Networks (GNNs) have gained significant popularity for learning representations of graph-structured data. Mainstream GNNs employ the message passing scheme that iteratively propagates information between connected nodes through edges. However, this scheme incurs high training costs, hindering the applicability of GNNs on large graphs. Recently, the database community has extensively researched effective solutions to facilitate efficient GNN training on massive graphs. In this tutorial, we provide a comprehensive overview of the GNN training process based on the graph data lifecycle, covering graph preprocessing, batch generation, data transfer, and model training stages. We discuss recent data management efforts aiming at accelerating individual stages or improving the overall training efficiency. Recognizing the distinct training issues associated with static and dynamic graphs, we first focus on efficient GNN training on static graphs, followed by an exploration of training GNNs on dynamic graphs. Finally, we suggest some potential research directions in this area. We believe this tutorial is valuable for researchers and practitioners to understand the bottleneck of GNN training and the advanced data management techniques to accelerate the training of different GNNs on massive graphs in diverse hardware settings.

Multi-Objective Evolutionary Neural Architecture Search with Weight-Sharing Supernet

Deep neural networks have played a crucial role in the field of deep learning, achieving significant success in practical applications. The architecture of neural networks is key to their performance. In the past few years, these architectures have been manually designed by experts with rich domain knowledge. Additionally, the optimal neural network architecture can vary depending on specific tasks and data distributions. Neural Architecture Search (NAS) is a class of techniques aimed at automatically searching for and designing neural network architectures according to the given tasks and data. Specifically, evolutionary-computation-based NAS methods are known for their strong global search capability and have aroused widespread interest in recent years. Although evolutionary-computation-based NAS has achieved success in a wide range of research and applications, it still faces bottlenecks in training and evaluating a large number of individuals during optimization. In this study, we first devise a multi-objective evolutionary NAS framework based on a weight-sharing supernet to improve the search efficiency of traditional evolutionary-computation-based NAS. This framework combines the population optimization characteristic of evolutionary algorithms with the weight-sharing ideas in one-shot models. We then design a bi-population MOEA/D algorithm based on the proposed framework to effectively solve the NAS problem. By constructing two sub-populations with different optimization objectives, the algorithm can effectively explore network architectures of various sizes in complex search spaces. An inter-population communication mechanism further enhances the algorithm’s exploratory capability, enabling it to find network architectures with uniform distribution and high diversity. Finally, we conduct performance comparison experiments on image classification datasets of different scales and complexities. Experimental results demonstrate the effectiveness of the proposed multi-objective evolutionary NAS framework and the practicality and transferability of the introduced bi-population MOEA/D-based NAS method compared to existing state-of-the-art NAS methods.

Eliminating Data Processing Bottlenecks in GNN Training over Large Graphs via Two-level Feature Compression

Training GNNs over large graphs faces a severe data processing bottleneck, involving both sampling and feature loading. To tackle this issue, we introduce F 2 CGT, a fast GNN training system incorporating feature compression. To avoid potential accuracy degradation, we propose a two-level, hybrid feature compression approach that applies different compression methods to various graph nodes. This differentiated choice strikes a balance between rounding errors, compression ratios, model accuracy loss, and preprocessing costs. Our theoretical analysis proves that this approach offers convergence and comparable model accuracy as the conventional training without feature compression. Additionally, we also co-design the on-GPU cache sub-system with compression-enabled training within F 2 CGT. The new cache sub-system, driven by a cost model, runs new cache policies to carefully choose graph nodes with high access frequencies, and well partitions the spare GPU memory for various types of graph data, for improving cache hit rates. Finally, extensive evaluation of F 2 CGT on two popular GNN models and four datasets, including three large public datasets, demonstrates that F 2 CGT achieves a compression ratio of up to 128 and provides GNN training speedups of 1.23-2.56× and 3.58--71.46× for single-machine and distributed training, respectively, with up to 32 GPUs and marginal accuracy loss.

SIMPLE: Efficient Temporal Graph Neural Network Training at Scale with Dynamic Data Placement

Dynamic graphs are essential in real-world scenarios like social media and e-commerce for tasks such as predicting links and classifying nodes. Temporal Graph Neural Networks (T-GNNs) stand out as a prime solution for managing dynamic graphs, employing temporal message passing to compute node embeddings at specific timestamps. Nonetheless, the high CPU-GPU data loading overhead has become the bottleneck for efficient training of T-GNNs over large-scale dynamic graphs. In this work, we present SIMPLE, a versatile system designed to address the major efficiency bottleneck in training existing T-GNNs on a large scale. It incorporates a dynamic data placement mechanism, which maintains a small buffer space in available GPU memory and dynamically manages its content during T-GNN training. SIMPLE is also empowered by systematic optimizations towards data processing flow. We compare SIMPLE to the state-of-the-art generic T-GNN training system TGL on four large-scale dynamic graphs with different underlying T-GNN models. Extensive experimental results show that SIMPLE effectively cuts down 80.5% ~ 96.8% data loading cost, and accelerates T-GNN training by 1.8× ~ 3.8× (2.6× on average) compared to TGL.

Bilateral Supervision Network for Semi-Supervised Medical Image Segmentation.

Massive high-quality annotated data is required by fully-supervised learning, which is difficult to obtain for image segmentation since the pixel-level annotation is expensive, especially for medical image segmentation tasks that need domain knowledge. As an alternative solution, semi-supervised learning (SSL) can effectively alleviate the dependence on the annotated samples by leveraging abundant unlabeled samples. Among the SSL methods, mean-teacher (MT) is the most popular one. However, in MT, teacher model's weights are completely determined by student model's weights, which will lead to the training bottleneck at the late training stages. Besides, only pixel-wise consistency is applied for unlabeled data, which ignores the category information and is susceptible to noise. In this paper, we propose a bilateral supervision network with bilateral exponential moving average (bilateral-EMA), named BSNet to overcome these issues. On the one hand, both the student and teacher models are trained on labeled data, and then their weights are updated with the bilateral-EMA, and thus the two models can learn from each other. On the other hand, pseudo labels are used to perform bilateral supervision for unlabeled data. Moreover, for enhancing the supervision, we adopt adversarial learning to enforce the network generate more reliable pseudo labels for unlabeled data. We conduct extensive experiments on three datasets to evaluate the proposed BSNet, and results show that BSNet can improve the semi-supervised segmentation performance by a large margin and surpass other state-of-the-art SSL methods.

Computation and Communication Efficient Federated Learning With Adaptive Model Pruning

Federated learning (FL) has emerged as a promising distributed learning paradigm that enables a large number of mobile devices to cooperatively train a model without sharing their raw data. The iterative training process of FL incurs considerable computation and communication overhead. The workers participating in FL are usually heterogeneous and the workers with poor capabilities may become the bottleneck of model training. To address the challenges of resource overhead and system heterogeneity, this paper proposes an efficient FL framework, called FedMP, that improves both computation and communication efficiency over heterogeneous workers through adaptive model pruning. We theoretically analyze the impact of pruning ratio on training performance, and employ a Multi-Armed Bandit based online learning algorithm to adaptively determine different pruning ratios for heterogeneous workers, even without any prior knowledge of their capabilities. As a result, each worker in FedMP can train and transmit the sub-model that fits its own capabilities, accelerating the training process without hurting model accuracy. To prevent the diverse structures of pruned models from affecting the training convergence, we further present a new parameter synchronization scheme, called Residual Recovery Synchronous Parallel (R2SP). Besides, our proposed framework can be extended to the peer-to-peer (P2P) setting. Extensive experiments on physical devices demonstrate that FedMP is effective for different heterogeneous scenarios and data distributions, and can provide up to 4.1× speedup compared to the existing FL methods.

Uncorking the bottleneck in anaesthesia training: novel approaches to a growing crisis

Despite the growing demand for highly qualified anaesthetists, the lack of training posts in the UK is preventing many trainees from progressing. Anaesthetic specialist trainee Adam Hunt and consultant David Walker outline the reasons behind this issue and suggest novel approaches to uncorking the training bottleneck.

Overcoming training bottlenecks: mixed-methods evaluation of digital training for non-specialists in postnatal depression self-help treatment

ABSTRACT Objectives This mixed-methods study evaluated the feasibility, acceptability and preliminary effectiveness of an interactive digital training programme for non-specialist supporters providing a guided self-help intervention for postnatal depression (PND). Methods A total of 49 non-specialist trainees participated. Six digital training modules were flexibly delivered over a 5-week period. Training included a chatroom, moderated by a supervised assistant psychologist. Quantitatively, feasibility was assessed via participation and retention levels; acceptability was examined using course evaluation questionnaires; and effectiveness was measured pre-test-post-test quantitatively using a self-report questionnaire and pre-post using scenario questions. Participant focus groups explored feasibility, acceptability and effectiveness. Results The training was feasible; 41 completed the course and 42 were assessed at follow-up. Quantitative course evaluation and thematic analysis of focus group feedback demonstrated high training acceptability. RANOVAs indicated training significantly improved knowledge and confidence pre- to post-test. There were demonstrable increases in specific skills at post-test as assessed via clinical scenarios. Conclusion This training is a feasible, acceptable and effective way to upskill non-specialists in supporting treatment for PND, however supervised practice is recommended to ensure participants embed knowledge competently into practice. The training offers an effective first step in upskilling non-specialist supporters to support women with PND treatment at scale.

ALBSNN: ultra-low latency adaptive local binary spiking neural network with accuracy loss estimator.

Spiking neural network (SNN) is a brain-inspired model with more spatio-temporal information processing capacity and computational energy efficiency. However, with the increasing depth of SNNs, the memory problem caused by the weights of SNNs has gradually attracted attention. In this study, we propose an ultra-low latency adaptive local binary spiking neural network (ALBSNN) with accuracy loss estimators, which dynamically selects the network layers to be binarized to ensure a balance between quantization degree and classification accuracy by evaluating the error caused by the binarized weights during the network learning process. At the same time, to accelerate the training speed of the network, the global average pooling (GAP) layer is introduced to replace the fully connected layers by combining convolution and pooling. Finally, to further reduce the error caused by the binary weight, we propose binary weight optimization (BWO), which updates the overall weight by directly adjusting the binary weight. This method further reduces the loss of the network that reaches the training bottleneck. The combination of the above methods balances the network's quantization and recognition ability, enabling the network to maintain the recognition capability equivalent to the full precision network and reduce the storage space by more than 20%. So, SNNs can use a small number of time steps to obtain better recognition accuracy. In the extreme case of using only a one-time step, we still can achieve 93.39, 92.12, and 69.55% testing accuracy on three traditional static datasets, Fashion- MNIST, CIFAR-10, and CIFAR-100, respectively. At the same time, we evaluate our method on neuromorphic N-MNIST, CIFAR10-DVS, and IBM DVS128 Gesture datasets and achieve advanced accuracy in SNN with binary weights. Our network has greater advantages in terms of storage resources and training time.

The growing bottlenecks in specialty training

The growing bottlenecks in specialty training

Beyond Training the Next Generation of Physicians: The Unmeasured Value Added by Residents to Teaching Hospitals and Communities.

Following medical school, most newly graduated physicians enter residency training. This period of graduate medical education (GME) is critical to creating a physician workforce with the specialized skills needed to care for the population. Completing GME training is also a requirement for obtaining medical licensure in all 50 states. Yet, crucial federal and state funding for GME is capped, creating a bottleneck in training an adequate physician workforce to meet future patient care needs. Thus, additional GME funding is needed to train more physicians.When considering this additional GME funding, it is imperative to take into account not only the future physician workforce but also the value added by residents to teaching hospitals and communities during their training. Residents positively affect patient care and health care delivery, providing intrinsic and often unmeasured value to patients, the hospital, the local community, the research enterprise, and undergraduate medical education. This added value is often overlooked in decisions regarding GME funding allocation. In this article, the authors underscore the value provided by residents to their training institutions and communities, with a focus on current and recent events, including the global COVID-19 pandemic and teaching hospital closures.

AFedAvg: communication-efficient federated learning aggregation with adaptive communication frequency and gradient sparse

ABSTRACTFederated learning enables a large number of clients (such as edge computing devices) to learn a model jointly without data sharing. However, the high amount of communication of the federated learning aggregation algorithm hinders the realisation of artificial intelligence in the last mile. Although FederatedAveraging (FedAvg) is the leading algorithm, its communication cost is still high. The method of communication delay and gradient sparse can reduce the communication cost, but there is no previous work to analyse the relationship and common effects of these two dimensions. Aiming at the problems that federated learning communication is expensive and it has become a training bottleneck, we improve the FedAvg algorithm and propose an adaptive communication frequency FederatedAveraging algorithm (AFedAvg). The gradient sparse operation in the algorithm reduces the quantity of parameters for a single communication, while the communication delay operation allows training to converge faster and obtain smaller losses. The number of sparse parameters is used to select the communication frequency of next round dynamically. Experimental results prove that, the AFedAvg algorithm is superior to the FedAvg and its variants in terms of communication cost. It achieves 2.4X–23.1X communication compression in different data distributions with minimal communication rounds required by the algorithm to converge.

DisCOV: Distributed COVID-19 Detection on X-Ray Images With Edge-Cloud Collaboration

Currently, the world is experiencing the rapid spread of Coronavirus Disease 2019 (COVID-19). Since the epidemic continues to take a devastating impact on the society, economy, and healthcare, the real-time detection of COVID-19 is essential for fast and cost-effective diagnosis services. Fortunately, deep learning (DL), as a promising technology, enables the COVID-19 diagnosis services on chest X-ray (CXR) images. The training task of DL model is generally implemented at the centralized cloud. However, due to the geo-distributed data sources and the transmission of large amounts of raw data to the centralized cloud, the transmission latency becomes a bottleneck of the COVID-19 diagnosis model training. In this paper, we propose a <u>Dis</u>tributed <u>COV</u>ID-19 detection model training method on CXR images with edge-cloud collaboration, named DisCOV. Specifically, to improve the training efficiency and guarantee the model accuracy, a distributed lightweight model-based training algorithm is designed with the cooperation of edge computing and cloud computing. In addition, a resource allocation algorithm is developed during the training to jointly minimize the time cost and energy consumption. Extensive experiments based on real-world CXR image datasets demonstrate that DisCOV is better performed and more promising than the existing baselines.

A Bayesian Domain Adversarial Neural Network for Corn Yield Prediction

Corn is the most widely grown crop in the U.S. and makes up a significant part of the American diet. Under the pressure of feeding a growing population, accurate and timely estimation of corn yield before the harvest is of great importance to supply chain management and regional food security. Recently, machine learning in conjunction with satellite remote sensing has been used for developing corn yield prediction models. Despite the success, a major bottleneck of training a reliable supervised machine learning model is the need for representative ground truth labels (e.g., yield records) which may be limited or even not available due to financial and manpower reasons. Also, due to domain shift, a machine learning model trained with labeled data from a label-rich region (i.e., source domain) could experience a significant performance decrease when directly applied to the region of interest (i.e., target domain). To address this issue, we proposed a Bayesian Domain Adversarial Neural Network (BDANN) for unsupervised domain adaptation on county-level corn yield prediction. By applying adversarial learning and Bayesian inference, BDANN was trained to reduce domain shift and accurately predict corn yield by extracting domain-invariant and task-informative features from both source and target domains. Moreover, the results also demonstrated that the BDANN model generalized well on small training sets. Experiments in two ecoregions in the U.S. corn belt have shown the effectiveness of the proposed BDANN and its superiority over other state-of-the-art methods.

SmaQ: Smart Quantization for DNN Training by Exploiting Value Clustering

Advancements in modern deep learning have shown that deeper networks with larger datasets can achieve state of the art results in many different tasks. As networks become deeper, the memory requirement of neural network training proves to be the primary bottleneck of single-machine training. In this letter, we first study the characteristics of neural network weight, gradient, feature map, gradient map, and optimizer state distributions for some popular neural network architectures. Our investigation shows that the majority of the data structures used by neural networks can have their value distributions be approximated with normal distributions. We then introduce Smart Quantization (SmaQ), a quantization scheme that exploits this observed normal distribution to quantize the data structures. Our dynamic quantization method calculates the sampled mean and standard deviation of tensors and quantizes each tensor element to 6 or 8 bits based on the z-score of that value. Our scheme reduces the memory usage during training by up to 6.7x with minor losses in accuracy.

Efficient Communication Scheduling for Parameter Synchronization of DML in Data Center Networks

It's common practice to speed up machine learning (ML) training by distributing it across a cluster of computing nodes. Data-parallel distributed ML (DML) training relieves the pressure of computing node; however, the communication traffic introduced during parameter synchronization becomes bottleneck of DML training. Regarding communication bottleneck, we identify two primary causes: high contention of concurrent communication and large volume of redundant transmission in the push and pull stage while synchronizing parameters. To address these issues, we propose a novel Group Stale Synchronous Parallel (GSSP) scheme, which divides the nodes into groups and coordinates groups to synchronize in a circular order. GSSP mitigates network contention and is proven to converge. We provide analysis of the optimal group's number based on bandwidth and buffer size. For reducing traffic redundancy, we propose a multicast-based scheme, which generates multicast trees by minimizing links overlap and allocates transmission rate to multicast flows by solving the min-max optimization problem. Finally, we conduct extensive simulations to evaluate the performance of our proposals. We simulate parameter transmission of All-Reduce and parameter server in Fat-Tree with traffics trace of ML models. Simulation results show that our proposals provide communication-efficiency for DML training by mitigating contention and reducing redundancy.

Empirical analysis of performance bottlenecks in graph neural network training and inference with GPUs

The graph neural network (GNN) has become a popular research area for its state-of-the-art performance in many graph analysis tasks. Recently, various graph neural network libraries have emerged. They make developing GNNs convenient, but the performance bottlenecks of GNNs on large datasets are not well studied. In this work, we analyze the performance bottlenecks in GNN training and inference with GPUs empirically. A GNN layer can be decomposed into two parts: the vertex and the edge calculation parts. We select four representative GNNs (GCN, GGNN, GAT, GaAN) for evaluation according to their computational complexity. We decompose their running time and memory usage, evaluate the effects of hyper-parameters and assess the efficiency of the sampling techniques. The experimental evaluation with PyTorch Geometric indicates that the edge-related calculation is the performance bottleneck for most GNNs, dominating the training/inference time and memory usage. The sampling techniques are essential for GNN training and inference on big graphs with GPUs, but their current implementation still has non-trivial overheads in sampling and data transferring.

Understanding the drivers of bottlenecks in RANZCP training: modelling and a calculator to determining sustainable trainee intake.

Demand for places in postgraduate psychiatry training programmes has increased over recent years. All systems have capacity limits, and concerns have been raised regarding the sustainability of the current intake. This paper presents a modelling exercise to exploring the presence and strategies to resolve bottleneck in the Queensland training programme. Mathematical modelling based on the RANZCP training regulations and the characteristics of the accredited training programme. A training bottleneck was identified which has been impacted by increased training intake, demand for Advanced Training certificates, and location factors. This investigation raises important questions regarding the future management of postgraduate training in psychiatry in Queensland that may be applicable more widely across Australia and New Zealand. In particular, it highlights the large impact that can result from even small incremental increases in trainees across varying levels of the postgraduate programme and the importance of limiting trainee intake in a manner proportional to the availability of mandatory terms.

On the Looming Physician Shortage and Strategic Expansion of Graduate Medical Education.

Among many other things, the novel coronavirus pandemic of 2020 highlighted the significance of physician shortages in the United States. Current projections anticipate a national shortage of up to 122,000 physicians by 2032, with shortfalls in both primary care physicians and specialists. Yet while this figure highlights the magnitude of the problem, it does not capture the distributional aspect of American physician shortages. Though some specialties and geographic areas have a surplus of physicians, others have a chronic undersupply. Appropriately addressing the looming physician shortage therefore requires not only creating more physicians, but also ensuring that those physicians practice in the areas of greatest societal need. This review explores the nature of physician shortages in the United States, identifies the present bottleneck in physician training at the level of graduate medical education, and considers potential legislative and policy solutions to allow strategic and deliberate expansion of graduate medical education and physician practice.

MOSS: End-to-End Dialog System Framework with Modular Supervision

A major bottleneck in training end-to-end task-oriented dialog system is the lack of data. To utilize limited training data more efficiently, we propose Modular Supervision Network (MOSS), an encoder-decoder training framework that could incorporate supervision from various intermediate dialog system modules including natural language understanding, dialog state tracking, dialog policy learning and natural language generation. With only 60% of the training data, MOSS-all (i.e., MOSS with supervision from all four dialog modules) outperforms state-of-the-art models on CamRest676. Moreover, introducing modular supervision has even bigger benefits when the dialog task has a more complex dialog state and action space. With only 40% of the training data, MOSS-all outperforms the state-of-the-art model on a complex laptop network trouble shooting dataset, LaptopNetwork, that we introduced. LaptopNetwork consists of conversations between real customers and customer service agents in Chinese. Moreover, MOSS framework can accommodate dialogs that have supervision from different dialog modules at both framework level and model level. Therefore, MOSS is extremely flexible to update in real-world deployment.