GPU Memory Research Articles

Optimizing GPU-Based Graph Sampling and Random Walk for Efficiency and Scalability

Graph sampling and random walk algorithms are playing increasingly important roles today because they can significantly reduce graph size while preserving structural information, thus enabling computationally intensive tasks on large-scale graphs. Current frameworks designed for graph sampling and random walk tasks are generally not efficient in terms of memory requirement and throughput. Not to mention that some of them result in biased results. To solve the above problems, we introduce Skywalker+, a high-performance graph sampling and random walk framework on multiple GPUs supporting multiple algorithms. Skywalker+ makes four key contributions: First, it realizes highly paralleled alias method on GPUs. Second, it applies finely adjusted workload-balancing techniques and locality-aware execution modes to present a highly efficient execution engine. Third, it optimizes the GPU memory usage with efficient buffering and data compression schemes. Last, it scales to multi-GPU to further enhance the system throughput. Abundant experiments show that Skywalker+ exhibits significant advantage over the baselines both in performance and utility.

Enhancing Biomedical ReQA With Adversarial Hard In-Batch Negative Samples.

Question answering (QA) plays a vital role in biomedical natural language processing. Among question answering tasks, the retrieval question answering (ReQA) aims to directly retrieve the correct answer from candidates and has attracted much attention in the community for its efficiency. Recently, researchers have introduced ReQA into the biomedical domain as BioReQA. Typically BioReQA models rely on the dual-encoder to gain semantic representation and are trained following the settings of dense retrieval. However, they normally utilize easy in-batch negative samples in training process to avoid the extra forwarding cost and GPU memory required by encoding additional negative samples. However, hard negative samples have been proved more important with regard to the overall performance of BioReQA tasks. Therefore in this research, we focus on effectively constructing hard in-batch negative samples. Inspired by the classic linear assignment problem, we propose an Iterative Linear Assignment Grouping (ILAG) algorithm to construct hard in-batch negative samples. To further enhance performance for given hard batches in a low-resource scenario, we also employ adversarial training to augment the difficulty of batches. Extensive experiments have shown our proposed method's promising potential in the area of biomedical retrieval question answering.

Runtime Construction of Large-Scale Spiking Neuronal Network Models on GPU Devices

Simulation speed matters for neuroscientific research: this includes not only how quickly the simulated model time of a large-scale spiking neuronal network progresses but also how long it takes to instantiate the network model in computer memory. On the hardware side, acceleration via highly parallel GPUs is being increasingly utilized. On the software side, code generation approaches ensure highly optimized code at the expense of repeated code regeneration and recompilation after modifications to the network model. Aiming for a greater flexibility with respect to iterative model changes, here we propose a new method for creating network connections interactively, dynamically, and directly in GPU memory through a set of commonly used high-level connection rules. We validate the simulation performance with both consumer and data center GPUs on two neuroscientifically relevant models: a cortical microcircuit of about 77,000 leaky-integrate-and-fire neuron models and 300 million static synapses, and a two-population network recurrently connected using a variety of connection rules. With our proposed ad hoc network instantiation, both network construction and simulation times are comparable or shorter than those obtained with other state-of-the-art simulation technologies while still meeting the flexibility demands of explorative network modeling.

Discovering the mechanism of action of drugs with a sparse explainable network

Although Deep Neural Networks (DDNs) have been successful in predicting the efficacy of cancer drugs, the lack of explainability in their decision-making process is a significant challenge. Previous research proposed mimicking the Gene Ontology structure to allow for interpretation of each neuron in the network. However, these previous approaches require huge amount of GPU resources and hinder its extension to genome-wide models. We developed SparseGO, a sparse and interpretable neural network, for predicting drug response in cancer cell lines and their Mechanism of Action (MoA). To ensure model generalization, we trained it on multiple datasets and evaluated its performance using three cross-validation schemes. Its efficiency allows it to be used with gene expression. In addition, SparseGO integrates an eXplainable Artificial Intelligence (XAI) technique, DeepLIFT, with Support Vector Machines to computationally discover the MoA of drugs. SparseGO's sparse implementation significantly reduced GPU memory usage and training speed compared to other methods, allowing it to process gene expression instead of mutations as input data. SparseGO using expression improved the accuracy and enabled its use on drug repositioning. Furthermore, gene expression allows the prediction of MoA using 265 drugs to train it. It was validated on understudied drugs such as parbendazole and PD153035. SparseGO is an effective XAI method for predicting, but more importantly, understanding drug response. The Accelerator Award Programme funded by Cancer Research UK [C355/A26819], Fundación Científica de la AECC and Fondazione AIRC, Project PIBA_2020_1_0055 funded by the Basque Government and the Synlethal Project (RETOS Investigacion, Spanish Government).

Hydrodynamic numerical simulations based on residual cooperative neural network

As a gridless computational method, physics-informed neural networks (PINNs) have been widely applied and are a research hotspot in the field of numerical simulation due to their potential to provide approximations that are infinitely close to the analytical solutions. To address the complex hydrodynamic simulation scenarios with a large solution domain, where the conventional PINNs are not fully applicable, we propose a residual cooperative neural network (RCNN) embedded with hard constraints and optimized by the automatic mixed-precision (AMP) computational technique. To perform extreme condition hydrodynamic simulations, we provide smooth surrogate functions for each of the point-source and step conditions. The applicability and advantages of the RCNN model are verified with a total of four typical hydrodynamic cases. The results show that the RCNN performs better than the conventional PINN in terms of convergence, applicability, and fitting accuracy, with an overall goodness-of-fit of over 0.99. The computational cost of the RCNN model can be significantly reduced by the AMP technique, with the average computation time and occupied GPU memory reduced by 45.77% and 43.16%, respectively. The RCNN can effectively address the gradient explosion and gradient disappearance problems in complex hydrodynamic simulations with a large solution domain.

Optimizing Single DGX-A100 System: Overcoming GPU Limitations via Efficient Parallelism and Scheduling for Large Language Models

In this study, we introduce a novel training algorithm specifically designed to overcome the limitations of GPU memory on a single DGX-A100 system. By utilizing the CPU and main memory in the training process and applying a strategy of division and parallelization, our algorithm enhances the size of the trainable language model and the batch size. In addition, we developed a comprehensive management system to effectively manage the execution of the algorithm. This system systematically controls the training process and resource usage, while also enabling the asynchronous deployment of tasks. Finally, we proposed a scheduling technique integrated into the management system, promoting efficient task scheduling in a complex, heterogeneous training environment. These advancements equip researchers with the ability to work with larger models and batch sizes, even when faced with limited GPU memory.

STR: Hybrid Tensor Re-Generation to Break Memory Wall for DNN Training

With the growth of the depth of neural networks and the scale of data, the difficulty of network training also increases. When the GPU memory is insufficient, it is challenging to train deeper models. Recent research uses tensor swapping and recomputation techniques in a combined manner to optimize memory usage. However, complex dependencies and enormous scales of the DNN graph limit the improvement of single GPU memory optimization. Improper swap and recomputation decisions even bring negative effects on training performance. In this paper, we propose a novel hybrid tensor re-generation strategy, called STR, which combines swap and recomputation techniques to find the optimal execution plan for the DNN training when the memory is limited. We formalize our memory optimization problem with constraints that describe the dependency of the operator calculation and the bandwidth usage of the swap. A <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">host checkpoint</i> mechanism is designed to make full use of the swapped tensors, which reduces the cost of the recomputation. We also present a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">recursive source tracing</i> algorithm to improve the optimization efficiency by constraint relaxation with a performance bound. To optimize large models, we further introduce an approximation method based on a weighted graph coarsening. We implement a prototype of STR as a plugin on TensorFlow and evaluated based on 5 popular DNN models. The experimental result shows that the approximate solution of STR improves the training throughput of ResNet series of models by up to 28.1% compared to the state-of-the-art hybrid optimization strategy.

Self-Distilled Hierarchical Network for Unsupervised Deformable Image Registration

Unsupervised deformable image registration benefits from progressive network structures such as Pyramid and Cascade. However, existing progressive networks only consider the single-scale deformation field in each level or stage and ignore the long-term connection across non-adjacent levels or stages. In this paper, we present a novel unsupervised learning approach named Self-Distilled Hierarchical Network (SDHNet). By decomposing the registration procedure into several iterations, SDHNet generates hierarchical deformation fields (HDFs) simultaneously in each iteration and connects different iterations utilizing the learned hidden state. Specifically, hierarchical features are extracted to generate HDFs through several parallel gated recurrent units, and HDFs are then fused adaptively conditioned on themselves as well as contextual features from the input image. Furthermore, different from common unsupervised methods that only apply similarity loss and regularization loss, SDHNet introduces a novel self-deformation distillation scheme. This scheme distills the final deformation field as the teacher guidance, which adds constraints for intermediate deformation fields on deformation-value and deformation-gradient spaces respectively. Experiments on five benchmark datasets, including brain MRI and liver CT, demonstrate the superior performance of SDHNet over state-of-the-art methods with a faster inference speed and a smaller GPU memory. Code is available at https://github.com/Blcony/SDHNet.

Efficient Latent Space Compression for Lightning-Fast Fine-Tuning and Inference of Transformer-Based Models

This paper presents a technique to reduce the number of parameters in a transformer-based encoder–decoder architecture by incorporating autoencoders. To discover the optimal compression, we trained different autoencoders on the embedding space (encoder’s output) of several pre-trained models. The experiments reveal that reducing the embedding size has the potential to dramatically decrease the GPU memory usage while speeding up the inference process. The proposed architecture was included in the BART model and tested for summarization, translation, and classification tasks. The summarization results show that a 60% decoder size reduction (from 96 M to 40 M parameters) will make the inference twice as fast and use less than half of GPU memory during fine-tuning process with only a 4.5% drop in R-1 score. The same trend is visible for translation and partially for classification tasks. Our approach reduces the GPU memory usage and processing time of large-scale sequence-to-sequence models for fine-tuning and inference. The implementation and checkpoints are available on GitHub.

Lite‐weight semantic segmentation with AG self‐attention

AbstractDue to the large computational and GPUs memory cost of semantic segmentation, some works focus on designing a lite weight model to achieve a good trade‐off between computational cost and accuracy. A common method is to combined CNN and vision transformer. However, these methods ignore the contextual information of multi receptive fields. And existing methods often fail to inject detailed information losses in the downsampling of multi‐scale feature. To fix these issues, we propose AG Self‐Attention, which is Enhanced Atrous Self‐Attention (EASA), and Gate Attention. AG Self‐Attention adds the contextual information of multi receptive fields into the global semantic feature. Specifically, the Enhanced Atrous Self‐Attention uses weight shared atrous convolution with different atrous rates to get the contextual information under the specific different receptive fields. Gate Attention introduces gating mechanism to inject detailed information into the global semantic feature and filter detailed information by producing “fusion” gate and “update” gate. In order to prove our insight. We conduct numerous experiments in common semantic segmentation datasets, consisting of ADE20 K, COCO‐stuff, PASCAL Context, Cityscapes, to show that our method achieves state‐of‐the‐art performance and achieve a good trade‐off between computational cost and accuracy.

AC2AS: Activation Consistency Coupled ANN-SNN framework for fast and memory-efficient SNN training

Spiking neural networks are efficient computation models for low-power environments. Spike-based BP algorithms and ANN-to-SNN (ANN2SNN) conversions are successful techniques for SNN training. Nevertheless, the spike-base BP training is slow and requires large memory costs, while ANN2SNN needs many inference steps to obtain good performance. In this paper, we propose an Activation Consistency Coupled ANN-SNN (AC2AS) framework to train the SNN in a fast and memory-efficient way. The AC2AS consists of two components: (a) a weight-shared architecture between ANN and SNN and (b) spiking mapping units. Firstly, the architecture trains the weight-shared parameters on the ANN branch, resulting in fast training and low memory costs for SNN. Secondly, the spiking mapping units are designed to ensure that the activation values of the ANN are the spiking features. As a result, the activation consistency is guaranteed, and the classification error of the SNN can be optimized by training the ANN branch. Besides, we design an adaptive threshold adjustment (ATA) algorithm to decrease the firing of noisy spikes. Experiment results show that our AC2AS-based models perform well on the benchmark datasets (CIFAR10, CIFAR100, and Tiny-ImageNet). Moreover, the AC2AS achieves comparable accuracy under 0.625× time steps, 0.377× training time, 0.27× GPU memory costs, and 0.33× spike activities of the Spike-based BP model. The code is available at https://github.com/TJXTT/AC2ASNN.

AMP-EBiLSTM: employing novel deep learning strategies for the accurate prediction of antimicrobial peptides.

Antimicrobial peptides are present ubiquitously in intra- and extra-biological environments and display considerable antibacterial and antifungal activities. Clinically, it has shown good antibacterial effect in the treatment of diabetic foot and its complications. However, the discovery and screening of antimicrobial peptides primarily rely on wet lab experiments, which are inefficient. This study endeavors to create a precise and efficient method of predicting antimicrobial peptides by incorporating novel machine learning technologies. We proposed a deep learning strategy named AMP-EBiLSTM to accurately predict them, and compared its performance with ensemble learning and baseline models. We utilized Binary Profile Feature (BPF) and Pseudo Amino Acid Composition (PSEAAC) for effective local sequence capture and amino acid information extraction, respectively, in deep learning and ensemble learning. Each model was cross-validated and externally tested independently. The results demonstrate that the Enhanced Bi-directional Long Short-Term Memory (EBiLSTM) deep learning model outperformed others with an accuracy of 92.39% and AUC value of 0.9771 on the test set. On the other hand, the ensemble learning models demonstrated cost-effectiveness in terms of training time on a T4 server equipped with 16GB of GPU memory and 8 vCPUs, with training durations varying from 0 to 30s. Therefore, the strategy we propose is expected to predict antimicrobial peptides more accurately in the future.

NTPP-MVSNet: Multi-View Stereo Network Based on Neighboring Tangent Plane Propagation

Although learning-based multi-view stereo algorithms have produced exciting results in recent years, few researchers have explored the specific role of deep sampling in the network. We posit that depth sampling accuracy more directly impacts the quality of scene reconstruction. To address this issue, we proposed NTPP-MVSNet, which utilizes normal vector and depth information from neighboring pixels to propagate tangent planes. Based on this, we obtained a more accurate depth estimate through homography transformation. We used deformable convolution to acquire continuous pixel positions on the surface and 3D-UNet to account for the regression of depth and normal vector maps without consuming additional GPU memory. Finally, we applied homography transformation to complete the mapping of the imaging plane and the neighborhood surface tangent plane to generate a depth hypothesis. Experimental trials on the DTU and Tanks and Temples datasets demonstrate the feasibility of NTPP-MVSNet, and ablation experiments confirm the superior performance of our deep sampling methodology.

An Alternate GPU-Accelerated Algorithm for Very Large Sparse LU Factorization

The LU factorization of very large sparse matrices requires a significant amount of computing resources, including memory and broadband communication. A hybrid MPI + OpenMP + CUDA algorithm named SuperLU3D can efficiently compute the LU factorization with GPU acceleration. However, this algorithm faces difficulties when dealing with very large sparse matrices with limited GPU resources. Factorizing very large matrices involves a vast amount of nonblocking communication between processes, often leading to a break in SuperLU3D calculation due to the overflow of cluster communication buffers. In this paper, we present an improved GPU-accelerated algorithm named SuperLU3D_Alternate for the LU factorization of very large sparse matrices with fewer GPU resources. The basic idea is “divide and conquer”, which means dividing a very large matrix into multiple submatrices, performing LU factorization on each submatrix, and then assembling the factorized results of all submatrices into two complete matrices L and U. In detail, according to the number of available GPUs, a very large matrix is first divided into multiple submatrices using the elimination tree. Then, the LU factorization of each submatrix is alternately computed with limited GPU resources, and its intermediate LU factors from GPUs are saved to the host memory or hard disk. Finally, after finishing the LU factorization of all submatrices, these factorized submatrices are assembled into a complete lower triangular matrix L and a complete upper triangular matrix U, respectively. The SuperLU3D_Alternate algorithm is suitable for hybrid CPU/GPU cluster systems, especially for a subset of nodes without GPUs. To accommodate different hardware resources in various clusters, we designed the algorithm to run in the following three cases: sufficient memory for GPU nodes, insufficient memory for GPU nodes, and insufficient memory for the entire cluster. The results from LU factorization test on different matrices in various cases show that the larger the matrix is, the more efficient this algorithm is under the same GPU memory consumption. In our numerical experiments, SuperLU3D_Alternate achieves speeds of up to 8× that of SuperLU3D (CPU only) and 2.5× that of SuperLU3D (CPU + GPU) on the hybrid cluster with six Tesla V100S GPUs. Furthermore, when the matrix is too big to be handled by SuperLU3D, SuperLU3D_Alternate can still utilize the cluster’s host memory or hard disk to solve it. By reducing the amount of data exchange to prevent exceeding the buffer’s limit of the cluster MPI nonblocking communication, our algorithm enhances the stability of the program.

An Improved Combination of Faster R-CNN and U-Net Network for Accurate Multi-Modality Whole Heart Segmentation.

Detailed information of substructures of the whole heart is usually vital in the diagnosis of cardiovascular diseases and in 3D modeling of the heart. Deep convolutional neural networks have been demonstrated to achieve state-ofthe-art performance in 3D cardiac structures segmentation. However, when dealing with high-resolution 3D data, current methods employing tiling strategies usually degrade segmentation performances due to GPU memory constraints. This work develops a two-stage multi-modality whole heart segmentation strategy, which adopts an improved Combination of Faster R-CNN and 3D U-Net (CFUN+). More specifically, the bounding box of the heart is first detected by Faster R-CNN, and then the original Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) images of the heart aligned with the bounding box are input into 3D U-Net for segmentation. The proposed CFUN+ method redefines the bounding box loss function by replacing the previous Intersection over Union (IoU) loss with Complete Intersection over Union (CIoU) loss. Meanwhile, the integration of the edge loss makes the segmentation results more accurate, and also improves the convergence speed. The proposed method achieves an average Dice score of 91.1% on the Multi-Modality Whole Heart Segmentation (MM-WHS) 2017 challenge CT dataset, which is 5.2% higher than the baseline CFUN model, and achieves state-of-the-art segmentation results. In addition, the segmentation speed of a single heart has been dramatically improved from a few minutes to less than 6 seconds.

Computationally efficient GPU based NS solver for two dimensional high-speed inviscid and viscous compressible flows

In this study, we proposed a novel GPU-based solution for modelling two-dimensional inviscid and viscous compressible supersonic/hypersonic flows. Texture and surface pointers are used to access GPU memory locations. For effective and efficient use of surface pointers, we grouped multiple 2D arrays referenced and indexed by a single 3D surface pointer. To enable the proposed solver for double-precision calculations, two consecutive 32-bit memory locations were grouped to maintain the efficiency of surface pointers while taking advantage/accuracy of 64-bit calculations. Resolving data and computation dependencies for parallel applications is another complex task that is the focus of this study. Computation dependencies have been solved by using multiple mutually synchronized GPU kernels and executing them sequentially using the GPU default stream to ensure that all relevant data is available to the threads or computed before they actually use it. Consequently, there is no intra-core data dependency in our proposed approach, while inter-core data dependency is successfully solved by stringing multiple kernels together. Using NVIDIA GTX 660 GPUs, we achieved 20x speedup compared to traditional Core i5® computers. This speedup is the result of the Surface Pointer's GPU capabilities for double precision computations. The simulation results are also consistent with the experimental and numerical results of this study.

NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension

One-shot neural architecture search (NAS) substantially improves the search efficiency by training one supernet to estimate the performance of every possible child architecture (i.e., subnet). However, the inconsistency of characteristics among subnets incurs serious interference in the optimization, resulting in poor performance ranking correlation of subnets. Subsequent explorations decompose supernet weights via a particular criterion, e.g., gradient matching, to reduce the interference; yet they suffer from huge computational cost and low space separability. In this work, we propose a lightweight and effective local intrinsic dimension (LID)-based method NAS-LID. NAS-LID evaluates the geometrical properties of architectures by calculating the low-cost LID features layer-by-layer, and the similarity characterized by LID enjoys better separability compared with gradients, which thus effectively reduces the interference among subnets. Extensive experiments on NASBench-201 indicate that NAS-LID achieves superior performance with better efficiency. Specifically, compared to the gradient-driven method, NAS-LID can save up to 86% of GPU memory overhead when searching on NASBench-201. We also demonstrate the effectiveness of NAS-LID on ProxylessNAS and OFA spaces. Source code:https://github.com/marsggbo/NAS-LID.

GAAMA 2.0: An Integrated System That Answers Boolean and Extractive Questions

Recent machine reading comprehension datasets include extractive and boolean questions but current approaches do not offer integrated support for answering both question types. We present a front-end demo to a multilingual machine reading comprehension system that handles boolean and extractive questions. It provides a yes/no answer and highlights the supporting evidence for boolean questions. It provides an answer for extractive questions and highlights the answer in the passage. Our system, GAAMA 2.0, achieved first place on the TyDi QA leaderboard at the time of submission. We contrast two different implementations of our approach: including multiple transformer models for easy deployment, and a shared transformer model utilizing adapters to reduce GPU memory footprint for a resource-constrained environment.

Rethinking Disparity: A Depth Range Free Multi-View Stereo Based on Disparity

Existing learning-based multi-view stereo (MVS) methods rely on the depth range to build the 3D cost volume and may fail when the range is too large or unreliable. To address this problem, we propose a disparity-based MVS method based on the epipolar disparity flow (E-flow), called DispMVS, which infers the depth information from the pixel movement between two views. The core of DispMVS is to construct a 2D cost volume on the image plane along the epipolar line between each pair (between the reference image and several source images) for pixel matching and fuse uncountable depths triangulated from each pair by multi-view geometry to ensure multi-view consistency. To be robust, DispMVS starts from a randomly initialized depth map and iteratively refines the depth map with the help of the coarse-to-fine strategy. Experiments on DTUMVS and Tanks\&amp;Temple datasets show that DispMVS is not sensitive to the depth range and achieves state-of-the-art results with lower GPU memory.

Kidney Segmentations Using CNN models

For medical diagnostic tests, kidney segmentation from high-volume imagery is an important major. Since 3D medical images need a lot of GPU memory, slices and patches are used for training and inference in traditional neural network variant architectures, which necessarily slows down contextual learning. In this research, Mobile Net and Efficient Net CNN models were trained for segmenting human kidney images generated from The Human Biomolecular Atlas Program (HuBMAP). The purpose of this work is to evaluate the effectiveness of different strategies for Glomeruli identification in order to solve the issue. The high size images were decoded to be fitted and trained in the models first, then the CNN models were trained. The CNN models result show that the Efficient Net has the highest accuracy rate with 99.49 %, and Mobile Net with 99.33 %.