Train Acceleration Research Articles

Accelerating one-shot neural architecture search via constructing a sparse search space

Neural Architecture Search (NAS) has garnered significant attention for its ability to automatically design high-quality deep neural networks (DNNs) tailored to various hardware platforms. The major challenge for NAS is the time-consuming network estimation process required to select optimal networks from a large pool of candidates. Rather than training each candidate from scratch, recent one-shot NAS methods accelerate the estimation process by only training a supernet and sampling sub-networks from it, inheriting partial network architectures and weights. Despite significant acceleration, the supernet training with a large search space (i.e., the number of candidate sub-networks) still requires thousands of GPU hours to support high-quality sub-network sampling. In this work, we propose SparseNAS, an approach for one-shot NAS acceleration by reducing the redundancy of the search space. We observe that many sub-networks in the space are underperforming, with significant performance disparity to high-performance sub-networks. Crucially, this disparity can be observed early in the beginning of the supernet training. Therefore, we train an early predictor to learn this disparity and filter out high-quality networks in advance. Then, the supernet training will be conducted in this space sub-space. Compared to the state-of-the-art one-shot NAS, our SparseNAS reports a 3.1× training speedup with comparable network performance on the ImageNet dataset. Compared to the state-of-the-art acceleration method, SparseNAS reports a maximum of 1.5% higher Top-1 accuracy and 28% training cost reduction with a 7× bigger search space. Extensive experiment results demonstrated that SparseNAS achieves better trade-offs between efficiency and performance than state-of-the-art one-shot NAS.

MixTrain: accelerating DNN training via input mixing.

Training Deep Neural Networks (DNNs) places immense compute requirements on the underlying hardware platforms, expending large amounts of time and energy. An important factor contributing to the long training times is the increasing dataset complexity required to reach state-of-the-art performance in real-world applications. To address this challenge, we explore the use of input mixing, where multiple inputs are combined into a single composite input with an associated composite label for training. The goal is for training on the mixed input to achieve a similar effect as training separately on each the constituent inputs that it represents. This results in a lower number of inputs (or mini-batches) to be processed in each epoch, proportionally reducing training time. We find that naive input mixing leads to a considerable drop in learning performance and model accuracy due to interference between the forward/backward propagation of the mixed inputs. We propose two strategies to address this challenge and realize training speedups from input mixing with minimal impact on accuracy. First, we reduce the impact of inter-input interference by exploiting the spatial separation between the features of the constituent inputs in the network's intermediate representations. We also adaptively vary the mixing ratio of constituent inputs based on their loss in previous epochs. Second, we propose heuristics to automatically identify the subset of the training dataset that is subject to mixing in each epoch. Across ResNets of varying depth, MobileNetV2 and two Vision Transformer networks, we obtain upto 1.6 × and 1.8 × speedups in training for the ImageNet and Cifar10 datasets, respectively, on an Nvidia RTX 2080Ti GPU, with negligible loss in classification accuracy.

Marvel: Towards Efficient Federated Learning on IoT Devices

Federated learning has gained significant attention as a distributed machine learning paradigm, particularly due to its ability to preserve privacy by keeping data on IoT devices and only transmitting model parameters to the central server. However, existing federated learning systems face several challenges in their deployment. These challenges can be categorized into three main areas: data relation obliviousness, model characteristic obliviousness, and low communication bandwidth. To address these issues, we propose Marvel, a novel consolidated federated learning system designed specifically for efficient model training on IoT devices. Marvel tackles the challenges by employing a series of innovative techniques. Firstly, it clusters devices based on their data similarities, leveraging data fingerprints and the Hamming distance between them. This clustering process allows Marvel to group devices with similar data, enabling more efficient model training. Once the device clustering is complete, Marvel initiates the formal model training process. During this process, Marvel dynamically selects one device from each cluster to upload model parameters based on the predicted model convergence speed. This prediction is achieved using a lightweight LSTM-based prediction model, which helps determine the most suitable devices for parameter updates. Furthermore, Marvel employs a Tucker decomposition approach to decompose the parameters before transmission in each communication round. This decomposition reduces the size of the transmitted data and enables faster communication between devices and the central server. The parameters are then reconstructed on the central server, and vice versa when dispatching model parameters to devices. We evaluate Marvel using a 180-device testbed and implement a prototype using PyTorch. The results demonstrate the effectiveness of Marvel, as it achieves an average model training speedup of 59.81% and reduces communication rounds by up to 66.67% compared to existing schemes.

An efficient bandwidth-adaptive gradient compression algorithm for distributed training of deep neural networks

In distributed deep learning with data parallelism, communication bottleneck throttles the efficiency of model training. Recent studies adopt versatile gradient compression techniques, with communication sparsification standing out as an effective approach for reducing the number of gradients to be transmitted. However, the deployment of gradient sparsification is adversely influenced by the change of network environment in real systems, and existing methods either neglect bandwidth dynamics during training or experience drastic fluctuation of compression ratios. In this paper, we propose ACE, a novel adaptive gradient compression mechanism with high communication efficiency under bandwidth variation. ACE adapts the sparsification ratio to the average bandwidth in a time window, other than following its dynamics exactly. To accurately compute the compression ratio, we first profile the compression time and model a single iteration time consisting of communication, computation and compression operations. We then present a practical model to fit the needed training rounds till convergence, and formulate an optimization problem to compute the optimal sparsification ratio. We conduct experiments on different DNN models in different network environments and compare various methods in terms of convergence and model quality. The experimental results show that ACE achieves up to 9.39× and 1.28× training speedups over fixed and state-of-the-art adaptive compression methods.

ETC: Efficient Training of Temporal Graph Neural Networks over Large-Scale Dynamic Graphs

Dynamic graphs play a crucial role in various real-world applications, such as link prediction and node classification on social media and e-commerce platforms. Temporal Graph Neural Networks (T-GNNs) have emerged as a leading approach for handling dynamic graphs, using temporal message passing to compute temporal node embeddings. However, training existing T-GNNs on large-scale dynamic graphs is prohibitively expensive due to the ill-suited batching scheme and significant data access overhead. In this paper, we introduce ETC, a generic framework designed specifically for efficient T-GNN training at scale. ETC incorporates a novel data batching scheme that enables large training batches improving model computation efficiency, while preserving model effectiveness by restricting information loss in each training batch. To reduce data access overhead, ETC employs a three-step data access policy that leverages the data access pattern in T-GNN training, significantly reducing redundant data access volume. Additionally, ETC utilizes an inter-batch pipeline mechanism, decoupling data access from model computation and further reducing data access costs. Extensive experimental results demonstrate the effectiveness of ETC, showcasing its ability to achieve significant training speedups compared to state-of-the-art training frameworks for T-GNNs on real-world dynamic graphs with millions of interactions. ETC provides a training speedup ranging from 1.6X to 62.4X, highlighting its potential for efficient training on large-scale dynamic graphs.

Linear Time Electromigration Analysis Based on Physics-Informed Sparse Regression

In this work, we propose a novel physics-informed sparse regression (PISR) framework to solve stress evolution (described by Korhonen’s equations) in general multi-segment wires using an unsupervised learning scheme. Unlike the existing physics-informed neural network framework (PINN), the PISR method trains the trainable weights through the Moore-Penrose generalized inverse algorithm used in extreme learning machine (ELM), which is extremely faster than the back-propagation algorithm. To improve the accuracy of PISR for complex multi-segment interconnects, we employ domain decomposition schemes in both space and time. For each subdomain, we use different trainable weights but the same shared neural network to represent each subsolution, which leads to more efficient memory usage. Furthermore, we propose to use sparse matrix techniques to accelerate the training speed of the PISR method and prove that the resulting PISR has linear time complexity for analyzing tree-structured interconnects. Finally, we divide the time into many time internals and apply an autoregressive model to simulate each time interval in sequence to further improve scalability and reduce memory cost so that the PISR method can perform EM analysis for large-scale multi-segment interconnects. Experimental results on different kinds of interconnect structures show that the proposed PISR method has the same accuracy level as the numerical methods. The results on NT T-junctions interconnect trees show that the proposed PISR method indeed demonstrates true linear time complexity. Furthermore, PISR can deliver 8.9×, 20.6×, and 1284× speedups over the recently proposed semi-analytic method (ASOV), finite difference method accelerated with model order reduction (FDM-MOR), FDM for the interconnect with NT= 5000, respectively. Furthermore, we show that PISR also achieves an 818× speedup in training over the plain PINN method based on the traditional back-propagation algorithm.

Privacy‐preserving model splitting and quality‐aware device association for federated edge learning

AbstractFederated edge learning (FEEL) provides a promising device‐edge collaborative learning paradigm, which enables edge devices to parallel participate in model co‐creation while preserving user privacy, opening countless opportunities to enable edge intelligence. With the growing demand for intelligent services, extensive FEEL deployment is inevitable. Nevertheless, existing FL schemes neglect two unique features (i.e., resource heterogeneity and data heterogeneity) in real‐world edge learning and thus may negatively affect the training efficiency and accuracy. Specifically, (1) heterogeneous and limited device resources cause massive laggards, which bring intolerable training delay; (2) heterogeneous data distribution causes device quality divergence, bringing severe training accuracy degradation. This article proposes a split‐based FEEL framework and an adaptive model splitting and quality‐aware device association scheme (MSDA) to tackle the aforementioned challenges. MSDA contains two levels: at the model splitting level, according to device capability and model structure, an adaptive splitting mechanism is proposed to provide a low‐latency and privacy‐preserving model splitting strategy for each device and guide subsequent device association. At the device association level, each device is simulated as a player with a quality weight in the potential game. Then a quality‐aware decentralized device association mechanism is designed to ensure that more high‐quality devices upload local updates before the deadline with the help of the edge server. Finally, experimental results demonstrate that MSDA yields significant improvements, achieving up to 3.1 training speedup and 39% accuracy improvement compared to state‐of‐the‐art methods.

Noise-boosted recurrent backpropagation

A statistical formulation of recurrent backpropagation (RBP) allows direct noise boosting for time-varying classification and regression. The noise boost reduces training iterations and improves accuracy. The injected noise is just that noise that makes the current signal more probable. This noise-boost result extends the two recent results that backpropagation is a special case of the generalized expectation maximization (EM) algorithm and that careful noise injection can always speed the average convergence of the EM algorithm to a local maximum of the log-likelihood surface. The noise-benefit conditions differ for additive and multiplicative noise in RBP. We tested noise-boosted RBP classifiers on 11 classes of sports video clips and tested RBP regressors on predicting the dollar-rupee exchange rate. Injecting noisy-EM (NEM) noise outperformed injecting blind noise or injecting no noise at all. Additive NEM noise usually outperformed multiplicative noise. The best case of NEM noise injection with RBP training of a recurrent neural classification model speeded up its training by 60% and improved its classification accuracy by 9.51% compared with noiseless RBP training and accuracy. The best performance of the NEM noise with the RBP training of a recurrent neural regression model yielded a 38% speed-up in training and also reduced the squared error by 49.3%. The injection of the additive NEM noise in the output and hidden neurons performed best.

A Faster Dynamic Feature Extractor and Its Application to Industrial Quality Prediction

The unsupervised dynamic models have been applied to various tasks in the process industry due to their excellent ability to represent the process dynamics. The recurrent-network-based dynamic feature extractor is a typical unsupervised dynamic model which extracts the dynamic data features using a recurrent encoder network. However, the recurrent-network-based dynamic feature extractor has low computational efficiency due to its recurrent nature, which prevents the model from being used for large-scale data sets. To improve computational efficiency, a new dynamic feature extractor called TempoATTNE-DFE is proposed in this work. In TempoATTNE-DFE, a new encoder structure is developed, which can be implemented in parallel for data sequences. Meanwhile, a kind of attention mechanism is proposed to extract the dynamic features within the input sequence. The proposed TempoATTNE-DFE can achieve higher computational efficiency in offline training and online inference. To evaluate the effectiveness of TempoATTNE-DFE, it is applied to the quality prediction task and validated with a numerical example and two real-world industrial processes. The application results demonstrate that TempoATTNE-DFE can achieve better prediction performance compared to other state-of-the-art methods. In addition, compared with the recurrent-network-based dynamic feature extractor, TempoATTNE-DFE gains <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$ 1.29\times$</tex-math></inline-formula> speedup in training and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$ 2.45\times$</tex-math></inline-formula> speedup in inference on the blast furnace data set.

Optimizing Training Efficiency and Cost of Hierarchical Federated Learning in Heterogeneous Mobile-Edge Cloud Computing

Federated learning (FL), an emerging distributed machine learning (ML) technique, allows massive embedded devices and a server to work together for training a global ML model without collecting user data on a server. Most existing approaches adopt the traditional centralized FL paradigm with a single server: one is the cloud-centric FL paradigm and the other is the edge-centric FL paradigm. The cloud-centric FL paradigm is able to manage a large-scale FL system across massive user devices with high communication cost, whereas the edge-centric FL paradigm is capable of coordinating a small-scale FL system benefiting from the low communication delay over wireless networks. To fully exploit the advantages of both, in this paper, we develop a distinctive hierarchical FL framework for the promising mobile-edge cloud computing (MECC) system, called HELCHFL, to achieve high-efficiency and low-cost hierarchical FL training. In particular, we formulate the corresponding theoretical foundation for our HELCHFL to ensure hierarchical training performance. Furthermore, to address the inherent communication and user heterogeneity issues of FL training, our HELCHFL develops a utility-driven and heterogeneity-aware heuristic user selection strategy to enhance training performance and reduce training delay. Subsequently, by analyzing and utilizing the slack time in FL training, our HELCHFL introduces a device operating frequency determination approach to reduce training energy cost. Experiments demonstrate that our HELCHFL can enhance the highest accuracy by up to 52.93%, gain the training speedup of up to 483.74%, and obtain up to 45.59% training energy savings compared to state-of-the-art baselines.

Expediting Distributed DNN Training With Device Topology-Aware Graph Deployment

This paper presents TAG, an automatic system to derive optimized DNN training graph and its deployment onto any device topology, for expedited training in device- and topology- heterogeneous ML clusters. We novelly combine both the DNN computation graph and the device topology graph as input to a graph neural network (GNN), and join the GNN with a search-based method to quickly identify optimized distributed training strategies. To reduce communication in a heterogeneous cluster, we further explore a lossless gradient compression technique and solve a combinatorial optimization problem to automatically apply the technique for training time minimization. We evaluate TAG with various representative DNN models and device topologies, showing that it can achieve up to 4.56x training speed-up as compared to existing schemes. TAG can produce efficient deployment strategies for both unseen DNN models and unseen device topologies, without heavy fine-tuning.

Analysis of Chest X-ray for COVID-19 Diagnosis as a Use Case for an HPC-Enabled Data Analysis and Machine Learning Platform for Medical Diagnosis Support.

The COVID-19 pandemic shed light on the need for quick diagnosis tools in healthcare, leading to the development of several algorithmic models for disease detection. Though these models are relatively easy to build, their training requires a lot of data, storage, and resources, which may not be available for use by medical institutions or could be beyond the skillset of the people who most need these tools. This paper describes a data analysis and machine learning platform that takes advantage of high-performance computing infrastructure for medical diagnosis support applications. This platform is validated by re-training a previously published deep learning model (COVID-Net) on new data, where it is shown that the performance of the model is improved through large-scale hyperparameter optimisation that uncovered optimal training parameter combinations. The per-class accuracy of the model, especially for COVID-19 and pneumonia, is higher when using the tuned hyperparameters (healthy: 96.5%; pneumonia: 61.5%; COVID-19: 78.9%) as opposed to parameters chosen through traditional methods (healthy: 93.6%; pneumonia: 46.1%; COVID-19: 76.3%). Furthermore, training speed-up analysis shows a major decrease in training time as resources increase, from 207 min using 1 node to 54 min when distributed over 32 nodes, but highlights the presence of a cut-off point where the communication overhead begins to affect performance. The developed platform is intended to provide the medical field with a technical environment for developing novel portable artificial-intelligence-based tools for diagnosis support.

Embarrassingly Parallel Independent Training of Multi-Layer Perceptrons with Heterogeneous Architectures

In this paper we propose a procedure to enable the training of several independent Multilayer Perceptron Neural Networks with a different number of neurons and activation functions in parallel (ParallelMLPs) by exploring the principle of locality and parallelization capabilities of modern CPUs and GPUs. The core idea of this technique is to represent several sub-networks as a single large network and use a Modified Matrix Multiplication that replaces an ordinal matrix multiplication with two simple matrix operations that allow separate and independent paths for gradient flowing. We have assessed our algorithm in simulated datasets varying the number of samples, features and batches using 10,000 different models as well as in the MNIST dataset. We achieved a training speedup from 1 to 4 orders of magnitude if compared to the sequential approach. The code is available online.

Handover Optimization Algorithm Based on T2RFS-FNN.

As a key technology for highly reliable communication in the fifth generation mobile communication for railway (5G-R) high-speed railway wireless communication system, once the handover fails, it will pose a serious risk to the safe operation of high-speed railway. As the speed of high-speed trains continues to increase, the handover will become more frequent, and how to improve the success rate of the handover is a key problem that needs to be solved. In this paper, we proposed an optimization algorithm based on the interval type 2 feature selection recurrent fuzzy neural network (T2RFS-FNN), which is a recurrent fuzzy neural network with interval type 2 feature selection, to address the problem of fixed hysteresis threshold and single consideration for the handover algorithm between the control plane and the user plane of the high-speed railway under 5G-R. The algorithm integrates reference signal receiving power (RSRP). Reference signal receiving quality (RSRQ) and throughput to optimise the hysteresis threshold. First, a feedforward neural network structure is designed to implement fuzzy logic inference, and an interval type-two Gaussian subordination function is used to improve the nonlinear expressiveness of the model. Then, a feature selection layer is added to determine the output of the affiliation function, which completes the optimization of the hysteresis threshold and overcomes the drawback of the fixed hysteresis threshold of the handover algorithm. Finally, simulation analysis of the control-plane and user-plane handover algorithms is carried out separately. The results show that the proposed method can effectively improve the success rate and reduce the ping-pong handover rate compared to the comparison algorithms. The results provide a theoretical reference for the speedup of high-speed railway trains and the evolution of the global system for mobile communications for railway (GSM-R) to 5G-R.

Optimizing DNN Compilation for Distributed Training With Joint OP and Tensor Fusion

This article proposes <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DisCo</i> , an automatic deep learning compilation module for data-parallel distributed training. Unlike most deep learning compilers that focus on training or inference on a single device, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DisCo</i> optimizes a DNN model for distributed training over multiple GPU machines. Existing single-device compilation strategies do not work well in distributed training, due mainly to communication inefficiency that they incur. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DisCo</i> generates optimized, joint computation operator and communication tensor fusion strategies to enable highly efficient distributed training. A GNN-based simulator is built to effectively estimate per-iteration training time achieved by operator/tensor fusion candidates. A backtracking search algorithm is driven by the simulator, navigating efficiently in the large strategy space to identify good operator/tensor fusion strategies that minimize distributed training time. We compare <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DisCo</i> with existing DL fusion schemes and show that it achieves good training speed-up close to the ideal, full computation-communication overlap case.

Speeding-up one-versus-all training for extreme classification via mean-separating initialization

In this paper, we show that a simple, data dependent way of setting the initial vector can be used to substantially speed up the training of linear one-versus-all classifiers in extreme multi-label classification (XMC). We discuss the problem of choosing the initial weights from the perspective of three goals. We want to start in a region of weight space (a) with low loss value, (b) that is favourable for second-order optimization, and (c) where the conjugate-gradient (CG) calculations can be performed quickly. For margin losses, such an initialization is achieved by selecting the initial vector such that it separates the mean of all positive (relevant for a label) instances from the mean of all negatives – two quantities that can be calculated quickly for the highly imbalanced binary problems occurring in XMC. We demonstrate a training speedup of up to 5times on Amazon-670K dataset with 670,000 labels. This comes in part from the reduced number of iterations that need to be performed due to starting closer to the solution, and in part from an implicit negative-mining effect that allows to ignore easy negatives in the CG step. Because of the convex nature of the optimization problem, the speedup is achieved without any degradation in classification accuracy. The implementation can be found at https://github.com/xmc-aalto/dismecpp.

3LegRace: Privacy-Preserving DNN Training over TEEs and GPUs

Leveraging parallel hardware (e.g. GPUs) for deep neural network (DNN) training brings high computing performance. However, it raises data privacy concerns as GPUs lack a trusted environment to protect the data. Trusted execution environments (TEEs) have emerged as a promising solution to achieve privacypreserving learning. Unfortunately, TEEs’ limited computing power renders them not comparable to GPUs in performance. To improve the trade-off among privacy, computing performance, and model accuracy, we propose an asymmetric model decomposition framework, AsymML, to (1) accelerate training using parallel hardware; and (2) achieve a strong privacy guarantee using TEEs and differential privacy (DP) with much less accuracy compromised compared to DP-only methods. By exploiting the low-rank characteristics in training data and intermediate features, AsymML asymmetrically decomposes inputs and intermediate activations into low-rank and residual parts. With the decomposed data, the target DNN model is accordingly split into a trusted and an untrusted part. The trusted part performs computations on low-rank data, with low compute and memory costs. The untrusted part is fed with residuals perturbed by very small noise. Privacy, computing performance, and model accuracy are well managed by respectively delegating the trusted and the untrusted part to TEEs and GPUs. We provide a formal DP guarantee that demonstrates that, for the same privacy guarantee, combining asymmetric data decomposition and DP requires much smaller noise compared to solely using DP without decomposition. This improves the privacy-utility trade-off significantly compared to using only DP methods without decomposition. Furthermore, we present a rank bound analysis showing that the low-rank structure is preserved after each layer across the entire model. Our extensive evaluations on DNN models show that AsymML delivers 7.6× speedup in training compared to the TEE-only executions while ensuring privacy. We also demonstrate that AsymML is effective in protecting data under common attacks such as model inversion and gradient attacks.

Enhancing Channelized Feature Interpretability Using Deep Learning Predictive Modeling

Automating geobodies using insufficient labeled training data as input for structural prediction may result in missing important features and a possibility of overfitting, leading to low accuracy. We adopt a deep learning (DL) predictive modeling scheme to alleviate detection of channelized features based on classified seismic attributes (X) and different ground truth scenarios (y), to imitate actual human interpreters’ tasks. In this approach, diverse augmentation method was applied to increase the accuracy of the model after we were satisfied with the refined annotated ground truth dataset. We evaluated the effect of dropout as a training regularizer and facies’ spatial representation towards optimized prediction results, apart from conventional hyperparameter tuning. From our findings, increasing batch size helps speedup training speed and improve performance stability. Finally, we demonstrate that the designed Convolutional Neural Network (CNN) is capable of learning channelized variation from complex deepwater settings in a fluvial-dominated depositional environment while producing outstanding mean Intersection of Union (IoU) (95%) despite utilizing 6.4% from the overall dataset and avoiding overfitting possibilities.

Toward the Optimal Design and FPGA Implementation of Spiking Neural Networks.

The performance of a biologically plausible spiking neural network (SNN) largely depends on the model parameters and neural dynamics. This article proposes a parameter optimization scheme for improving the performance of a biologically plausible SNN and a parallel on-field-programmable gate array (FPGA) online learning neuromorphic platform for the digital implementation based on two numerical methods, namely, the Euler and third-order Runge-Kutta (RK3) methods. The optimization scheme explores the impact of biological time constants on information transmission in the SNN and improves the convergence rate of the SNN on digit recognition with a suitable choice of the time constants. The parallel digital implementation leads to a significant speedup over software simulation on a general-purpose CPU. The parallel implementation with the Euler method enables around 180× ( 20× ) training (inference) speedup over a Pytorch-based SNN simulation on CPU. Moreover, compared with previous work, our parallel implementation shows more than 300× ( 240× ) improvement on speed and 180× ( 250× ) reduction in energy consumption for training (inference). In addition, due to the high-order accuracy, the RK3 method is demonstrated to gain 2× training speedup over the Euler method, which makes it suitable for online training in real-time applications.

A generalized framework for unsupervised learning and data recovery in computational fluid dynamics using discretized loss functions

The authors present generalized finite-volume-based discretized loss functions integrated into pressure-linked algorithms for physics-based unsupervised training of neural networks (NNs). In contrast to automatic differentiation-based counterparts, discretized loss functions leverage well-developed numerical schemes of computational fluid dynamics (CFD) for tailoring NN training specific to the flow problems. For validation, neural network-based solvers (NN solvers) are trained by posing equations such as the Poisson equation, energy equation, and Spalart–Allmaras model as loss functions. The predictions from the trained NNs agree well with the solutions from CFD solvers while also providing solution time speed-ups of up to seven times. Another application of unsupervised learning is the novel hybrid loss functions presented in this study. Hybrid learning combines the information from sparse or partial observations with a physics-based loss to train the NNs accurately and provides training speed-ups of up to five times compared with a fully unsupervised method. Also, to properly utilize the potential of discretized loss functions, they are formulated in a machine learning (ML) framework (TensorFlow) integrated with a CFD solver (OpenFOAM). The ML-CFD framework created here infuses versatility into the training by giving loss functions access to the different numerical schemes of the OpenFOAM. In addition, this integration allows for offloading the CFD programming to OpenFOAM, circumventing bottlenecks from manually coding new flow conditions in a solely ML-based framework like TensorFlow.