Higher Training Efficiency Research Articles

A frequency domain-based machine learning architecture for short-term wave height forecasting

Accurate and reliable short-term waves forecasting plays a crucial role in ensuring the safety of offshore operations. However, the non-stationary and high-noise characteristics of wave height time series in real-world environments present significant challenges for prediction. This paper introduces a novel lightweight machine learning forecasting architecture, named FreMixer, that leverages frequency domain information. Specifically designed for sequences with a low signal-to-noise ratio (SNR), FreMixer utilizes a Gaussian filter block for noise reduction and a Fourier block for frequency domain feature extraction. The Mixer block, which incorporates skip-connections, serves as the key component for feature learning. Towing tank experiments demonstrate the superior performance of FreMixer compared to benchmark models such as the autoregressive moving average (ARMA), long short-term memory (LSTM), and extreme gradient boosting (XGBoost) models. Due to its simple structure, FreMixer offers low deployment costs and high training efficiency, making it an attractive option for practical applications. Furthermore, the impact of training set size and historical window size on the performance is discussed.

Generating Image Descriptions of Rice Diseases and Pests Based on DeiT Feature Encoder

We propose a DeiT (Data-Efficient Image Transformer) feature encoder-based algorithm for identifying disease types and generating relevant descriptions of diseased crops. It solves the scarcity problem of the image description algorithm applied in agriculture. We divided the original image into a sequence of image patches to fit the input form of the DeiT encoder, which was distilled by RegNet. Then, we used the Transformer decoder to generate descriptions. Compared to “CNN + LSTM” models, our proposed model is entirely convolution-free and has high training efficiency. On the Rice2k dataset created by us, the model achieved a 47.3 BLEU-4 score, 65.0 ROUGE_L score, and 177.1 CIDEr score. The extensive experiments demonstrate the effectiveness and the strong robustness of our model. It can be better applied to automatically generate descriptions of similar crop disease characteristics.

PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

It is widely acknowledged that large models have the potential to deliver superior performance across a broad range of domains. Despite the remarkable progress made in the field of machine learning systems research, which has enabled the development and exploration of large models, such abilities remain confined to a small group of advanced users and industry leaders, resulting in an implicit technical barrier for the wider community to access and leverage these technologies. In this paper, we introduce PyTorch Fully Sharded Data Parallel (FSDP) as an industry-grade solution for large model training. FSDP has been closely co-designed with several key PyTorch core components including Tensor implementation, dispatcher system, and CUDA memory caching allocator, to provide non-intrusive user experiences and high training efficiency. Additionally, FSDP natively incorporates a range of techniques and settings to optimize resource utilization across a variety of hardware configurations. The experimental results demonstrate that FSDP is capable of achieving comparable performance to Distributed Data Parallel while providing support for significantly larger models with near-linear scalability in terms of TFLOPS.

Self Correspondence Distillation for End-to-End Weakly-Supervised Semantic Segmentation

Efficiently training accurate deep models for weakly supervised semantic segmentation (WSSS) with image-level labels is challenging and important. Recently, end-to-end WSSS methods have become the focus of research due to their high training efficiency. However, current methods suffer from insufficient extraction of comprehensive semantic information, resulting in low-quality pseudo-labels and sub-optimal solutions for end-to-end WSSS. To this end, we propose a simple and novel Self Correspondence Distillation (SCD) method to refine pseudo-labels without introducing external supervision. Our SCD enables the network to utilize feature correspondence derived from itself as a distillation target, which can enhance the network's feature learning process by complementing semantic information. In addition, to further improve the segmentation accuracy, we design a Variation-aware Refine Module to enhance the local consistency of pseudo-labels by computing pixel-level variation. Finally, we present an efficient end-to-end Transformer-based framework (TSCD) via SCD and Variation-aware Refine Module for the accurate WSSS task. Extensive experiments on the PASCAL VOC 2012 and MS COCO 2014 datasets demonstrate that our method significantly outperforms other state-of-the-art methods. Our code is available at https://github.com/Rongtao-Xu/RepresentationLearning/tree/main/SCD-AAAI2023.

DPText-DETR: Towards Better Scene Text Detection with Dynamic Points in Transformer

Recently, Transformer-based methods, which predict polygon points or Bezier curve control points for localizing texts, are popular in scene text detection. However, these methods built upon detection transformer framework might achieve sub-optimal training efficiency and performance due to coarse positional query modeling. In addition, the point label form exploited in previous works implies the reading order of humans, which impedes the detection robustness from our observation. To address these challenges, this paper proposes a concise Dynamic Point Text DEtection TRansformer network, termed DPText-DETR. In detail, DPText-DETR directly leverages explicit point coordinates to generate position queries and dynamically updates them in a progressive way. Moreover, to improve the spatial inductive bias of non-local self-attention in Transformer, we present an Enhanced Factorized Self-Attention module which provides point queries within each instance with circular shape guidance. Furthermore, we design a simple yet effective positional label form to tackle the side effect of the previous form. To further evaluate the impact of different label forms on the detection robustness in real-world scenario, we establish an Inverse-Text test set containing 500 manually labeled images. Extensive experiments prove the high training efficiency, robustness, and state-of-the-art performance of our method on popular benchmarks. The code and the Inverse-Text test set are available at https://github.com/ymy-k/DPText-DETR.

A Method for Predicting Surface Finish of Polylactic Acid Parts Printed Using Fused Deposition Modeling

Accurately predicting the surface finish of fused deposition modeling (FDM) parts is an important task for the engineering application of FDM technology. So far, many prediction models have been proposed by establishing a mapping relationship between printing parameters and surface roughness. Each model can work well in its specific context; however, existing prediction models cannot meet the requirements of multi-factor and multi-category prediction of surface finish and cope with imbalanced data. Aiming at these issues, a prediction method based on a combination of the adaptive particle swarm optimization and K-nearest neighbor (APSO-KNN) algorithms is proposed in this paper. Seven input variables, including nozzle diameter, layer thickness, number of perimeters, flow rate, print speed, nozzle temperature, and build orientation, are considered. The printing values of each specimen are determined using an L27 Taguchi experimental design. A total of 27 specimens are printed and experimental data for the 27 specimens are used for model training and validation. The results indicate that the proposed method can achieve a minimum classification error of 0.01 after two iterations, with a maximum accuracy of 99.0%, and high model training efficiency. It can meet the requirements of predicting surface finish for FDM parts with multiple factors and categories and can handle imbalanced data. In addition, the high accuracy demonstrates the potential of this method for predicting surface finish, and its application in actual industrial manufacturing.

General framework for E(3)-equivariant neural network representation of density functional theory Hamiltonian

The combination of deep learning and ab initio calculation has shown great promise in revolutionizing future scientific research, but how to design neural network models incorporating a priori knowledge and symmetry requirements is a key challenging subject. Here we propose an E(3)-equivariant deep-learning framework to represent density functional theory (DFT) Hamiltonian as a function of material structure, which can naturally preserve the Euclidean symmetry even in the presence of spin–orbit coupling. Our DeepH-E3 method enables efficient electronic structure calculation at ab initio accuracy by learning from DFT data of small-sized structures, making the routine study of large-scale supercells (>104 atoms) feasible. The method can reach sub-meV prediction accuracy at high training efficiency, showing state-of-the-art performance in our experiments. The work is not only of general significance to deep-learning method development but also creates opportunities for materials research, such as building a Moiré-twisted material database.

A Novel Minimal-Cost Power Allocation Strategy for Fuel Cell Hybrid Buses Based on Deep Reinforcement Learning Algorithms

Energy management strategy (EMS) is critical for improving the economy of hybrid powertrains and the durability of energy sources. In this paper, a novel EMS based on a twin delayed deep deterministic policy gradient algorithm (TD3) is proposed for a fuel cell hybrid electric bus (FCHEB) to optimize the driving cost of the vehicle. First, a TD3-based energy management strategy is established to embed the limits of battery aging and fuel cell power variation into the strategic framework to fully exploit the economic potential of FCHEB. Second, the TD3-based EMS is compared and analyzed with the deep deterministic policy gradient algorithm (DDPG)-based EMS using real-world collected driving conditions as training data. The results show that the TD3-based EMS has 54.69% higher training efficiency, 36.82% higher learning ability, and 2.45% lower overall vehicle operating cost compared to the DDPG-based EMS, validating the effectiveness of the proposed strategy.

Merak: An Efficient Distributed DNN Training Framework With Automated 3D Parallelism for Giant Foundation Models

Foundation models are in the process of becoming the dominant deep learning technology. Pretraining a foundation model is always time-consuming due to the large scale of both the model parameter and training dataset. Besides being computing-intensive, the pretraining process is extremely memory- and communication-intensive. These challenges make it necessary to apply 3D parallelism, which integrates data parallelism, pipeline model parallelism, and tensor model parallelism, to achieve high training efficiency. However, current 3D parallelism frameworks still encounter two issues: i) they are not transparent to model developers, requiring manual model modification to parallelize training, and ii) their utilization of computation resources, GPU memory, and network bandwidth is insufficient. We propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Merak</i> , an automated 3D parallelism deep learning training framework with high resource utilization. Merak automatically deploys 3D parallelism with an automatic model partitioner, which includes a graph-sharding algorithm and proxy node-based model graph. Merak also offers a non-intrusive API to scale out foundation model training with minimal code modification. In addition, we design a high-performance 3D parallel runtime engine that employs several techniques to exploit available training resources, including a shifted critical path pipeline schedule that increases computation utilization, stage-aware recomputation that makes use of idle worker memory, and sub-pipelined tensor model parallelism that overlaps communication and computation. Experiments on 64 GPUs demonstrate Merak's capability to speed up training performance over state-of-the-art 3D parallelism frameworks of models with 1.5, 2.5, 8.3, and 20 billion parameters by up to 1.42, 1.39, 1.43, and 1.61×, respectively.

Combining flamelet-generated manifold and machine learning models in simulation of a non-premixed diffusion flame

Flamelet Generated Manifold (FGM) is an example of a chemistry tabulation or a flamelet method that is under attention because of its accuracy and speed in predicting combustion characteristics. However, the main problem in applying the model is a large amount of memory required. One way to solve this problem is to apply machine learning (ML) to replace the stored tabulated data. Four different machine learning methods, including two Artificial Neural Networks (ANNs), a Random Forest (RF), and a Gradient Boosted Trees (GBT), are trained, validated, and compared in terms of various performance measures. The progress variable source term and transport properties are replaced with the ML models. Particular attention was paid to the progress variable source term due to its high gradient and wide range of its value in the control variables space. Data preprocessing is shown to play an essential role in improving the performance of the models. Two ensemble models, namely RF and GBT, exhibit high training efficiency and acceptable accuracy. On the other hand, the ANN models have lower training errors and take longer to train. The four models are then combined with a one-dimensional combustion code to simulate a counterflow non-premixed diffusion flame in engine-relevant conditions. The predictions of the ML-FGM models are compared with detailed chemical simulations and the original FGM model for key combustion properties and representative species profiles.

A Self-Rectifying Synaptic Memristor Array with Ultrahigh Weight Potentiation Linearity for a Self-Organizing-Map Neural Network

Two-terminal self-rectifying (SR)-synaptic memristors are preeminent candidates for high-density and efficient neuromorphic computing, especially for future three-dimensional integrated systems, which can self-suppress the sneak path current in crossbar arrays. However, SR-synaptic memristors face the critical challenges of nonlinear weight potentiation and steep depression, hindering their application in conventional artificial neural networks (ANNs). Here, a SR-synaptic memristor (Pt/NiOx/WO3-x:Ti/W) and cross-point array with sneak path current suppression features and ultrahigh-weight potentiation linearity up to 0.9997 are introduced. The image contrast enhancement and background filtering are demonstrated on the basis of the device array. Moreover, an unsupervised self-organizing map (SOM) neural network is first developed for orientation recognition with high recognition accuracy (0.98) and training efficiency and high resilience toward both noises and steep synaptic depression. These results solve the challenges of SR memristors in the conventional ANN, extending the possibilities of large-scale oxide SR-synaptic arrays for high-density, efficient, and accurate neuromorphic computing.

Dual Optical Frequency Comb Neuron: Co‐Developing Hardware and Algorithm

Previous studies on photonic neural network have demonstrated that algorithm can inspire hardware design. This study seeks to demonstrate that hardware can also inspire algorithm design. To further exploit the advantages of photonic analog computing, the authors develop hardware and algorithm simultaneously for photonic convolutional neural networks. Specifically, this work developed an architecture called dual optical frequency comb neuron (DOFCN) enabled by an integrated microcomb to perform cosinusoidal nonlinear activation and vector convolution without temporal or spatial dispersion and large‐scale modulator arrays. Furthermore, DOFCN‐based composite vector convolutional neural networks (CVCNNs), an optical‐electric hybrid model, are proposed to perform classification and regression tests in signal modulation format identification and optical structure inverse design tasks, respectively. The ablation experiments show that under 4‐bit precision limit, the element‐wise activation CVCNN has 14% higher classification accuracy, 76% lower regression residuals, and 100% higher training efficiency than that of the 32‐bit standard convolutional neural network (CNN). DOFCN exhibits impressive spectral information processing ability to facilitate signal‐processing tasks related to optics and electromagnetics.

MS4D-Net: Multitask-Based Semi-Supervised Semantic Segmentation Framework with Perturbed Dual Mean Teachers for Building Damage Assessment from High-Resolution Remote Sensing Imagery

In the aftermath of a natural hazard, rapid and accurate building damage assessment from remote sensing imagery is crucial for disaster response and rescue operations. Although recent deep learning-based studies have made considerable improvements in assessing building damage, most state-of-the-art works focus on pixel-based, multi-stage approaches, which are more complicated and suffer from partial damage recognition issues at the building-instance level. In the meantime, it is usually time-consuming to acquire sufficient labeled samples for deep learning applications, making a conventional supervised learning pipeline with vast annotation data unsuitable in time-critical disaster cases. In this study, we present an end-to-end building damage assessment framework integrating multitask semantic segmentation with semi-supervised learning to tackle these issues. Specifically, a multitask-based Siamese network followed by object-based post-processing is first constructed to solve the semantic inconsistency problem by refining damage classification results with building extraction results. Moreover, to alleviate labeled data scarcity, a consistency regularization-based semi-supervised semantic segmentation scheme with iteratively perturbed dual mean teachers is specially designed, which can significantly reinforce the network perturbations to improve model performance while maintaining high training efficiency. Furthermore, a confidence weighting strategy is embedded into the semi-supervised pipeline to focus on convincing samples and reduce the influence of noisy pseudo-labels. The comprehensive experiments on three benchmark datasets suggest that the proposed method is competitive and effective in building damage assessment under the circumstance of insufficient labels, which offers a potential artificial intelligence-based solution to respond to the urgent need for timeliness and accuracy in disaster events.

Remote Sensing Image Segmentation and Extraction Based on U-Net Convolutional Neural Network Model

Remote sensing images are essential for quickly acquiring large-scale ground information. Segmentation and extraction of high-resolution remote sensing images are widely used in many fields, such as agricultural monitoring, urban and rural planning, and map production and updating. In this paper, a U-Net convolutional neural network model is built on the Tensor Flow framework. A data enhancement strategy is specially designed for the training task of remote sensing image parcel segmentation to enhance the model's generalization ability. The experimental results choose accuracy as the evaluation index, and the final model accuracy can reach 0.9440. The remote sensing image parcel segmentation method proposed in this paper has high training efficiency and is suitable for high-accuracy remote sensing image segmentation and extraction.

Predicting smart cities’ electricity demands using k-means clustering algorithm in smart grid

This work aims to perform the unified management of various departments engaged in smart city construction by big data, establish a synthetic data collection and sharing system, and provide fast and convenient big data services for smart applications in various fields. A new electricity demand prediction model based on back propagation neural network (BPNN) is proposed for China?s electricity industry according to the smart city?s big data characteristics. This model integrates meteorological, geographic, demographic, corporate, and economic information to form a big intelligent database. Moreover, the K-means clustering algorithm mines and analyzes the data to optimize the power consumers? information. The BPNN model is used to extract features for prediction. Users with weak daily correlation obtained by the K-means clustering algorithm only input the historical load of adjacent moments into the BPNN model for prediction. Finally, the electricity market is evaluated by exploring the data correlation in-depth to verify the proposed model?s effectiveness. The results indicate that the K-mean algorithm can significantly improve the segmentation accuracy of power consumers, with a maximum accuracy of 85.25% and average accuracy of 83.72%. The electricity consumption of different regions is separated, and the electricity consumption is classified. The electricity demand prediction model can enhance prediction accuracy, with an average error rate of 3.27%. The model?s training significantly speeds up by adding the momentum factor, and the average error rate is 2.13%. Therefore, the electricity demand prediction model achieves high accuracy and training efficiency. The findings can provide a theoretical and practical foundation for electricity demand prediction, personalized marketing, and the development planning of the power industry.

Groundwater contamination source identification and high-dimensional parameter inversion using residual dense convolutional neural network

Data assimilation for high-dimensional parameter joint inversion of multiple time-varying source strength and hydraulic conductivity fields can be computationally intensive as a large number of forward model runs are usually required. In this study, a deep neural network-based surrogate is proposed to replace the forward model in the data assimilation process to efficiently achieve high-dimensional parameter inversion. The deep convolutional encoding-decoding network architecture is used to leverage the advantages of the convolutional network in processing image-like data, where the high-dimensional input and output fields of the forward model are expressed as images. The encoding network extracts the main features of the multiple time-varying source input image and high-dimensional hydraulic conductivity fields, and the decoding network refines the extracted features to generate the output hydraulic head and contaminant concentration fields. The Residual Dense Convolutional Neural Network (RDCNN) is proposed by combining residual learning with a densely-connected convolutional neural network to solve the image regression problem for surrogate construction. Iterative local updating ensemble smoother (ILUES) is utilized to assimilate hydraulic head and contaminant concentration data to inverse high-dimensional parameters. The proposed method is evaluated by jointly inversing the high-dimensional hydraulic conductivity and source parameters for a synthetic multiple-source contaminated aquifer. The results indicate that compared to the surrogate constructed by Dense Convolutional Neural Network (DCNN) without addressing residual learning, the RDCNN surrogate captures the model input-output relationship better with relatively high training efficiency. The RDCNN surrogate-coupled ILUES achieves state-of-the-art performance in terms of inversion accuracy of 9 pollution source parameters and 255 hydraulic conductivity field parameters and computational efficiency in comparison to the ILUES based on the forward model.

Ranking-based contrastive loss for recommendation systems

The recommendation system is fundamental technology of the internet industry intended to solve the information overload problem in the big data era. Top-k recommendation is an important task in this field. It generally functions through the comparison of positive pairs and negative pairs based on Bayesian personalized ranking (BPR) loss. We find that the contrastive loss (CL) function used in contrastive learning is well-suited for top-k recommendation. However, there are two problems in the existing loss functions. First, all samples are treated the same, and hard samples are not considered. Second, all nonpositive samples are considered negative samples, which ignores the fact that they are unlabelled data containing items that users may like. Moreover, in our experiments, we find that when items are sorted by their similarities to the user, many negative items (or samples) appear before the positive items. We regard these negative items as hard samples and those at the top as potentially positive samples due to their high level of similarities with users. Therefore, we propose a ranking-based contrastive loss (RCL) function to exploit both hard samples and potentially positive samples. Experimental results demonstrate the effectiveness, broad applicability, and high training efficiency of the proposed RCL function. The code and data are available at https://github.com/haotangxjtu/RCL.

Quick intention identification of an enemy aerial target through information classification processing

Rapidly and accurately identifying the tactical intention of an enemy aerial target is an important issue for combat decision making. To this end, a quick intention identification model based on hybrid neural network is established in this paper. With available feature sequential measurements of the enemy target as inputs of the model, possibilities of different intentions are calculated timely. To increase the training efficiency and accuracy of recognition, the measurement information is processed using different neural network. Maneuvering data with large variations over time are processed using gated recurrent unit (GRU), while other data are processed using back propagation (BP) neural network. Besides, the fitting cubic sample interpolation is adopted to deal with incomplete information. Monte Carlo simulations demonstrate the robustness and accuracy of the established model, and training comparison with conventional models shows that the proposed method has higher training efficiency and better identification performance.

An Algorithm–Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers

The Transformer has been an indispensable staple in deep learning. However, for real-life applications, it is very challenging to deploy efficient Transformers due to the immense parameters and operations of models. To relieve this burden, exploiting sparsity is an effective approach to accelerate Transformers. Newly emerging Ampere graphics processing units (GPUs) leverage a 2:4 sparsity pattern to achieve model acceleration, while it can hardly meet the diverse algorithm and hardware constraints when deploying models. By contrast, we propose an algorithm–hardware co-optimized framework to flexibly and efficiently accelerate Transformers by utilizing general N:M sparsity patterns. First, from an algorithm perspective, we propose a sparsity inheritance mechanism along with inherited dynamic pruning (IDP) to obtain a series of N:M sparse candidate Transformers rapidly. A model compression scheme is further proposed to significantly reduce the storage requirement for deployment. Second, from a hardware perspective, we present a flexible and efficient hardware architecture, namely, STA, to achieve significant speedup when deploying N:M sparse Transformers. STA features not only a computing engine unifying both sparse–dense and dense–dense matrix multiplications with high computational efficiency but also a scalable softmax module eliminating the latency from intermediate off-chip data communication. Experimental results show that, compared to other methods, N:M sparse Transformers, generated using IDP, achieves an average of 6.7% improvement on accuracy with high training efficiency. Moreover, STA can achieve <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$14.47\times $ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$11.33\times $ </tex-math></inline-formula> speedups compared to Intel i9-9900X and NVIDIA RTX 2080 Ti, respectively, and perform <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.00 \,\,\sim 19.47 \times $ </tex-math></inline-formula> faster inference than the state-of-the-art field-programmable gate array (FPGA)-based accelerators for Transformers.

Accelerating reinforcement learning with case-based model-assisted experience augmentation for process control

In the context of intelligent manufacturing in the process industry, traditional model-based optimization control methods cannot adapt to the situation of drastic changes in working conditions or operating modes. Reinforcement learning (RL) directly achieves the control objective by interacting with the environment, and has significant advantages in the presence of uncertainty since it does not require an explicit model of the operating plant. However, most RL algorithms fail to retain transfer learning capabilities in the presence of mode variation, which becomes a practical obstacle to industrial process control applications. To address these issues, we design a framework that uses local data augmentation to improve the training efficiency and transfer learning (adaptability) performance. Therefore, this paper proposes a novel RL control algorithm, CBR-MA-DDPG, organically integrating case-based reasoning (CBR), model-assisted (MA) experience augmentation, and deep deterministic policy gradient (DDPG). When the operating mode changes, CBR-MA-DDPG can quickly adapt to the varying environment and achieve the desired control performance within several training episodes. Experimental analyses on a continuous stirred tank reactor (CSTR) and an organic Rankine cycle (ORC) demonstrate the superiority of the proposed method in terms of both adaptability and control performance/robustness. The results show that the control performance of the CBR-MA-DDPG agent outperforms the conventional PI and MPC control schemes, and that it has higher training efficiency than the state-of-the-art DDPG, TD3, and PPO algorithms in transfer learning scenarios with mode shift situations.