Efficient Training Research Articles

Device placement using Laplacian PCA and graph attention networks

Abstract The exponential growth in data and parameters in modern neural networks has created the need to distribute these models across multiple devices for efficient training, resulting in the device placement problem. Existing graph encoding approaches for device placement suffer from low efficiency when searching for optimal parallel strategies, primarily due to suboptimal positional information retrieval. To address these challenges, we propose the Laplacian Principal Component Analysis-graph attention networks (LPCA-GAT) model. Firstly, we employ GAT to capture complex relationships between nodes and generate node encodings. Secondly, we leverage LPCA on the graph Laplacian matrix to extract crucial low-dimensional positional information. Finally, by integrating these two components, we obtain the final node encodings. This enhances the representation capability of nodes within the graph, enabling efficient device placement. The experimental results demonstrate that LPCA-GAT achieves superior device placement results, specifically accelerating execution and computation time by 13.24% and 96.38%, respectively, leading to significant improvements in both operational efficiency and performance.

Deep belief network with fuzzy parameters and its membership function sensitivity analysis

Over the last few years, deep belief networks (DBNs) have been extensively utilized for efficient and reliable performance in several complex systems. One critical factor contributing to the enhanced learning of the DBN layers is the handling of network parameters, such as weights and biases. The efficient training of these parameters significantly influences the overall enhanced performance of the DBN. However, the initialization of these parameters is often random, and the data samples are normally corrupted by unwanted noise. This causes the uncertainty to arise among weights and biases of the DBNs, which ultimately hinders the performance of the network. To address this challenge, we propose a novel DBN model with weights and biases represented using fuzzy sets. The approach systematically handles inherent uncertainties in parameters resulting in a more robust and reliable training process. We show the working of the proposed algorithm considering four widely used benchmark datasets such as: MNSIT, n-MNIST (MNIST with additive white Gaussian noise (AWGN) and MNIST with motion blur) and CIFAR-10. The experimental results show superiority of the proposed approach as compared to classical DBN in terms of robustness and enhanced performance. Moreover, it has the capability to produce equivalent results with a smaller number of nodes in the hidden layer; thus, reducing the computational complexity of the network architecture. Additionally, we also study the sensitivity analysis for stability and consistency by considering different membership functions to model the uncertain weights and biases. Further, we establish the statistical significance of the obtained results by conducting both one-way and Kruskal-Wallis analyses of variance tests.

Enhancing Federated Learning in Heterogeneous Internet of Vehicles: A Collaborative Training Approach

The current Internet of Vehicles (IoV) faces significant challenges related to resource heterogeneity, which adversely impacts the convergence speed and accuracy of federated learning models. Existing studies have not adequately addressed the problem of resource-constrained vehicles that slow down the federated learning process particularly under conditions of high mobility. To tackle this issue, we propose a model partition collaborative training mechanism that decomposes training tasks for resource-constrained vehicles while retaining the original data locally. By offloading complex computational tasks to nearby service vehicles, this approach effectively accelerates the slow training speed of resource-limited vehicles. Additionally, we introduce an optimal matching method for collaborative service vehicles. By analyzing common paths and time delays, we match service vehicles with similar routes and superior performance within mobile service vehicle clusters to provide effective collaborative training services. This method maximizes training efficiency and mitigates the negative effects of vehicle mobility on collaborative training. Simulation experiments demonstrate that compared to benchmark methods, our approach reduces the impact of mobility on collaboration, achieving large improvements in the training speed and the convergence time of federated learning.

Key-Value Cache Quantization in Large Language Models: A Safety Benchmark

The misuse of large language models (LLMs) is an ongoing concern with the expansion of general public access to LLMs. One reason for expanded access is the development of key-value (KV) cache quantization, a technique that significantly reduces both the large computing resource requirements and memory bottlenecks that are characteristic of LLMs. As more developers and vendors begin to prioritize efficiency in LLM training, protective measures against the misuse of language models become more of an afterthought. To address the expected increase of LLM misuse accompanying KV cache quantization, this paper covers a proof-of-concept benchmark to evaluate the proficiency of LLMs in response safety when tested against a sample of unsafe questions consisting of 13 different question categories. Response safety is a model’s ability to both clearly deny providing a response to a given question and avoid providing any additional information that attempts to provide an accurate answer. By testing the sample against the Meta Llama-2-7B pretrained chat model, we determine response safety fine-tuning considerations that address performance bias among the 13 question categories. We hope this study brings attention to not only the sacrifices of accuracy but also response safety of KV cache quantization in large language models. (Code and data are available at https://github.com/TimochiL/llm_benchmark.) Disclaimer: This paper contains examples of harmful language. Reader discretion is recommended.

Resource‐adaptive and OOD‐robust inference of deep neural networks on IoT devices

AbstractEfficiently executing inference tasks of deep neural networks on devices with limited resources poses a significant load in IoT systems. To alleviate the load, one innovative method is branching that adds extra layers with classification exits to a pre‐trained model, enabling inputs with high‐confidence predictions to exit early, thus reducing inference cost. However, branching networks, not originally tailored for IoT environments, are susceptible to noisy and out‐of‐distribution (OOD) data, and they demand additional training for optimal performance. The authors introduce BrevisNet, a novel branching methodology designed for creating on‐device branching models that are both resource‐adaptive and noise‐robust for IoT applications. The method leverages the refined uncertainty estimation capabilities of Dirichlet distributions for classification predictions, combined with the superior OOD detection of energy‐based models. The authors propose a unique training approach and thresholding technique that enhances the precision of branch predictions, offering robustness against noise and OOD inputs. The findings demonstrate that BrevisNet surpasses existing branching techniques in training efficiency, accuracy, overall performance, and robustness.

A Novel Wiener‐Type Dynamic Neural Network Method for Large Signal Modeling of Power Amplifiers

ABSTRACTA novel Wiener‐type dynamic neural network (Wiener‐type DNN) method for large signal modeling of power amplifiers (PAs) is proposed in this paper. In this method, the proposed model structure for the PA contains two Wiener‐type DNNs to describe the input impedance and the amplification efficiency, respectively. The Wiener‐type DNN includes a simplified linear dynamic equation module and a nonlinear static equation module. For the simplification process of the linear dynamic equation, it is proposed to process fundamental frequency components of the large signal harmonic data of the PA by vector fitting. The neural network is used to implement the static equation in the Wiener‐type DNN. The formulas for training the proposed Wiener‐type DNN model of the PA using large signal data are derived. The training algorithm of the PA model is proposed to improve the training efficiency. A Motorola MOSFET PA example and a Freescale lateral double‐diffused MOSFET PA example are present to validate the proposed Wiener‐type DNN method. For the Motorola MOSFET PA, a dynamic neural network (DNN) model and a time‐delay deep neural network (TDDNN) are established for comparison. For the Freescale PA, the DNN model is used for comparison experiment. The comparison figures show that the Wiener‐type DNN model has better convergence properties than the DNN model in the nonlinear region of the PA. The established accurate PA model is conducive to the design of subsequent communication circuits.

Multi-UAV Cooperative Pursuit of a Fast-Moving Target UAV Based on the GM-TD3 Algorithm

Recently, developing multi-UAVs to cooperatively pursue a fast-moving target has become a research hotspot in the current world. Although deep reinforcement learning (DRL) has made a lot of achievements in the UAV pursuit game, there are still some problems such as high-dimensional parameter space, the ease of falling into local optimization, the long training time, and the low task success rate. To solve the above-mentioned issues, we propose an improved twin delayed deep deterministic policy gradient algorithm combining the genetic algorithm and maximum mean discrepancy method (GM-TD3) for multi-UAV cooperative pursuit of high-speed targets. Firstly, this paper combines GA-based evolutionary strategies with TD3 to generate action networks. Then, in order to avoid local optimization in the algorithm training process, the maximum mean difference (MMD) method is used to increase the diversity of the policy population in the updating process of the population parameters. Finally, by setting the sensitivity weights of the genetic memory buffer of UAV individuals, the mutation operator is improved to enhance the stability of the algorithm. In addition, this paper designs a hybrid reward function to accelerate the convergence speed of training. Through simulation experiments, we have verified that the training efficiency of the improved algorithm has been greatly improved, which can achieve faster convergence; the successful rate of the task has reached 95%, and further validated UAVs can better cooperate to complete the pursuit game task.

SIAT: Document-level Event Extraction via Spatiality-Augmented Interaction Model with Adaptive Thresholding

Document-level event extraction endeavors to automatically extract structural events from a given document. Many existing approaches focus on modeling entity interactions and decoding these interactions into events, assigning each entity as an event argument. However, these approaches encounter two primary limitations: they exclusively capture semantic dependencies to model entity interactions, overlooking the indication of the spatial distribution features of entities; they decode interactions imprecisely with a hard binary-classification boundary, potentially failing to calibrate micro differences in interactions. To overcome these limitations, we introduce a novel approach termed the S patiality-augmented I nteraction Model with A daptive T hresholding (SIAT). Our method addresses the first limitation by calculating the relative position encoding of entities to represent spatial interaction features. These features are then integrated with multi-granularity semantic interactions, enhancing the modeling of entity interactions for each entity pair. Furthermore, we introduce an adaptive event decoding mechanism, which establishes a more flexible decision boundary for different entity interactions. Additionally, an adaptive loss function for threshold learning is designed to further refine the model. Experimental results demonstrate that our proposed method achieves competitive performance compared to state-of-the-art methods on two public event extraction datasets while maintaining considerable training efficiency.

Image super-resolution based on improved ESRGAN and its application in camera calibration

Camera calibration is a key technique in the field of computer vision. The Zhang’s method is currently the most commonly used camera calibration method, but its accuracy is mainly dependent on the image quality, thusly constrained by camera hardware conditions. Considering that high-performance cameras are often expensive, a cost-effective method to improve the quality of calibration images is of great practical value. Without changing the existing hardware conditions, image super-resolution can be considered to enhance the quality of the calibration image. Image super-resolution (SR) is the process of reconstructing an image from low resolution (LR) to high resolution (HR). It can enhance the clarity and detail of the calibration images, which will be beneficial in improving the accuracy of camera calibration. However, there is very little research on the application of image super-resolution to camera calibration. Therefore, this study innovatively proposed a method and process for applying image super-resolution in camera calibration. Firstly, the ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks) was optimized and improved, followed by the proposal of the MMRSRGAN (Multi-scale Multi-adaptive-weights Residual Super-Resolution Generative Adversarial Networks). Two methods for local image super-resolution were also proposed for efficient training and testing of datasets, and two evaluation metrics based on reprojection error were proposed evaluate the effectiveness of the proposed model. Training and testing experiments of image super-resolution for camera calibration were conducted on both virtual and real image datasets. The proposed MMRSRGAN was compared with several advanced image super-resolution networks developed in recent years across multiple dimensions and indicators. The positive results have demonstrated the feasibility of applying image super-resolution to camera calibration and showcased the good performance of the MMRSRGAN.

Rapid-Learning Collaborative Pushing and Grasping via Deep Reinforcement Learning and Image Masking

When multiple objects are positioned close together or stacked, pre-grasp operations such as pushing objects can be used to create space for the grasp, thereby improving the grasping success rate. This study develops a model based on a deep Q-learning network architecture and introduces a fully convolutional network to accurately identify pixels in the workspace image that correspond to target locations for exploration. In addition, this study incorporates image masking to limit the exploration area of the robotic arm, ensuring that the agent consistently explores regions containing objects. This approach effectively addresses the sparse reward problem and improves the convergence rate of the model. Experimental results from both simulated and real-world environments show that the proposed method accelerates the learning of effective grasping strategies. When image masking is applied, the success rate in the grasping task reaches 80% after 600 iterations. The time required to reach 80% success rate is 25% shorter when image masking is used compared to when it is not used. The main finding of this study is the direct integration of image masking technique with a deep reinforcement learning (DRL) algorithm, which offers significant advancement in robotic arm control. Furthermore, this study shows that image masking technique can substantially reduce training time and improve the object grasping success rate. This innovation enables the robotic arm to better adapt to scenarios that conventional DRL methods cannot handle, thereby improving training efficiency and performance in complex and dynamic industrial applications.

Numerical investigation on the heat dissipation of phase change materials used in the high-speed train brake system

The massive heat dissipation demands of the brake system in high-speed trains pose a significant obstacle to achieving higher operation speeds. Phase change material has attracted considerable attention in various fields due to their exceptional heat dissipation capabilities, yet their utilization in the brake system of high-speed trains remains unexplored. This study aims to investigate the feasibility of phase change material application in the brake system of high-speed train. Specifically, in Case A, the introduction of phase change material resulted in a notable 21% decrease in the average temperature and a remarkable 40% reduction in the maximum temperature difference within the brake system. The latent heat of the phase change material plays a crucial role in maintaining a substantial temperature differential between the cooling components and discs, thereby enhancing heat flux in the brake system. Phase change materials exhibit superior cooling performance compared to traditional air cooling methods in the brake system. To expedite the cooling process of phase change material and facilitate its transition from liquid to solid, an optimized brake system structure utilizing phase change material was proposed. This optimized design holds promise in enhancing the overall heat dissipation efficiency of the high-speed train brake system.

The application of multimodal AI large model in the green supply chain of energy industry

With the accelerated advancements in artificial intelligence and the increasing emphasis on sustainable supply chain management, the integration of multimodal artificial intelligence (AI) into green supply chains has emerged as a critical research frontier. This study delves into the synergistic potential and challenges of combining multimodal AI, which leverages diverse data types such as text, images, and numerical data, to enhance decision-making processes in green supply chains. Through the meticulous design of a data strategy and model framework, this research establishes a sophisticated and efficient data processing and model training pipeline. The experimental results reveal that the comprehensive analysis and fusion of multimodal data significantly improve the prediction accuracy of key supply chain metrics, with observed increases in accuracy and recall rates by 12.4% and 9.8%, respectively. Additionally, the model's limitations are critically assessed, and targeted improvement strategies are proposed. The practical implications of this study are profound, offering actionable insights for the application of multimodal AI in real-world energy sector scenarios. The findings underscore the potential of this technology to optimize operations, reduce environmental impact, and drive sustainable growth in the energy industry.

Development of auditory training application for volleyball players

In various ball games, players make appropriate plays by obtaining information from their surroundings. In volleyball, players also obtain auditory information from the sound of balls being hit, people's voices, footsteps, etc. However, details such as how to use it have not been clarified. The purpose of this research is to develop a method for improving volleyball competitiveness using auditory information. The authors investigated the acoustic characteristics of good and bad sounds that athletes usually express intuitively in words, and also investigated the relationship between acoustic characteristics and athletes' subjective evaluations of the quality of the meat. We investigated the relationship. As a result, we confirmed that there are differences in the sounds that are perceived as "good" or "bad." Furthermore, we conducted acoustic measurements and acoustic analysis of serves and spikes for experienced volleyball players, and based on the analysis results, we developed an auditory training system to enable accurate discrimination of sounds during competition. In the future, we will collect logs from experiment participants who trained using the application and analyze differences in training effects due to differences in years of experience. We will also consider how to create a more efficient training system.

Beamforming-integrated neural networks for ultrasound imaging

Sparse matrix beamforming (SMB) is a computationally efficient reformulation of delay-and-sum (DAS) beamforming as a single sparse matrix multiplication. This reformulation can potentially dovetail with machine learning platforms like TensorFlow and PyTorch that already support sparse matrix operations. In this work, using SMB principles, we present the development of beamforming-integrated neural networks (BINNs) that can rationally infer ultrasound images directly from pre-beamforming channel-domain radiofrequency (RF) datasets. To demonstrate feasibility, a toy BINN was first designed with two 2D-convolution layers that were respectively placed both before and after an SMB layer. This toy BINN correctly updated kernel weights in all convolution layers, demonstrating efficiency in both training (PyTorch – 133 ms, TensorFlow – 22 ms) and inference (PyTorch – 4 ms, TensorFlow – 5 ms). As an application demonstration, another BINN with two RF-domain convolution layers, an SMB layer, and three image-domain convolution layers was designed to infer high-quality B-mode images in vivo from single-shot plane-wave channel RF data. When trained using 31-angle compounded plane wave images (3000 frames from 22 human volunteers), this BINN showed mean-square logarithmic error improvements of 21.3 % and 431 % in the inferred B-mode image quality respectively comparing to an image-to-image convolutional neural network (CNN) and an RF-to-image CNN with the same number of layers and learnable parameters (3,777). Overall, by including an SMB layer to adopt prior knowledge of DAS beamforming, BINN shows potential as a new type of informed machine learning framework for ultrasound imaging.

A path planning method based on deep reinforcement learning for AUV in complex marine environment

The potential of autonomous underwater vehicle (AUV) on future applications is significant due to advancements in autonomy and intelligence. Path planning is a critical technology for AUV to perform operational missions in complex marine environments. To this end, this paper proposes a path planning method for AUV based on deep reinforcement learning. Initially, considering actual requirements, a complex marine environment model containing underwater terrain, sonobuoy detection, and ocean currents is established. Subsequently, the corresponding state space, action space, and reward function are formulated. Furthermore, to address the inherent limitations of existing deep reinforcement learning algorithms in terms of training efficiency, a mixed experience replay (MER) strategy is proposed. This strategy aims to enhance the efficiency of sample learning by integrating prior knowledge and exploration experience. Lastly, a novel HMER-SAC algorithm for AUV path planning is proposed by integrating the Soft Actor–Critic (SAC) algorithm with the hierarchical reinforcement learning strategy and the MER strategy. The results of the simulation and experiment demonstrate that the method is capable of efficiently planning executable paths in complex marine environments and exhibits superior training efficiency, stability, and performance.

Towards automatic farrowing monitoring-A Noisy Student approach for improving detection performance of newborn piglets.

Nowadays, video monitoring of farrowing and automatic video evaluation using Deep Learning have become increasingly important in farm animal science research and open up new possibilities for addressing specific research questions like the determination of husbandry relevant indicators. A robust detection performance of newborn piglets is essential for reliably monitoring the farrowing process and to access important information about the welfare status of the sow and piglets. Although object detection algorithms are increasingly being used in various scenarios in the field of livestock farming, their usability for detecting newborn piglets has so far been limited. Challenges such as frequent animal occlusions, high overlapping rates or strong heterogeneous animal postures increase the complexity and place new demands on the detection model. Typically, new data is manually annotated to improve model performance, but the annotation effort is expensive and time-consuming. To address this problem, we propose a Noisy Student approach to automatically generate annotation information and train an improved piglet detection model. By using a teacher-student model relationship we transform the image structure and generate pseudo-labels for the object classes piglet and tail. As a result, we improve the initial detection performance of the teacher model from 0.561, 0.838, 0.672 to 0.901, 0.944, 0.922 for the performance metrics Recall, Precision and F1-score, respectively. The results of this study can be used in two ways. Firstly, the results contribute directly to the improvement of piglet detection in the context of birth monitoring systems and the evaluation of the farrowing progress. Secondly, the approach presented can be transferred to other research questions and species, thereby reducing the problem of cost-intensive annotation processes and increase training efficiency. In addition, we provide a unique dataset for the detection and evaluation of newborn piglets and sow body parts to support researchers in the task of monitoring the farrowing process.

DB-FL: DAG blockchain-enabled generalized federated dropout learning

To protect user privacy and data security, the integration of Federated Learning (FL) and blockchain has become an emerging research hotspot. However, the limited throughput and high communication complexity of traditional blockchains limit their application in large-scale FL tasks, and the synchronous traditional FL will also reduce the training efficiency. To address these issues, in this paper, we propose a Directed Acyclic Graph (DAG) blockchain-enabled generalized Federated Dropout (FD) learning strategy, which could improve the efficiency of FL while ensuring the model generalization. Specifically, the DAG maintained by multiple edge servers will guarantee the security and traceability of the data, and the Reputation-based Tips Selection Algorithm (RTSA) is proposed to reduce the blockchain consensus delay. Second, the semi-asynchronous training among Intelligent Devices (IDs) is adopted to improve the training efficiency, and a reputation-based FD technology is proposed to prevent overfitting of the model. In addition, a Hybrid Optimal Resource Allocation (HORA) algorithm is introduced to minimize the network delay. Finally, simulation results demonstrate the effectiveness and superiority of the proposed algorithms.

A joint vehicular device scheduling and uncertain resource management scheme for Federated Learning in Internet of Vehicles

Federated learning (FL) offers an effective framework for the efficient process in vehicular edge computing. However, FL encompasses the process of distributing and uploading model parameters, which are inevitably transmitted in a wireless network environment. Some challenges in FL-assisted Internet of Vehicles (IoV) sceneries gradually emerging, such as data heterogeneity, concerned device resources, and unstable communication environment, which necessitate intelligent vehicle selection schemes that accelerate training efficiency. Based on these, we consider a new scenario, specifically an FL-assisted IoV system under uncertain communication conditions, and develop an interval many-objective vehicle selection and bandwidth allocation (IMoVSBA) joint optimization scheme. This scheme takes into account computation latency, energy consumption, server utilization, and data quality, while meeting multi-criteria resource optimization requirements. Among these, server utilization is a new objective designed specifically for this joint optimization problem. For the proposed problem, a novel interval many-objective evolutionary algorithm with individual comprehensive indicator to control the evolution direction (IMaOEACI) is designed. Simulation results demonstrate that this method outperforms other schemes in terms of accuracy, training cost, and server utilization, effectively improving training efficiency in wireless channel environments and reasonably utilizing bandwidth resources. It provides significant scientific value and application potential in the field of the IoVs.

A parallel and multi-scale probabilistic temporal convolutional neural networks for forecasting the key monitoring parameters of gas turbine

As a crucial equipment in the power industry, gas turbines need effective condition monitoring techniques to maintain safe and reliable operations. Fast yet accurate long-term forecasting of the key monitoring parameters of gas turbine is vital for achieving the overall effective condition monitoring. This however poses challenges in terms of both efficacy and efficiency for conventional time series models like recurrent neural networks (RNN), since they run in a sequential manner. To this end, this paper introduces a novel parallel probabilistic time series prediction model. Particularly, the proposed model leverages the parallelism of temporal convolutional neural network (TCN) and exploits the power of latent space in the Bayesian framework. Besides, a multi-scale feature extraction strategy based on dense connections and a lagged observation strategy are presented to improve the model performance. In the comparative study against state-of-the art competitors on four classical system identification benchmarks, the proposed model significantly reduces the training time, while consistently achieving highly accurate predictions and providing appropriate uncertainty quantification. Thereafter, the proposed model is adopted to build a systematic workflow for the fast yet accurate long-term forecasting of the temperature of blade channel of gas turbine. The comprehensive results again highlight the superiority of the proposed model in terms of both the prediction quality and the training efficiency.

Effects of deep-sea water on training efficiency, locomotor function and respiratory metabolism in young and aged mice

Deep sea water (DSW) contains many trace minerals, and its applications, which include its use as drinking water, have gradually been expanding. Generally, humans tend to be lacking in mineral intake and deficiencies of trace minerals may increase the risk of several health problems. In recent years, the lack of exercise among the elderly has become an issue, leading to the onset of frailty and sarcopenia, which in turn increases the risk of dementia. Therefore, we investigated whether the daily intake of DSW-extract-added water (DSW; hardness 300 mg/L) impacted the training effect in aged mice. Treatment with DSW significantly induced a training effect in aged mice subjected to treadmill exercise. Locomotor function and metabolic capacity were also significantly increased in aged mice after DSW treatment. The results indicate that daily intake of DSW may enhance the training effect of exercise and affect locomotor performance.