Efficient Training Research Articles

Transformers significantly improve splice site prediction

Mutations that affect RNA splicing significantly impact human diversity and disease. Here we present a method using transformers, a type of machine learning model, to detect splicing from raw 45,000-nucleotide sequences. We generate embeddings with residual neural networks and apply hard attention to select splice site candidates, enabling efficient training on long sequences. Our method surpasses the leading tool, SpliceAI, in detecting splice sites in GENCODE and ENSEMBL annotations. Using extensive RNA sequencing data from an Icelandic cohort of 17,848 individuals and the Genotype-Tissue Expression (GTEx) project, our method demonstrates superior performance in detecting splice junctions compared to SpliceAI-10k (PR-AUC = 0.834 vs. PR-AUC = 0.820) and is more effective at identifying disease-related splice variants in ClinVar (PR-AUC = 0.997 vs. PR-AUC = 0.996). These advancements hold promise for improving genetic research and clinical diagnostics, potentially leading to better understanding and treatment of splicing-related diseases.

Severity Estimation of Inter-Turn Short-Circuit Fault in PMSM for Agricultural Machinery Using Bayesian Optimization and Enhanced Convolutional Neural Network Architecture

The permanent magnet synchronous motor (PMSM) is a key power component in agricultural machinery. The harsh and variable working environments encountered during the operation of agricultural machinery pose significant challenges to the safe operation of PMSMs. Early diagnosis of inter-turn short-circuit (ITSC) faults is crucial for improving the safety of the motor. In this study, a fault diagnosis method based on an improved convolutional neural network (CNN) architecture is proposed, featuring two main contributions. First, a dilated convolutional neural network is combined with residual structures, multi-scale structures, and channel attention mechanisms to enhance the training efficiency of the model and the quality of feature extraction. Second, Bayesian optimization algorithms are applied for the automatic tuning of architecture hyperparameters in deep learning models, achieving automatic optimization of the hyperparameters for the fault diagnosis model of ITSCs. To validate the effectiveness of the proposed algorithm, 17 simulated tests of ITSC fault severities were conducted under both constant conditions and dynamic conditions. The results show that the proposed model achieves the best performance regarding the validation accuracy (98.2%), standard deviation, F1 scores, and feature learning capability compared to four other models with different architectures, demonstrating the effectiveness and superiority of the algorithm.

Cooperative dual-actor proximal policy optimization algorithm for multi-robot complex control task

This paper introduces a novel multi-agent Deep Reinforcement Learning (DRL) framework named the Cooperative Dual-Actor Proximal Policy Optimization (CDA-PPO) algorithm, designed to address complex humanoid robot cooperative learning control tasks. Effective cooperation among multiple humanoid robots, particularly in scenarios involving complex walking gait control and external disturbances in dynamic environments, is a critical challenge. This is especially pertinent for tasks requiring precise coordination and control, such as joint object transportation. In various real-life scenarios, humanoid robots might need to cooperate to carry large objects in many scenarios. This capability is crucial for logistics, manufacturing, intelligent transportation, and search-and-rescue missions applications. Humanoid robots have gained significant popularity, and their use in these cooperative tasks is becoming more common. To address this challenge, we propose CDA-PPO, which introduces a learning-based communication platform between agents and employs two distinct policy networks for each agent. This dual-policy approach enhances the robots’ ability to adapt to complex interactions and maintain stability while performing intricate tasks. We demonstrate the efficacy of CDA-PPO in a cooperative object-transportation scenario, where two humanoid robots collaborate to carry a table. The experimental results show that CDA-PPO significantly outperforms traditional methods, such as Independent PPO (IPPO), Multi-Agent PPO (MAPPO), and Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (MATD3), in terms of training efficiency, stability, reward acquisition, and humanoid robot cooperative balance control with effective coordination between robots. The findings underscore the potential of CDA-PPO to advance the field of cooperative multi-agent control problems, proposing the way for future research in complex robotics applications.

FEVERLESS: Fast and Secure Vertical Federated Learning Based on XGBoost for Decentralized Labels

Vertical Federated Learning (VFL) enables multiple clients to collaboratively train a global model over vertically partitioned data without leaking private local information. Tree-based models, like XGBoost and LightGBM, have been widely used in VFL to enhance the interpretation and efficiency of training. However, there is a fundamental lack of research on how to conduct VFL securely over distributed labels. This work is the first to fill this gap by designing a novel protocol, called FEVERLESS, based on XGBoost. FEVERLESS leverages secure aggregation via information masking technique and global differential privacy provided by a fairly and randomly selected noise leader to prevent private information from being leaked in the training process. Furthermore, it provides label and data privacy against honest-but-curious adversaries even in the case of collusion of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$n - 2$</tex-math></inline-formula> out of n clients. We present a comprehensive security and efficiency analysis for our design, and the empirical results from our experiments demonstrate that FEVERLESS is fast and secure. In particular, it outperforms the solution based on additive homomorphic encryption in runtime cost and provides better accuracy than the local differential privacy approach.

Efficient, Scalable, and Sustainable DNN Training on SoC-Clustered Edge Servers

Efficient, Scalable, and Sustainable DNN Training on SoC-Clustered Edge Servers

An efficient convolutional neural network for adversarial training against adversarial attack

Convolutional neural networks (CNN) are widely used by researchers due to their extensive advantages over various applications. However, images are highly susceptible to malicious attacks using perturbations that are unrecognized even under human intervention. This causes significant security perils and challenges to CNN-related applications. In this article, an efficient adversarial training model against malevolent attacks is demonstrated. This model is highly robust to black-box malicious examples, it is processed with different malicious samples. Initially, malicious training models like fast gradient descent (FGS), recursive-FGSM (I-FGS), Deep-Fool, and Carlini and Wagner (CW) techniques are utilized that generate adversarial input by means of the CNN acknowledged to the attacker. In the experimentation process, the MNIST dataset comprising 60K and 10K training and testing grey-scale images are utilized. In the experimental section, the adversarial training model reduces the attack accuracy rate (ASR) by an average of 29.2% for different malicious inputs, when preserving the accuracy of 98.9% concerning actual images in the MNIST database. The simulation outcomes show the preeminence of the model against adversarial attacks.

Flow-models 2.2: Efficient and parallel elephant flow modeling with machine learning

This article introduces the latest version of the flow-models framework for IP network flow analysis. Key improvements include support for Dask to enable parallel computing, dataset reduction techniques for efficient training, and new modules for entropy analysis and granular flow table simulations. The codebase has been refined, with improved documentation and the incorporation of automated testing via ruff. The framework is now compatible with forthcoming releases of Python and NumPy, making it a useful resource for researchers and professionals involved in network flow analysis and machine learning-driven traffic classification.

PHYSICS PERSPECTIVES ON HUMAN LOCOMOTION: INVESTIGATING THE MECHANICS OF WALKING PATTERNS

This study investigated the gait metric differences between males and females aged 18 to 22. Our investigation revealed some significant changes in gait metrics between the two genders. Firstly, males exhibited greater leg and step lengths than females. Males exhibited wider strides, suggesting typical anthropometric variations. Except for BMI and cadence, males had more variation in practically every other gait metric, whereas females had more variability. There was a non-significant positive connection between BMI and gait metrics such as height, stride length, and step width, with BMI having a minor effect on those measures. Overall, both genders showed a distinct pattern of variability, revealing individual variances in gait analysis across the same genders. A direct correlation was observed between the Froude number and height, indicating that individuals with a higher Froude number exhibit a more dynamic and faster walking style. This phenomenon can be ascribed to the biomechanical superiority that taller individuals may possess, enabling them to generate longer strides and achieve higher speeds. A thorough study of walking speed and gait parameters revealed difficulty in finding the right balance between stability, efficiency, and speed in human movement. Understanding these connections will help people develop more efficient training programs, assistive technologies, and more efficient and safe walking in general.

Reputation-Aware Federated Learning Client Selection Based on Stochastic Integer Programming

Federated Learning(FL) has attracted wide research interest due to its potential in building machine learning models while preserving users&#x0027; data privacy. However, due to the distributive nature of FL, it is vulnerable to misbehavior from participating worker nodes. Thus, it is important to select clients to participate in FL. Recent studies on FL client selection focus on the perspective of improving model training efficiency and performance, without holistically considering potential misbehavior and the cost of hiring. To bridge this gap, we propose a first-of-its-kind reputation-aware <u>S</u>tochastic integer programming-based FL <u>C</u>lient <u>S</u>election method (SCS). It can optimally select and compensate clients with different reputation profiles. Extensive experiments show that SCS achieves the most advantageous performance-cost trade-off compared to other existing state-of-the-art approaches.

Application Research of 3D Virtual Interactive Technology in Interactive Teaching of Arts and Crafts

In the virtual reality teaching of arts and crafts, the uploaded image data often contains a lot of noise, which significantly reduces the quality of uploaded images and seriously affects the teaching effect of interactive teaching of arts and crafts based on virtual reality. For this reason, the research proposes an image denoising model based on convolutional neural network and wavelet transform, which adopts residual neural network structure and batch normalization algorithm, aiming at gradient explosion and gradient disappearance that may be caused by convolutional neural network, And the problem of low training efficiency has been optimized. The results show that when the noise standard deviation is 50, model 1 can achieve the target accuracy with only 82 iterations, while model 2 requires as many as 56 iterations. For images of different types and noise intensities, the average PSNR values of model 1 are 29.3dB, 30.5dB and 28.9dB, respectively. In addition, by applying Model 1 to VR teaching of arts and crafts, the average score of class 1 is 9.3 points higher than that of class 2. The proposed model effectively optimizes the virtual reality teaching process of arts and crafts, and provides a useful contribution to the application of virtual reality in the field of education.

Intelligent Identification of Foodborne Pathogenic Bacteria by Self-Transfer Deep Learning and Ensemble Prediction Based on Single-Cell Raman Spectrum

Foodborne pathogenic infections pose a significant threat to human health. Accurate detection of foodborne diseases is essential in preventing disease transmission. This study proposed an AI model for precisely identifying foodborne pathogenic bacteria based on single-cell Raman spectrum. Self-transfer deep learning and ensemble prediction algorithms had been incorporated into the model framework to improve training efficiency and predictive performance, significantly improving prediction results. Our model can identify simultaneously gram-negative and positive, genus, species of foodborne pathogenic bacteria with an accuracy over 99.99%, as well as recognized strain with over 99.49%. At all four classification levels, unprecedented excellent predictive performance had been achieved. This advancement holds practical significance for medical detection and diagnosis of foodborne diseases by reducing false negatives.

Robust and energy-efficient RPL optimization algorithm with scalable deep reinforcement learning for IIoT

The increasing complexity and quantity of the Industrial Internet of Things (IIoT) pose new challenges to the traditional routing protocol for low-power and lossy networks (RPL) in terms of dynamic management, data transmission reliability, and energy efficiency optimization. This paper proposes a scalable deep reinforcement learning (DRL) algorithm with a multi-attention actor double critic model for routing optimization (MADC) to meet the requirements of IIoT for efficient and intelligent routing decisions while improving data transmission reliability and energy efficiency. Specifically, MADC employs the centralized training and decentralized execution (CTDE) learning paradigm to decouple the model’s training and inference tasks, which reduces the difficulty and computational cost of model learning and improves the training efficiency. In addition, a lightweight actor network based on multi-scale convolutional attention mechanism is designed in MADC, which can provide intelligent and real-time decision-making capabilities for resource-constrained nodes with low computational and storage complexities. Moreover, a scalable critic network utilizing multiple attention mechanisms is proposed. It is not only suitable for dynamic and changing network environments but also can more comprehensively and accurately evaluate local observation states, providing more accurate and efficient guidance for model optimization. Furthermore, MADC incorporates a double critic network architecture to mitigate potential overestimation issues during training, thereby ensuring the model’s robustness and reliability. Simulation results demonstrate that MADC outperforms existing RPL optimization algorithms in terms of energy efficiency, data transmission reliability, and adaptability.

Recent progress, challenges and future propects of applied deep reinforcement learning : A practicial perspecive in path planning

Path planning is one of the most crucial elements in the field of robotics, such as autonomous driving, minimally invasive surgery and logistics distribution. This review begins by summarizing the limitations of conventional path planning methods and recent work on DRL-based path planning methods. Subsequently, the paper systematically reviews the construction of key elements of DRL methods in recent work, with the aim of assisting readers in comprehending the foundation of DRL research, along with the underlying logic and considerations from a practical perspective. Facing issues of sparse rewards and the exploration-exploitation balance during the practical training process, the paper reviews enhancement methods for training efficiency and optimization results in DRL path planning. In the end, the paper summarizes the current research limitations and challenges in practical path planning applications, followed by future research directions.

Sensor attack online classification for UAVs using machine learning

Unmanned Aerial Vehicle (UAV) sensors play a vital role in maintaining flight safety and stability. However, the increasing frequency and complexity of sensor attacks have emerged as a critical threat to UAV systems. The current lack of robust multi-classification methods for detecting sensor attacks limits the effectiveness and completeness of existing defense strategies. This research addresses these challenges by leveraging machine learning (ML) techniques to classify various sensor attacks using heterogeneous sensor data and control parameters, thereby enhancing UAV system security. In this study, we design and implement multiple sensor attack scenarios targeting gyroscopes, accelerometers, barometers, and GPS. Comprehensive datasets are collected during UAV flight, integrating diverse sensor readings, flight states, and control parameters. By analyzing the characteristics of sensor attacks and their impact on position estimation and attitude control, we identify and extract key features. To optimize the classification model, we employ feature importance analysis, correlation analysis, and ablation experiments, significantly reducing data dimensionality and enhancing model training efficiency. The experimental results demonstrate the proposed ML-based multi-classification model’s superior performance, achieving a detection rate of 89.38%, significantly outperforming traditional single-attack detection methods in terms of generalization capability. Our approach efficiently handles complex multi-sensor attack scenarios. Moreover, deploying the optimized model on UAV firmware enables real-time monitoring and classification, achieving an online detection rate of 74% with a response time of approximately 0.495ms per detection. The model’s lightweight design, requiring only 48KB of storage, makes it ideal for resource-constrained UAV environments. These contributions highlight the potential of our approach to enhance real-time anomaly detection and improve UAV system resilience against diverse sensor attacks.

End to end polysemantic cooperative mixed task trainer for UAV target detection

With the rapid advancement and application of Unmanned Aerial Vehicles (UAVs), target detection in urban scenes has made significant progress. Achieving precise 3D reconstruction from oblique imagery is essential for accurate urban object detection in UAV images. However, challenges persist due to low detection accuracy caused by subtle target features, complex backgrounds, and the prevalence of small targets. To address these issues, we introduce the Polysemantic Cooperative Detection Transformer (Pc-DETR), a novel end-to-end UAV image target detection network. Our primary innovation, the Polysemantic Transformer (PoT) Backbone, enhances visual representation by leveraging contextual information to guide a dynamic attention matrix. This matrix, formed through convolutions, captures both static and dynamic features, resulting in superior detection. Additionally, we propose the Polysemantic Cooperative Mixed-Task Training scheme, which employs multiple auxiliary heads for diverse label assignments, boosting the encoder’s learning capacity. This approach customizes queries and optimizes training efficiency without increasing inference costs. Comparative experiments show that Pc-DETR achieves a 3% improvement in detection accuracy over the current state-of-the-art MFEFNet, setting a new benchmark in UAV image detection and advancing methodologies for intelligent UAV surveillance systems.

AutoLDT: a lightweight spatio-temporal decoupling transformer framework with AutoML method for time series classification

Time series classification finds widespread applications in civil, industrial, and military fields, while the classification performance of time series models has been improving with the recent development of deep learning. However, the issues of feature extraction effectiveness, model complexity, and model design uncertainty constrain the further development of time series classification. To address the above issues, we propose a Lightweight Spatio-Temporal Decoupling Transformer framework based on Automated Machine Learning technique (AutoLDT). The framework proposes a novel lightweight Transformer with fuzzy position encoding, TS-separable linear self-attention mechanism, and convolutional feedforward network, which mine the temporal and spatial features, as well as the local and global relationship of time series. Fuzzy positional encoding integrates fuzzy ideas to enhance the generalization performance of model information mining. TS-separable linear self-attention mechanism and convolutional feedforward network achieve feature extraction in a lightweight way by decoupling temporal and spatial features of time series. Notably, we adopt the Covariance Matrix Adaptation Evolution Strategy and global adaptive pruning technique to realize automated network structure design, which further improves the model training efficiency and automation, and avoids the uncertainty problem of network design. Finally, we validate the effectiveness of the proposed framework on publicly available UCR and UEA time series datasets. The experimental results show that the proposed framework not only improves the model classification performance in a lightweight way but also dramatically improves the model training efficiency.

Reliable federated learning based on delayed gradient aggregation for intelligent connected vehicles

As an organic combination of the Internet of Vehicles and intelligent vehicles, Intelligent Connected Vehicles (ICVs) have very high research and application value. Traditional data application methods require the local aggregation of sensitive user data, which poses a threat to user data privacy. Federated learning (FL) is a promising machine learning method that leverages distributed, personalized datasets to enhance performance while preserving user privacy. However, in mobile environments, unreliable client data can degrade the global model, reducing accuracy. Additionally, the mobility of ICVs can destabilize the training process, prolonging model updates and diminishing aggregation accuracy. To address these challenges, this paper proposes a dynamic asynchronous aggregation method that improves both reliability and training efficiency in FL for mobile networks. Therefore, it becomes crucial to find reliable aggregation of mobile device participation in FL tasks. To this end, we propose a reliable FL scheme, which only selects reliable mobile devices to participate in model aggregation to improve the generalization ability of the model. In addition, we design a dynamic asynchronous aggregation method based on reputation scores without affecting the model. Reduce model training time without compromising performance. Through experimental analysis, it is proved that this method can improve the reliability and effectiveness of FL tasks in mobile networks.

The Planning and Design of Digital Smart Campus

Under the background of “Digital Liaoning, Smart Province,” as the application of a new generation of network information technology in colleges and universities, the digital smart campus is an important part of digital Liaoning education informationization. Combined with the overall actual situation of the university campus, in line with the principle of “People-oriented, Service-oriented, Information Interoperability and Data Sharing,” through the idea of “management + service,” the traditional application system with management as the core to the service as the core is realized. On the basis of digital information and networks, a new digital smart campus is built based on several information technologies. From the aspects of architecture design, platform and application integration, systematic overall design and scientific and reasonable deployment, to create an efficient one-stop digital intelligent information service platform to fully optimize the use of digital information resources, to provide timely, accurate and efficient campus information services for teachers, students and staff, and to provide cross-departmental business management and convenient information services for the whole school users through the virtual digital means for display and publicity. The overall planning of the campus is carried out step by step, and the campus network, digitalization, virtualization and intelligence are gradually realized. Expand the time and space dimension of the real campus, improve the overall operation efficiency, and then improve the quality of talent training, management level and management efficiency.

From policy to practice: decoding the evolution of K-12 English teacher training course selection in Zhejiang through topic modelling analysis (2014–2023)

ABSTRACT This article aimed to strengthen training programmes in Zhejiang by aligning them with educational policies to respond to ongoing reforms drawing on Latent Dirichlet allocation (LDA) topic modelling and content analysis. The topic modelling approach uncovers dominant training themes and trends by analysing extensive unstructured training topics from an online training platform in Zhejiang, thus bridging the gap between theoretical research and practical application. This approach offers data-driven insights, supporting dynamic tracking of educational trends in real-time for timely policy and curriculum adjustments. Ultimately, the article argues that real-time data and stakeholder feedback should be integrated to ensure educational policies align closely with actual teachers’ needs. However, due to data limitations, the article acknowledges a challenge in fully utilizing topic modelling and suggests combining it with qualitative analysis to gain a deeper understanding of educational practices. The Incorporation of a topic modelling approach into teacher education research could contribute significantly to training efficiency and policy relevance, fostering an adaptive educational policy environment.

Depth-Wise Convolutions in Vision Transformers for efficient training on small datasets

The Vision Transformer (ViT) leverages the Transformer’s encoder to capture global information by dividing images into patches and achieves superior performance across various computer vision tasks. However, the self-attention mechanism of ViT captures the global context from the outset, overlooking the inherent relationships between neighboring pixels in images or videos. Transformers mainly focus on global information while ignoring the fine-grained local details. Consequently, ViT lacks inductive bias during image or video dataset training. In contrast, convolutional neural networks (CNNs), with their reliance on local filters, possess an inherent inductive bias, making them more efficient and quicker to converge than ViT with less data. In this paper, we present a lightweight Depth-Wise Convolution module as a shortcut in ViT models, bypassing entire Transformer blocks to ensure the models capture both local and global information with minimal overhead. Additionally, we introduce two architecture variants, allowing the Depth-Wise Convolution modules to be applied to multiple Transformer blocks for parameter savings, and incorporating independent parallel Depth-Wise Convolution modules with different kernels to enhance the acquisition of local information. The proposed approach significantly boosts the performance of ViT models on image classification, object detection, and instance segmentation by a large margin, especially on small datasets, as evaluated on CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet for image classification, and COCO for object detection and instance segmentation. The source code can be accessed at https://github.com/ZTX-100/Efficient_ViT_with_DW.