Model Training Efficiency Research Articles

Machine learning-based EEG analysis for enhanced schizophrenia classification and diagnosis

Abstract. Schizophrenia can have a significant impact on patients' lives, studies, and work. Patients may experience delusions, hallucinations, disorganized thinking, and abnormal behavior, among other symptoms. Therefore, research related to schizophrenia is of great importance. This paper proposes the design of a classifier based on machine learning for diagnosing schizophrenia. The classifier extracts N100 features, P300 features, and power features across different frequency bands from both time-domain and frequency-domain characteristics from Electroencephalography (EEG) signals. Given the small dataset, a Support Vector Machine (SVM) machine learning algorithm was chosen to process and classify the selected features. To address the limitations of SVM in handling high-dimensional data and nonlinear problems, this study introduces a comprehensive improvement method based on Bayesian optimization, Recursive Feature Elimination (RFE), and data augmentation. Bayesian optimization was used to find the best combination of hyperparameters for the model, thereby improving its performance; RFE was employed to assess feature importance and remove the least important features, enhancing the model's training efficiency and generalization capability. Additionally, data augmentation was included to increase the sample size and introduce diversity, thereby improving the model's robustness. The study found that these methods effectively improved classification accuracy and generalization ability.

Neural Machine Translation (NMT): Deep learning approaches through Neural Network Models

Abstract. This paper explores the significant advancements in Neural Machine Translation (NMT) models, focusing on the impact of different architectures, training methodologies, and optimization techniques on translation quality. The study contrasts the performance of Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and the Transformer model, highlighting the superior capabilities of the Transformer in handling long-range dependencies and providing contextually accurate translations. Key optimization techniques, such as learning rate scheduling, dropout regularization, and gradient clipping, are discussed in detail, emphasizing their roles in enhancing model performance and training efficiency. Furthermore, the paper presents a comparative analysis of NMT and traditional Statistical Machine Translation (SMT) systems, showcasing NMT's superior BLEU scores and fluency. The application of model distillation is also examined, demonstrating how smaller models can achieve high performance with reduced computational resources. These findings underscore the transformative potential of NMT in achieving state-of-the-art translation quality and efficiency.

AMW-YOLOv8n: Road Scene Object Detection Based on an Improved YOLOv8

This study introduces an improved YOLOv8 model tailored for detecting objects in road scenes. To overcome the limitations of standard convolution operations in adapting to varying targets, we introduce Adaptive Kernel Convolution (AKconv). AKconv dynamically adjusts the convolution kernel’s shape and size, enhancing the backbone network’s feature extraction capabilities and improving feature representation across different scales. Additionally, we employ a Multi-Scale Dilated Attention (MSDA) mechanism to focus on key target features, further enhancing feature representation. To address the challenge posed by YOLOv8’s large down sampling factor, which limits the learning of small target features in deeper feature maps, we add a small target detection layer. Finally, to improve model training efficiency, we introduce a regression loss function with a Wise-IoU dynamic non-monotonic focusing mechanism. With these enhancements, our improved YOLOv8 model excels in road scene object detection tasks, achieving a 5.6 percentage point improvement in average precision over the original YOLOv8n on real road datasets.

Fault Diagnosis Method for Hydropower Units Based on Dynamic Mode Decomposition and the Hiking Optimization Algorithm–Extreme Learning Machine

The diagnosis of vibration faults in hydropower units is essential for ensuring the safe and stable operation of these systems. This paper proposes a fault diagnosis method for hydropower units that combines Dynamic Mode Decomposition (DMD) with an optimized Extreme Learning Machine (ELM) utilizing the Hiking Optimization Algorithm (HOA). To address the issue of noise interference in the vibration signals of hydropower units, this study employs DMD technology alongside a thresholding technique for noise reduction, demonstrating its effectiveness through comparative trials. Furthermore, to facilitate a thorough analysis of the operational status of hydropower units, this paper extracts multidimensional features from denoised signals. To improve the efficiency of model training, Principal Component Analysis (PCA) is applied to streamline the data. Given that the weights and biases of the ELM are generated randomly, which may impact the model’s stability and generalization capabilities, the HOA is introduced for optimization. The HOA-ELM model achieved a classification accuracy of 95.83%. A comparative analysis with alternative models substantiates the superior performance of the HOA-ELM model in the fault diagnosis of hydropower units.

An automatic and efficient fault diagnosis strategy for air conditioning units by combining attention mechanisms

The air conditioning system is a very important energy-consuming device in the field of construction. Timely troubleshooting of air conditioning chillers is an important aspect of reducing building energy consumption. Aiming at the problems of many model parameters, large amount of calculation and strong dependence of diagnostic performance on input features in data-driven diagnostic methods, this paper proposes an efficient fault diagnosis strategy combining RepVGG network and SENet attention mechanism. The multi-parameter feature signal is converted into a two-dimensional matrix to obtain multiple feature images. Based on the SENet attention mechanism, the multi-channel weights of the images are extracted, and the weighted feature image is input into the RepVGG network for feature depth mining and classification. Therefore, the method proposed in this paper greatly reduces the calculation amount of parameters, improves the representativeness of signal features to faults and the efficiency of model training. The effectiveness of the method is validated using the ASHRAE RP-1043 chiller unit dataset. The results show that the accuracy of this method reaches 96.55%, which is significantly better than the comparison methods PSO-ELM(96.4%), DT(95.1%), RepVGG(91.6%), ELM(87.6%) and BP(80.8%).

Enhanced Privacy-Preserving Federated Learning with Exit Tolerance and Verifiable Aggregation

With the escalating importance of data privacy, Federated Learning (FL) has drawn significant attention as a machine learning framework that facilitates collaborative model training among multiple users without the need for direct data sharing. This paper presents an enhanced Federated Learning scheme designed to bolster user privacy protection while maintaining the efficiency and performance of model training. Our approach integrates the use of random seeds and One-Time Password (OTP) technology to reinforce data encryption, and employs an advanced masking mechanism coupled with bilinear pairing for verification, thereby enhancing the security of the aggregation process. Additionally, our design accommodates user exits during the training process without compromising the overall training outcome. Through rigorous experimental analysis, we have demonstrated the effectiveness of our scheme, showcasing its acceptable computational and communication overheads. This research introduces a novel solution to privacy preservation challenges in Federated Learning, laying a solid foundation for the advancement of related technologies.

Cross-Task Rumor Detection: Model Optimization Based on Model Transfer Learning and Graph Convolutional Neural Networks (GCNs)

With the widespread adoption of social media, the rapid dissemination of rumors poses a severe threat to public perception and social stability, emerging as a major challenge confronting society. Hence, the development of efficient and accurate rumor detection models has become an urgent need. Given the challenges of rumor detection tasks, including data scarcity, feature complexity, and difficulties in cross-task knowledge transfer, this paper proposes a BERT–GCN–Transfer Learning model, an integrated rumor detection model that combines BERT (Bidirectional Encoder Representations from Transformers), Graph Convolutional Networks (GCNs), and transfer learning techniques. By harnessing BERT’s robust text representation capabilities, the GCN’s feature extraction prowess on graph-structured data, and the advantage of transfer learning in cross-task knowledge sharing, the model achieves effective rumor detection on social media platforms. Experimental results indicate that this model achieves accuracies of 0.878 and 0.892 on the Twitter15 and Twitter16 datasets, respectively, significantly enhancing the accuracy of rumor detection compared to baseline models. Moreover, it greatly improves the efficiency of model training.

PreparedLLM: effective pre-pretraining framework for domain-specific large language models

ABSTRACT The direct application of large language models (LLMs) to specific domain tasks frequently encounters challenges due to the scarcity of domain data, variations in domain semantics, and the complexity of domain knowledge. Further pretraining of advanced foundational models on extensive domain-specific corpora can infuse these models with domain-specific knowledge and enhancing their ability of solving domain-specific tasks. However, the development of most domain-specific models focuses primarily on collecting large-scale domain data, often overlooking the crucial optimization of the pre-pretraining stage, which significantly impacts both model performance and training efficiency. This paper introduces PRE -PretrAining FRamEwork for Domain-specific Large Language Models (PreparedLLM), a framework designed to enhance the pre-pretraining stage for domain specialization of LLMs. PreparedLLM employs advanced techniques in data recipe, data cleaning, vocabulary expansion, and embedding initialization. These techniques are implemented to optimize both the composition and quality of the training data, and to enhance understanding of domain terminologies and concepts, as well as improve token embedding initialization. Utilizing the geoscience domain as a case study, this paper applies PreparedLLM for the domain specialization of the Llama, a widely recognized general-purpose LLM. Experimental results demonstrate that PreparedLLM enhances model convergence speed, training speed, inference speed, the text volume of the context window, and overall performance in domain specialization. The utilization of PreparedLLM in developing domain-specific LLMs has significantly increased performance while reducing both time and resource investment. The case study provides valuable insights into the development of domain-specific LLMs.

A fault diagnosis method for analog circuits based on EEMD-PSO-SVM

Analog circuit is an crucial component of electronic equipment, and the ability to diagnose its fault state quickly and accurately is essential for ensuring the safety and reliability of these electronic equipment. This paper addresses the problems of low diagnostic accuracy and the difficulties associated with model parameter selection in traditional fault diagnosis methods, particularly when dealing with nonlinear and non-stationary fault signals. A fault diagnosis method for analog circuits is proposed, which utilizes Ensemble Empirical Mode Decomposition (EEMD) for features extraction, the Maximum Information Coefficient algorithm (MIC) for features selection, and Particle Swarm Optimization (PSO) for optimizing Support Vector Machine (SVM) classification. Firstly, EEMD is used to adaptively decompose the fault signals in the circuit to extract multi-scale fault features. Secondly, the extracted features are quantitatively evaluated using the Pearson correlation coefficient and energy value analysis, leading to the construction of a fault feature vector is constructed. On this basis, the MIC feature selection algorithm is applied to further optimize the feature vector. Finally, an efficient fault classification model is developed by optimizing the hyperparameters of the SVM model using PSO. The simulation results show that the proposed method effectively overcome the problems of model complexity and low classification accuracy caused by improper selection of wavelet basis function. The accuracy of fault diagnosis and the efficiency of model training are significantly superior to those of traditional methods.

Improved semantic-guided network for skeleton-based action recognition

A fundamental issue in skeleton-based action recognition is the extraction of useful features from skeleton joints. Unfortunately, the current state-of-the-art models for this task have a tendency to be overly complex and parameterized, which results in low model training and inference time efficiency for large-scale datasets. In this work, we develop a simple but yet an efficient baseline for skeleton-based Human Action Recognition (HAR). The architecture is based on adaptive GCNs (Graph Convolutional Networks) to capture the complex interconnections within skeletal structures automatically without the need of a predefined topology. The GCNs are followed and empowered with an attention mechanism to learn more informative representations. This paper reports interesting accuracy on a large-scale dataset NTU-RGB+D 60, 89.7% and 95.0% on respectively Cross-Subject, and Cross-View benchmarks. On NTU-RGB+D 120, 84.6% and 85.8% over Cross-Subject and Cross-Setup settings, respectively. This work provides an improvement of the existing model SGN (Semantic-Guided Neural Networks) when extracting more discriminant spatial and temporal features.

Detection of Road Risk Sources Based on Multi-Scale Lightweight Networks.

Timely discovery and disposal of road risk sources constitute the cornerstone of road operation safety. Presently, the detection of road risk sources frequently relies on manual inspections via inspection vehicles, a process that is both inefficient and time-consuming. To tackle this challenge, this paper introduces a novel automated approach for detecting road risk sources, termed the multi-scale lightweight network (MSLN). This method primarily focuses on identifying road surfaces, potholes, and scattered objects. To mitigate the influence of real-world factors such as noise and uneven brightness on test results, pavement images were carefully collected. Initially, the collected images underwent grayscale processing. Subsequently, the median filtering algorithm was employed to filter out noise interference. Furthermore, adaptive histogram equalization techniques were utilized to enhance the visibility of cracks and the road background. Following these preprocessing steps, the MSLN model was deployed for the detection of road risk sources. Addressing the challenges associated with two-stage network models, such as prolonged training and testing times, as well as deployment difficulties, this study adopted the lightweight feature extraction network MobileNetV2. Additionally, transfer learning was incorporated to elevate the model's training efficiency. Moreover, this paper established a mapping relationship model that transitions from the world coordinate system to the pixel coordinate system. This model enables the calculation of risk source dimensions based on detection outcomes. Experimental results reveal that the MSLN model exhibits a notably faster convergence rate. This enhanced convergence not only boosts training speed but also elevates the precision of risk source detection. Furthermore, the proposed mapping relationship coordinate transformation model proves highly effective in determining the scale of risk sources.

A new short-term wind power prediction methodology based on linear and nonlinear hybrid models

Fast and accurate wind power prediction is of great significance for grid planning. However, wind power dataset tends to be highly stochastic and volatile, while showing more stable seasonal and cyclical changes, which is a linear and nonlinear superposition features, and it is difficult to use a single linear or nonlinear model to make accurate predictions. Considering these essential features of wind power data, a short-term wind power prediction method based on linear and nonlinear hybrid models is proposed in this paper. First, the complete ensemble empirical mode decomposition (CEEMD) is used to decompose and reconstruct the wind speed and direction in the original dataset to reduce the volatility. Then, to capture the linear features in the dataset, the autoregressive integrated moving average model with exogenous input variables (AIRMAX) is used for linear modeling, and the sparrow search algorithm optimized gated recurrent unit (SSA-GRU) is used to model the nonlinear features. At the same time, to improve the efficiency of model training, federated learning (FL) is utilized for distributed model training. Finally, the ARIMAX and SSA-GRU models are weighted and summed the prediction results to gain the final prediction results using equal weighted averaging (EWA), minimum sum of squares error (MSSE), and maximum grey relational analysis (MGRA), which are commonly used to compute the weights of the combined models, respectively. The wind power dataset from a wind farm in northwest China from January 1, 2022 to October 31, 2022 are used for experiments and analysis, and the results show that the root mean square error (RMSE), R-square (R2), and accuracy (ACC) of the proposed method are 11.944, 0.987, and 0.613, respectively, which are better than all the compared models. The main contribution of this paper is twofold, on the one hand, it indirectly proves that the wind power dataset is formed by the superposition of linear and nonlinear features, and on the other hand, it puts forward a theoretical model and research paradigm for predicting short-term wind power and presents a scientific basis for grid scheduling.

A method for extracting buildings from remote sensing images based on 3DJA-UNet3+

Building extraction aims to extract building pixels from remote sensing imagery, which plays a significant role in urban planning, dynamic urban monitoring, and many other applications. UNet3+ is widely applied in building extraction from remote sensing images. However, it still faces issues such as low segmentation accuracy, imprecise boundary delineation, and the complexity of network models. Therefore, based on the UNet3+ model, this paper proposes a 3D Joint Attention (3DJA) module that effectively enhances the correlation between local and global features, obtaining more accurate object semantic information and enhancing feature representation. The 3DJA module models semantic interdependence in the vertical and horizontal dimensions to obtain feature map spatial encoding information, as well as in the channel dimensions to increase the correlation between dependent channel graphs. In addition, a bottleneck module is constructed to reduce the number of network parameters and improve model training efficiency. Many experiments are conducted on publicly accessible WHU,INRIA and Massachusetts building dataset, and the benchmarks, BOMSC-Net, CVNet, SCA-Net, SPCL-Net, ACMFNet, MFCF-Net models are selected for comparison with the 3DJA-UNet3+ model proposed in this paper. The experimental results show that 3DJA-UNet3+ achieves competitive results in three evaluation indicators: overall accuracy, mean intersection over union, and F1-score. The code will be available at https://github.com/EnjiLi/3DJA-UNet3Plus.

Research and application of short-term load forecasting based on CEEMDAN-LSTM modeling

Accurate short-term load forecasting plays a crucial role in guiding and regulating the operation of power companies. In this study, we propose a combined prediction model based on Long Short-Term Memory (LSTM) and Improved Whale Optimization Algorithm (IWOA) to enhance prediction accuracy and address the limitations of existing single methods. The parameter optimization of the LSTM model requires significant computing resources and time. To tackle this issue, we introduced the Whale Optimization Algorithm (WOA) and improved the algorithm through the roulette method to form a whale optimization algorithm that enhances its global search capability and avoids falling into local optimal solutions. Additionally, in order to handle the nonlinearity and non-smoothness of the input data of the model, we utilized fully integrated Empirical Mode Decomposition with Adaptive Noise (CEEMDAN) in this study to improve the efficiency of model training. By combining these techniques, we constructed a new CEEMDAN-IWOA-LSTM combined prediction model. The simulation results in MATLAB demonstrate that the prediction accuracy of this model reaches 99.05 %, which surpasses other prediction models in all evaluation indexes.

Clarity in Wireless Quantum Communication: Exploring Explainable AI for Signal Modulation Classification

The advent of artificial intelligence (AI) and edge computing technologies has ushered in a new era for automatic modulation classification (AMC), with a particular focus on deep learning (DL)-based approaches. These AI-based AMC systems, deployed at edge devices, have shown remarkable advancements in recognizing and classifying a wide range of wireless signals. However, a significant challenge in this domain lies in the lack of adequate explanations for the models employed, which, in turn, hampers model accuracy and training efficiency. Consequently, the practical applications and further enhancements of these models are constrained. To address this limitation, researchers have started to explore interpretable methods for the technical analysis of DL-based AMC systems. In this paper, we delve into the subject, leveraging recent developments in interpretable methods, to provide a comprehensive review of applicable techniques and to outline the existing research challenges in the field of interpretable AMC. Moreover, we introduce a novel interpretable AI-based AMC framework designed to augment the interpretability of AMC outcomes. This framework achieves this by elucidating the contribution of wireless signal features to the training process of DL networks. In addition, we explore the potential impact of quantum computing on AMC systems. Quantum computing, with its ability to process complex computations at unprecedented speeds, offers a promising avenue for enhancing the efficiency and accuracy of AMC. By integrating quantum algorithms with AI-based AMC frameworks, it is possible to significantly improve the performance of wireless signal classification, especially in environments with high signal variability and noise. Our experimental results demonstrate that the proposed approach possesses the capacity to unravel the classification mechanisms hidden within opaque auto-modulated classification networks. Furthermore, it exhibits the potential to facilitate the training of networks on edge devices with reduced energy consumption and enhanced classification accuracy, thereby addressing critical challenges in the field of wireless signal classification. Additionally, the integration of quantum computing techniques holds the promise of further advancing the capabilities of AMC systems, paving the way for more robust and efficient wireless communication technologies.

CycPeptMP: enhancing membrane permeability prediction of cyclic peptides with multi-level molecular features and data augmentation.

Cyclic peptides are versatile therapeutic agents that boast high binding affinity, minimal toxicity, and the potential to engage challenging protein targets. However, the pharmaceutical utility of cyclic peptides is limited by their low membrane permeability-an essential indicator of oral bioavailability and intracellular targeting. Current machine learning-based models of cyclic peptide permeability show variable performance owing to the limitations of experimental data. Furthermore, these methods use features derived from the whole molecule that have traditionally been used to predict small molecules and ignore the unique structural properties of cyclic peptides. This study presents CycPeptMP: an accurate and efficient method to predict cyclic peptide membrane permeability. We designed features for cyclic peptides at the atom-, monomer-, and peptide-levels and seamlessly integrated these into a fusion model using deep learning technology. Additionally, we applied various data augmentation techniques to enhance model training efficiency using the latest data. The fusion model exhibited excellent prediction performance for the logarithm of permeability, with a mean absolute error of $0.355$ and correlation coefficient of $0.883$. Ablation studies demonstrated that all feature levels contributed and were relatively essential to predicting membrane permeability, confirming the effectiveness of augmentation to improve prediction accuracy. A comparison with a molecular dynamics-based method showed that CycPeptMP accurately predicted peptide permeability, which is otherwise difficult to predict using simulations.

Imbalanced data fault diagnosis method for nuclear power plants based on convolutional variational autoencoding Wasserstein generative adversarial network and random forest

Data-driven fault diagnosis techniques are significant for the stable operation of nuclear power plants (NPPs). However, in practical applications, the fault diagnosis of NPPs usually faces imbalance data problems with small fault samples and much redundant data which results in low model training efficiency and poor generalization performance. Thus, this paper proposes a convolutional variational autoencoding gradient-penalty Wasserstein generative adversarial network with random forest (CVGR) to reduce the impact of imbalanced samples on fault diagnosis. Firstly, a feature selection method based on the random forest is used to identify the most relevant measurements and reduce the impact of redundant data on fault diagnosis. Then, variational autoencoding is introduced into gradient-penalty Wasserstein generative adversarial to effectively extract original sample features and generate high-quality samples with high rationality and diversity. In addition, the convolutional neural network is used to extract the features of mixed samples to realize intelligent fault diagnosis. Finally, several experiments based on the Fuqing Unit 2 full-scope simulator under different operating conditions are used to validate the performance of the CVGR in data enhancement and intelligent fault diagnosis. The results show that the proposed method can effectively mitigate the imbalance data problem, which gives insights into intelligent fault diagnosis of NPPs.

Integrated CNN and Federated Learning for COVID-19 Detection on Chest X-Ray Images.

Currently, Coronavirus Disease 2019 (COVID-19) is still endangering world health and safety and deep learning (DL) is expected to be the most powerful method for efficient detection of COVID-19. However, patients' privacy concerns prohibit data sharing between medical institutions, leading to unexpected performance of deep neural network (DNN) models. Fortunately, federated learning (FL), as a novel paradigm, allows participating clients to collaboratively train models without exposing source data outside original location. Nevertheless, the current FL-based COVID-19 detection methods prefer optimizing secondary objectives including delay, energy consumption and privacy, while few works focus on improving the model accuracy and stability. In this paper, we propose a federated learning framework with dynamic focus for COVID-19 detection on CXR images, named FedFocus. Specifically, to improve the training efficiency and accuracy, the training loss of each model is taken as the basis for parameter aggregation weights. As training layer deepens, a constantly updated dynamic factor is designed to stabilize the aggregation process. In addition, to highly restore the real dataset, the training sets in our experiments are divided based on the population and the infection of three real cities. Extensive experiments conducted on the real-world CXR images dataset demonstrate that FedFocus outperforms the baselines in model training efficiency, accuracy and stability.

Utilizing excitation-emission matrix fluorescence spectroscopy and convolutional neural networks for dark tea brand and aging period identification

This study utilized excitation-emission matrix (EEM) fluorescence spectroscopy in conjunction with convolutional neural networks (CNNs) to identify the dark tea brands and their aging periods, which were both key factors of market value. The adaptation of EEM data to the CNN model was examined, encompassing the configuration of the data structure and the implementation of data augmentation. Three typical CNN models including AlexNet, GoogLeNet, and ResNet were evaluated. ResNet was selected because it demonstrated the highest classification accuracy on the test set and superior clustering performance in the t-SNE visualization of extracted features. Analysis of feature contribution suggested that the CNN model exhibited enhanced sensitivity towards the boundaries of the fluorescence region. Furthermore, transfer learning was proved to be effective in facilitating adaptation of dark tea recognition models to databases with different sample sizes, which was meaningful for the application of this method in dark tea market supervision. Finally, the multi-task model was developed and achieved a 96.9 % accuracy in brand identification and 87.5 % in aging period determination, suggesting the advantages of multi-task learning in enhancing model generalization and training efficiency.

Channel Adaptive and Sparsity Personalized Federated Learning for Privacy Protection in Smart Healthcare Systems.

With the booming development of Smart Healthcare Systems (SHSs), employing federated learning (FL) in SHS devices has become a research hotspot. FL, as a distributed learning framework, can train models without sharing the original data among users, and then protect the user privacy. Existing research has proposed many methods to improve the security and efficiency of FL, which may not fully consider the characteristics of SHSs. Specifically, the requirements of privacy protection and efficiency pose significant challenges to FL. Current studies have struggled to balance privacy security and efficiency, and the degradation of model training efficiency in SHSs can be critical to patient health. Therefore, to improve the privacy protection of healthcare data and ensure communication efficiency, this work proposes a novel personalized FL framework based on Communication quality and Adaptive Sparsification (pFedCAS). In order to achieve privacy protection, a control unit is proposed and introduced to adjust the sparsity of the local model adaptively. To further improve the training efficiency, a selection unit is added during global model aggregation to select suitable clients for parameter updates. Finally, we validate the proposed method operated on the HAM10000 dataset. Simulation results validate that pFedCAS can not only improve privacy protection, but also gain an improvement of 15% in training accuracy and a reduction of 30% in training costs based on communication quality. The simulation results also validate the excellent robustness of pFedCAS to non-iid data.