Regularization Technique Research Articles

Unveiling anomalies: transformative insights from transformer-based autoencoder models

Nowadays, Intrusion Detection Systems (IDS) play a critical role in safeguarding networks by identifying and mitigating unauthorized access and malicious activities, yet face challenges in accurately detecting sophisticated threats while minimizing false positives. In this paper, we present a comprehensive analysis of a Transformer-based Autoencoder model for network intrusion detection using the NSL-KDD dataset. We evaluate the model's anomaly detection performance based on key metrics such as Mean Squared Error (MSE), Precision-Recall Area Under Curve (AUC), and Root Mean Squared Error (RMSE) on the test set. Our findings reveal promising results, with the model achieving low MSE, high AUC, and relatively low RMSE values, indicating its effectiveness in detecting anomalies. Additionally, we provide an in-depth examination of the model architecture, highlighting the role of encoder and decoder layers, dropout regularization, and optimization techniques such as Adam optimizer. Our analysis sheds light on the efficacy of Transformer-based Autoencoder models for network intrusion detection tasks, offering insights into their architectural design and performance evaluation. The novelty of this study lies in the implementation of a Transformer-based Autoencoder architecture for network intrusion detection, showcasing its superior performance in anomaly detection compared to traditional methods.

Regularized Variational Approximation for Partially Confirmatory Factor Analysis

The recently developed partially confirmatory factor analysis (PCFA) framework offers a promising solution that effectively accommodates varying levels of available knowledge, blending the strengths of both data- and theory-driven approaches. Nevertheless, its reliance on Markov Chain Monte Carlo (MCMC) techniques for parameter estimation often comes with challenges due to its computationally intensive nature and issues related to convergence. To address that, this study introduces a novel and compelling alternative - the regularized variational approximation approach within the PCFA framework (PCFA-VA). The proposed method integrates regularization techniques and converts the classical Bayesian inference problem into an optimization problem by approximating the posterior distribution with a predefined family of distributions and employing Kullback–Leibler (KL) divergence to quantify such differences. In this sense, PCFA-VA avoids the complex integration problem required to obtain the exact posterior distribution and results in substantial computational savings. Based on the simulated and real data analysis, PCFA-VA has demonstrated the potential to achieve considerable accuracy while maintaining computational efficiency, making it scalable for large-scale problems.

Regression of Likelihood Probability for Time-Varying MIMO Systems with One-Bit ADCs

This study proposes a regression-based approach for calculating the likelihood probability in time-varying multi-input multi-output (MIMO) systems using one-bit analog-to-digital converters. These time-varying MIMO systems often face performance challenges because of the difficulty in tracking changes in the likelihood probability. To address this challenge, the proposed method leverages channel statistics and decoded outputs to refine the likelihood. An optimization problem is then formulated to minimize the mean-squared error between the true and refined likelihood probabilities. A linear regression approach is derived to solve this problem, and a regularization technique is applied to further optimize the calculation. The simulation results indicate that the proposed method improves reliability by effectively tracking temporal variations in the likelihood probability and outperforms conventional methods in terms of performance.

ML-Based Pain Recognition Model Using Mixup Data Augmentation

Machine learning (ML) has revolutionized healthcare by enhancing diagnostic capabilities because of its ability to analyze large datasets and detect minor patterns often overlooked by humans. This is beneficial, especially in pain recognition, where patient communication may be limited. However, ML models often face challenges such as memorization and sensitivity to adversarial examples. Regularization techniques like mixup, which trains models on convex combinations of data pairs, address these issues by enhancing model generalization. While mixup has proven effective in image, speech, and text datasets, its application to time-series signals like electrodermal activity (EDA) is less explored. This research uses ML for pain recognition with EDA signals from the BioVid Heat Pain Database to distinguish pain by applying mixup regularization to manually extracted EDA features and using a support vector machine (SVM) for classification. The results show that this approach achieves an average accuracy of 75.87% using leave-one-subject-out cross-validation (LOSOCV) compared to 74.61% without mixup. This demonstrates mixup’s efficacy in improving ML model accuracy for pain recognition from EDA signals. This study highlights the potential of mixup in ML as a promising approach to enhance pain assessment in healthcare.

Strong stationarity for non-smooth control problems with fractional semi-linear elliptic equations in dimension $$N\le 3$$

AbstractIn this paper, we investigate the optimal control of a semi-linear fractional PDEs involving the spectral diffusion operator, or the realization of the integral fractional Laplace operator with the zero Dirichlet exterior condition, both of order s with $$s\in (0,1)$$ s ∈ ( 0 , 1 ) . The state equation contains a non-smooth nonlinearity, and the objective functional is convex in the control variable but contains non-smooth terms. As the mappings involved may not be Gâteaux differentiable, we use a regularization technique to regularize these nonlinear terms, aiming to obtain Gâteaux differentiable mappings. By employing this regularization technique, we are able to derive the first-order optimality condition for the regularized control problem by using the associated adjoint system. Furthermore, we conduct a limit analysis on the regularized term resulting in an optimality system for the non-smooth problem of C-stationary type. Subsequently, we establish a primal optimality condition, specifically B-stationarity. Under the assumption of “constraint qualification”, we derive the strong stationarity conditions for the non-smooth optimization problem with control constraints and establish the equivalence between B-stationarity and strong stationarity conditions.

Optical coherence tomography image despeckling based on saliency enhancement and high-order singular value Marchenko-Pastur truncation

Abstract Optical Coherence Tomography (OCT) is a high-resolution imaging technique extensively used in various fields, including medical diagnosis. The presence of speckle noise significantly degrades the quality of OCT images. To address this, a novel denoising approach based on High-Order Singular Value Decomposition (HOSVD), along with saliency enhancement and Marchenko-Pastur (MP) truncation, has been proposed. Initially, the method leverages the weighted absolute distance based on variance and information entropy to locate non-local patches that are highly correlated with a given reference block. Following this, a three-dimensional tensor is decomposed via HOSVD and then contracted based on the principles from the MP theorem in random matrix theory. An iterative regularization technique, coupled with a saliency enhancement strategy, is also employed to improve the denoising performance. Experimental results show that the method is comparable to existing advanced denoising algorithms in terms of reducing speckle noise and preserving image details.

Prediction of Higher Education Student Dropout based on Regularized Regression Models

This study explores the critical topic of student dropout in higher education institutions. To allow early and precise interventions and to provide a multifaceted view of student performance, this study combined two predictive models for dropout classification and score prediction. At first, a logistic regression model was developed to predict student dropout at an early stage. Then, to enhance dropout prediction, a second-degree polynomial regression model was used to predict student results based on available academic variables (access, tests, exams, projects, and assignments) from a Moodle course. Dealing with a limited dataset is a key challenge due to the high risk of overfitting. To address this issue and achieve a balance between overfitting, data size, and model complexity, the predictive models were evaluated with L1 (Lasso) and L2 (Ridge) regularization terms. The regularization techniques of the predictive models led to an accuracy of up to 89% and an R2 score of up to 86%.

Transfer graph feature alignment guided multi-source domain adaptation network for machinery fault diagnosis

In recent years, the application of unsupervised multi-source domain adaptation (MSDA) techniques for fault diagnosis has gained significant traction. Current research typically overlooks or fails to effectively capture critical data structure information during feature extraction. Another challenge is optimising the integration of information from multiple source domains to diagnose the target domain while avoiding negative transfer. To address these challenges, this study proposes a transfer graph feature alignment guided multi-source domain adaptation network (MDTGAL). In the proposed method, a transfer graph sample generator module (GSG) is constructed to model the data structure between source and target domains, and multiple graph feature extractors are employed to learn the data structure information from different domain combinations. A regularisation technique is introduced to extract the domain-invariant features by aligning the parameters across multiple independent graph feature extractors. In addition, a weighted soft-voting mechanism based on the polynomial kernel-induced maximum mean difference metric (PK-MMD) is designed to fuse the outputs from multiple classifiers, to comprehensively account for the influence of each source domain. The proposed method was tested on multi-source domain transfer tasks involving various operating conditions of rotating machinery. The experimental results demonstrate that the MDTGAL exhibits superior cross-domain diagnostic performance, outperforming existing mainstream methods. In addition, this study explored the impact of varying numbers of source domains on the diagnostic accuracy of the target domain, providing insights into selecting the correct number of source domains for specific MSDA tasks.

Impact of flexible walls of curved channel on biological flow of Reiner-Rivlin fluid

In this study, the Reiner-Rivlin fluid model is opted for examination as the complex dynamical fluid and is flowing on account of the waves travelling on the walls of flexible curved channel. The governing equations are solved analytically up to the first order to analyse the velocity profiles. As the governing equation is highly nonlinear, so the problem is solved by regular perturbation technique about small wave number. After the implication of regular perturbation, the second order Cauchy Euler ordinary differential equations are obtained for both the systems. A detailed analysis is conducted on the effects of diverse factors, such as the rheological properties of fluid, the curvature radius of the channel, the wave amplitude and the wall properties. The velocity of the fluid enhanced by increasing wall properties parameters, wave number, Reynolds number and the amplitude ratio parameter. It is also observed that the Reiner-Rivlin fluid flows slowly as compare to the Newtonian fluid, this type of findings can be helpful for the understanding of many physiological processes, like blood flow and food swallowing. This study enlightens the interaction between the curved nature of the channel and Reiner-Rivlin fluid flow in peristalsis that can be a great contribution for the designing of biomedical devices. Such types of problems have great significance in many physiological processes and engineering applications like microfluidic devices and drug delivery systems.

Label Deconvolution for Node Representation Learning on Large-Scale Attributed Graphs Against Learning Bias.

Node representation learning on attributed graphs-whose nodes are associated with rich attributes (e.g., texts and protein sequences)-plays a crucial role in many important downstream tasks. To encode the attributes and graph structures simultaneously, recent studies integrate pre-trained models with graph neural networks (GNNs), where pre-trained models serve as node encoders (NEs) to encode the attributes. As jointly training large NEs and GNNs on large-scale graphs suffers from severe scalability issues, many methods propose to train NEs and GNNs separately. Consequently, they do not take feature convolutions in GNNs into consideration in the training phase of NEs, leading to a significant learning bias relative to the joint training. To address this challenge, we propose an efficient label regularization technique, namely Label Deconvolution (LD), to alleviate the learning bias by a novel and highly scalable approximation to the inverse mapping of GNNs. The inverse mapping leads to an objective function that is equivalent to that by the joint training, while it can effectively incorporate GNNs in the training phase of NEs against the learning bias. More importantly, we show that LD converges to the optimal objective function values by the joint training under mild assumptions. Experiments demonstrate LD significantly outperforms state-of-the-art methods on Open Graph Benchmark datasets.

Regularization of the Ensemble Kalman Filter using a non-parametric, non-stationary spatial model

The sample covariance matrix of a random vector is a good estimate of the true covariance matrix if the sample size is much larger than the length of the vector. In high-dimensional problems, this condition is never met. As a result, in high dimensions the Ensemble Kalman Filter’s (EnKF) ensemble does not contain enough information to specify the prior covariance matrix accurately. This necessitates the need for regularization of the analysis (observation update) problem. We propose a regularization technique based on a new spatial model. The model is a constrained version of the general Gaussian process convolution model. The constraints include local stationarity and smoothness of local spectra. We regularize EnKF by postulating that its prior covariances obey the spatial model. Placing a hyperprior distribution on the model parameters and using the likelihood of the prior ensemble data allows for an optimized use of both the ensemble and the hyperprior. A linear version of the respective estimator is shown to be consistent. A more accurate nonlinear neural-Bayes implementation of the estimator is developed. In simulation experiments, the new technique led to substantially better EnKF performance than several existing techniques.

Optimizing LSTM with Grid Search and Regularization Techniques to Enhance Accuracy in Human Activity Recognition

Optimizing LSTM with Grid Search and Regularization Techniques to Enhance Accuracy in Human Activity Recognition

KL-FedDis: A federated learning approach with distribution information sharing using Kullback-Leibler divergence for non-IID data

Data Heterogeneity or Non-IID (non-independent and identically distributed) data identification is one of the prominent challenges in Federated Learning (FL). In Non-IID data, clients have their own local data, which may not be independently and identically distributed. This arises because clients involved in federated learning typically have their own unique, local datasets that vary significantly due to factors like geographical location, user behaviors, or specific contexts. Model divergence is another critical challenge where the local models trained on different clients, data may diverge significantly but making it difficult for the global model to converge. To identify the non-IID data, few federated learning models have been introduced as FedDis, FedProx and FedAvg, but their accuracy is too low. To address the clients Non-IID data along with ensuring privacy, federated learning emerged with appropriate distribution mechanism is an effective solution. In this paper, a modified FedDis learning method called KL-FedDis is proposed, which incorporates Kullback-Leibler (KL) divergence as the regularization technique. KL-FedDis improves accuracy and computation time over the FedDis and FedAvg technique by successfully maintaining the distribution information and encouraging improved collaboration among the local models by utilizing KL divergence.

X-DoF: Automatic Degree-of-Freedom Subset Selection for Inverse Blocked Force Characterization

Abstract Selecting the proper set of degrees-of-freedom (DoFs) is essential in inverse blocked force calculation. Including too many degrees-of-freedom in the computation can lead to overfitting, resulting in inaccurate force estimations and poor prediction quality. The discrepancy arises from errors within the dataset, such as measurement noise or other artifacts. This article presents a solution to the overfitting problem, introducing the X-DoF procedure to automatically identify the relevant subset of blocked force degrees-of-freedom. Its effectiveness is showed through numerical and experimental validation and compared against regularization techniques.

Seemingly unrelated penalized regression models

This study suggests utilizing LASSO and Elastic-net regularization techniques to address multicollinearity issues and improve variable selection in seemingly unrelated regression (SUR) models. The effectiveness of these techniques is assessed through simulation studies using mean squared and mean absolute prediction error metrics. The simulation results indicate that the LASSO method for variable selection, along with the Elastic-Net method for variable selection in high collinearity conditions, outperforms the ordinary generalized least squares and Ridge regression solutions. The practical applications of this methodology demonstrate its ability to enhance the reliability and interpretability of results in SUR models.

Securing IoT devices with zero day intrusion detection system using binary snake optimization and attention based bidirectional gated recurrent classifier

The fast improvement of cyberattacks in the area of the Internet of Things (IoT) presents novel safety challenges to zero-day attacks. Intrusion detection systems (IDS) are generally focused on exact attacks to defend the use of IoT. However, the attacks were unidentified, for IDS still signifies tasks and concerns about consumers’ data privacy and safety. Anomaly-detection models are generally based on machine learning (ML) models. Conventional ML-based models have been recognized to have low estimate excellence and recognition rates. DL-based models, particularly convolutional neural networks (CNN) with regularization techniques, direct this problem, offer a superior prediction value with unidentified data, and prevent over-fitting. This manuscript presents a Binary Snake Optimizer with DL-Enabled Zero-Day Attack Detection and Classification (BSODL-ZDADC) method. The objective of the BSODL-ZDADC method is to employ metaheuristics with the DL method for enhanced recognition and classification of zero-day attacks. For data normalization, the BSODL-ZDADC method uses a Z-score normalization approach. To reduce the high dimensionality issue and improve the classification results, the BSODL-ZDADC technique designs a BSO method to choose a set of related features. Besides, the attention-based bidirectional gated recurrent unit (ABi-GRU) method helps recognize zero-day attacks. Since the hyperparameters play a vital part in the classification performance, the BSODL-ZDADC technique employs an improved sparrow search algorithm (ISSA). The experimental validation of the BSODL-ZDADC technique is verified by utilizing the ToN-IoT dataset. The performance validation of the BSODL-ZDADC technique portrayed a superior accuracy value of 98.28% over other models.

Short-term Passenger Flow Prediction of Subway based on Neural Network LSTM Algorithm

The present study investigates the application of the Long Short-Term Memory (LSTM) neural network algorithm for short-term passenger flow forecasting in subway systems. The dataset, sourced from Hangzhou subway stations, records passenger flow at 10-minute intervals over 30 days, distinguishing between weekdays and weekends. The LSTM model was trained using 80% of the available data, while the remaining 20% was reserved for testing purposes. Results showed the model achieved a mean squared error (MSE) of 19.33 on training data, indicating a good fit. However, the test data MSE was significantly higher at 460.78, suggesting overfitting. This implies the model captures training data patterns well but struggles with generalizing to new data. The findings underline the potential of LSTM networks in handling long-term dependencies and gradient vanishing issues inherent in traditional Recurrent Neural Networks (RNNs). Despite these advancements, the study highlights the need for model refinement to address overfitting and improve generalizability. Future research should focus on data augmentation, model enhancement, and regularization techniques to develop more robust prediction models. Accurate short-term predictions of passenger flow play a crucial role in optimizing train scheduling, resource allocation, and overall service quality within urban rail transit systems.

Multi-output discrete grey model tailored for electricity consumption forecast

To address the limitation of many conventional grey forecasting methods that prioritize temporal dynamics analysis while often overlooking the essential spatial characteristics, we developed a multi-output discrete grey model tailored for forecasting electricity consumption in China's Yangtze River Delta, incorporating spatial effects. Specifically, we design a dynamic spatial interaction matrix to precisely capture the spatial correlations among neighboring areas, and integrate multiple sets of fixed-effect parameters to adeptly address spatial heterogeneity. Additionally, we implement regularization technique to prevent overfitting and utilize the Bayesian Optimization algorithm for hyper-parameter adjustment, considerably enhancing the model's stability of parameter estimation and its resilience to noise. Finally, a comprehensive case study, benchmarked against seven other models, highlights our model's outstanding multi-step predictive accuracy and robustness against input data uncertainty, exceeding the performance of existing methods.

Design of intelligent Bayesian regularized deep cascaded NARX neurostructure for predictive analysis of FitzHugh-Nagumo bioelectrical model in neuronal cell membrane

The electrophysiological modelling paradigm unravels the complex oscillatory and excitable behaviors inherent to neural dynamics, with the FitzHugh-Nagumo model—rooted in the reductionist abstraction of Hodgkin-Huxley formalism—providing a quintessential framework for exploring the underpinnings of action potential propagation and diverse phase plane dynamics characterizing excitable membranes. The objective of this study is to present the novel intelligent computing-based Bayesian regularized multilayered deep dual cascaded nonlinear autoregressive exogenous neurostructure to model and predict the intricate dynamics of FitzHugh-Nagumo model. The numerical treatment of the FitzHugh-Nagumo (FHN) model is handled with a modified Adams-Bashforth-Moulton predictor–corrector method for three sundry scenarios encompassing the oscillatory, excitable, and bistable dynamics. These simulated temporal sequences are arbitrarily divided into training and test sets for Deep Cascaded NARX neural networks, with backpropagated refinement using the Bayesian regularization technique (DC-NARX-BR). This novel strategy is verified across reference numerical outcomes with the help of diversified evaluation metrics including mean squared error convergence plots, error histogram analysis, error regression metrics, input-error cross-correlation, and error autocorrelation plots. Comparative analysis charts present the efficacious use of the DC-NARX intelligent computing paradigm with absolute errors ranging from 10-05 to 10-12. Predictive intelligence of the step ahead DC-NARX-BR strategy is observed for the intricate FHN model dynamics with marginal deviances from realized behavior. These diminutive errors can be comprehended by the low MSE losses in the range 10-02–10-05. Exhaustive experimentational averages validate the robustness of DC-NARX intelligent computing paradigm for the correct integration with the complex bioelectrical phenomena occurring within a neuronal cell membrane.

Recognition of synthesized images using modified convolutional neural network model VGG16

This paper presents a new approach to recognizing synthesized images using transfer learning, specifically the VGG16 model. With the growing prevalence of AI-generated content on social media and the increasing use of synthesized images for fraudulent purposes, the ability to accurately distinguish between real and synthesized images is of utmost importance. The study addresses the limitations of existing image recognition technologies, which often have difficulty when working with high-quality images created by AI. The proposed method uses a custom-made dataset of more than 200 000 images, balanced between AI-generated and real images of several classes, to train the model. By fine-tuning the VGG16 model and unfreezing all layers, this approach achieves great accuracy. Experimental results show that the model achieves an overall accuracy of 97%, compared to 93% accuracy of baseline model, indicating its effectiveness in distinguishing between real and synthesized images. However, shortco-mings such as slight overfitting are noted, and suggestions for future improvement include regularization techniques and exploring more advanced architectures and techniques. This research highlights the potential of transfer learning in developing robust solutions for synthesized image recognition.