Regularization Method Research Articles

An Implicit Trust Region Approach to Behavior Regularized Offline Reinforcement Learning

We revisit behavior regularization, a popular approach to mitigate the extrapolation error in offline reinforcement learning (RL), showing that current behavior regularization may suffer from unstable learning and hinder policy improvement. Motivated by this, a novel reward shaping-based behavior regularization method is proposed, where the log-probability ratio between the learned policy and the behavior policy is monitored during learning. We show that this is equivalent to an implicit but computationally lightweight trust region mechanism, which is beneficial to mitigate the influence of estimation errors of the value function, leading to more stable performance improvement. Empirical results on the popular D4RL benchmark verify the effectiveness of the presented method with promising performance compared with some state-of-the-art offline RL algorithms.

Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization

Data-Free Knowledge Distillation (DFKD) enables knowledge transfer from a pretrained teacher to a light-weighted student without original training data. Existing works are limited by a strong assumption that samples used to pretrain the teacher model are balanced, which is, however, unrealistic for many real-world tasks. In this work, we investigated a pragmatic yet under-explored problem: how to perform DFKD from a teacher model pretrained from imbalanced data. We observe a seemingly counter-intuitive phenomenon, i.e., adversarial DFKD algorithms favour minority classes, while causing a disastrous impact on majority classes. We theoretically prove that a biased teacher could cause severe disparity on different groups of synthetic data in adversarial distillation, which further exacerbates the mode collapse of a generator and consequently degenerates the overall accuracy of a distilled student model. To tackle this problem, we propose a class-adaptive regularization method, aiming to encourage impartial representation learning of a generator among different classes under a constrained learning formulation. We devise a primal-dual algorithm to solve the target optimization problem. Through extensive experiments, we show that our method mitigates the biased learning of majority classes in DFKD and improves the overall performance compared with baselines. Code will be available at https://github.com/ldpbuaa/ipad.

Iterative Regularization with k-support Norm: An Important Complement to Sparse Recovery

Sparse recovery is ubiquitous in machine learning and signal processing. Due to the NP-hard nature of sparse recovery, existing methods are known to suffer either from restrictive (or even unknown) applicability conditions, or high computational cost. Recently, iterative regularization methods have emerged as a promising fast approach because they can achieve sparse recovery in one pass through early stopping, rather than the tedious grid-search used in the traditional methods. However, most of those iterative methods are based on the l1 norm which requires restrictive applicability conditions and could fail in many cases. Therefore, achieving sparse recovery with iterative regularization methods under a wider range of conditions has yet to be further explored. To address this issue, we propose a novel iterative regularization algorithm, IRKSN, based on the k-support norm regularizer rather than the l1 norm. We provide conditions for sparse recovery with IRKSN, and compare them with traditional conditions for recovery with l1 norm regularizers. Additionally, we give an early stopping bound on the model error of IRKSN with explicit constants, achieving the standard linear rate for sparse recovery. Finally, we illustrate the applicability of our algorithm on several experiments, including a support recovery experiment with a correlated design matrix.

An Efficient Segmentation and Classification of Brain Tumor Detection using Deep Learning

In medical diagnostics, the identification of brain tumors is a crucial endeavour that frequently necessitates the careful review of intricate imaging data. With the development of deep learning methods, especially convolutional neural networks (CNNs), automated brain tumor detection has become a viable way to help radiologists diagnose patients accurately and quickly. This study provides an extensive overview of current developments in deep learning approaches for brain tumor identification. The suggested techniques usually start with preprocessing medical images, like CT or MRI scans, and then use CNN architectures to extract features. Different CNN architectures, such as U-Net and conventional CNNs, have been used to extract discriminative characteristics from brain pictures. The performance of brain tumor detection systems has also been improved using transfer learning techniques, which make use of pre-trained models on sizable datasets. These results are especially encouraging when there is a shortage of training data. Additionally, research has been done on using ensemble learning approaches to strengthen the generality and durability of brain tumor detection models. By combining several base models into one prediction, these methods improve overall performance and lower the likelihood of overfitting. Furthermore, the incorporation of sophisticated regularisation methods such batch normalisation and dropout has enhanced the deep learning models' capacity for generalisation in brain tumor identification tasks. Moreover, the use of deep learning models in practical clinical environments demands the resolution of issues of uncertainty estimation and model interpretability

Multi-class hate speech detection in the Norwegian language using FAST-RNN and multilingual fine-tuned transformers

The growth of social networks has provided a platform for individuals with prejudiced views, allowing them to spread hate speech and target others based on their gender, ethnicity, religion, or sexual orientation. While positive interactions within diverse communities can considerably enhance confidence, it is critical to recognize that negative comments can hurt people’s reputations and well-being. This emergence emphasizes the need for more diligent monitoring and robust policies on these platforms to protect individuals from such discriminatory and harmful behavior. Hate speech is often characterized as an intentional act of aggression directed at a specific group, typically meant to harm or marginalize them based on certain aspects of their identity. Most of the research related to hate speech has been conducted in resource-aware languages like English, Spanish, and French. However, low-resource European languages, such as Irish, Norwegian, Portuguese, Polish, Slovak, and many South Asian, present challenges due to limited linguistic resources, making information extraction labor-intensive. In this study, we present deep neural networks with FastText word embeddings using regularization methods for multi-class hate speech detection in the Norwegian language, along with the implementation of multilingual transformer-based models with hyperparameter tuning and generative configuration. FastText outperformed other deep learning models when stacked with Bidirectional LSTM and GRU, resulting in the FAST-RNN model. In the concluding phase, we compare our results with the state-of-the-art and perform interpretability modeling using Local Interpretable Model-Agnostic Explanations to achieve a more comprehensive understanding of the model’s decision-making mechanisms.

Inverse Calculation and Regularization Process for the Solar Aspect System (SAS) of HXI Payload on ASO-S Spacecraft

For the ASO-S/HXI payload, the accuracy of the flare reconstruction is reliant on important factors such as the alignment of the dual grating and the precise measurement of observation orientation. To guarantee optimal functionality of the instrument throughout its life cycle, the Solar Aspect System (SAS) is imperative to ensure that measurements are accurate and reliable. This is achieved by capturing the target motion and utilizing a physical model-based inversion algorithm. However, the SAS optical system’s inversion model is a typical ill-posed inverse problem due to its optical parameters, which results in small target sampling errors triggering unacceptable shifts in the solution. To enhance inversion accuracy and make it more robust against observation errors, we suggest dividing the inversion operation into two stages based on the SAS spot motion model. First, the as-rigid-as-possible (ARAP) transformation algorithm calculates the relative rotations and an intermediate variable between the substrates. Second, we solve an inversion linear equation for the relative translation of the substrates, the offset of the optical axes, and the observation orientation. To address the ill-posed challenge, the Tikhonov method grounded on the discrepancy criterion and the maximum a posteriori (MAP) method founded on the Bayesian framework are utilized. The simulation results exhibit that the ARAP method achieves a solution with a rotational error of roughly ±3.″5 (1/2-quantile); both regularization techniques are successful in enhancing the stability of the solution, the variance of error in the MAP method is even smaller—it achieves a translational error of approximately ±18 μm (1/2-quantile) in comparison to the Tikhonov method’s error of around ±24 μm (1/2-quantile). Furthermore, the SAS practical application data indicates the method’s usability in this study. Lastly, this paper discusses the intrinsic interconnections between the regularization methods.

Innovative agricultural ontology construction using NLP methodologies and graph neural network

Advancements in technology brought various innovations to agricultural practices. As a part of the development, establishing an agricultural ontology would unleash the growth of cross-domain agriculture and Natural Language Processing (NLP). For constructing such domain-based ontology, semantic and syntactic understanding of the domain data is needed. In agriculture, the availability of pre-determined domain-based data is not sufficient hence, a standard methodology with syntactic and general semantic features are required for processing the data. In this research work, Agricultural Domain based Ontology Construction (ADOC) is proposed and the overall framework has three approaches for establishing the agriculture domain based ontologies. The input text documents undergo anaphora resolution phase utilizing the semantic-based method. In the first method of ADOC the ontology is developed using the terms and relationships that are extracted from the NLP techniques. The second method of ADOC uses pretrained BERT model and Hearst patterns while the third model of ADOC is based on pretrained BERT with regular expressions and unsupervised Graph Neural Network (GNN) for creating the agricultural ontology. The efficacy of the proposed ADOC utilizing BERT with regular expressions and GNN method shows an outstanding result when compared to other proposed and prevailing systems, with a precision and recall of 96.67% and 98.31%.

Boundedness and Convergence of Mini-batch Gradient Method with Cyclic Dropconnect and Penalty

Dropout is perhaps the most popular regularization method for deep learning. Due to the stochastic nature of the Dropout mechanism, the convergence analysis of Dropout learning is challenging and the existing convergence results are mainly of probability nature. In this paper, we investigate the deterministic convergence of the mini-batch gradient learning method with Dropconnect and penalty. By drawing and presenting a set of samples of the mask matrix of Dropconnect regularization into the learning process in a cyclic manner, we establish an upper bound of the norm of the weight vector sequence and prove that the gradient of the cost function, the cost function itself, and the weight vector sequence deterministically converge to zero, a constant, and a fixed point respectively. Considering Dropout is mathematically a specific realization of Dropconnect, the established theoretical results in this paper are also valid for Dropout learning. Illustrative simulations on the MNIST dataset are provided to verify the theoretical analysis.

Wavelet denoising based on comprehensive index optimization and improved L2 regularization for load identification

Load identification, as an inverse problem, is still one of the challenges in the field of vibration engineering. In this paper, a comprehensive load identification method combining entropy weight Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) noise reduction evaluation algorithm and improved L2 regularization method is proposed to improve the load identification accuracy from the following two perspectives. Firstly, the entropy weight TOPSIS method is used to evaluate the wavelet denoising scheme comprehensively, and the optimal denoising scheme is found to reduce the noise interference in the process of inverse problem solving. Secondly, we introduce a new regularization filter function, and search for the regularization parameters by Generalized Cross-Validation (GCV) criterion and one-dimensional optimization algorithm to improve the ill-posedness of the inverse problem. Finally, the simulation and experimental verification of single-source load identification are carried out with the scaled crankcase model, while the accuracy is compared with the Tikhonov regularization method. The results show that the integrated algorithm proposed in this paper can improve the accuracy of load identification while overcoming the inadequacy of inverse problem. Unlike previous studies, the comprehensive evaluation method of denoising effect introduced in this paper provides a way to judge the denoising effect in practical engineering, and the experimental model is closer to practical engineering structures, which shows potential engineering application value.

A novel regularization method for decorrelation learning of non-parallel hyperplanes

Non-parallel hyperplane classifiers have attracted many attention due to their wider applicability than parallel hyperplane classifiers, and it accelerates the training speed through separately training of each hyperplane. However, it does not consider the correlation relationship among the non-parallel hyperplanes. In addition, since these non-parallel hyperplanes are built on the same training data, there may exist correlation among them due to redundancy of the data. In this paper, we investigate the decorrelation problem of non-parallel hyperplane classifiers. We integrate non-parallel hyperplanes into a unified model and explore this relationship through a joint classifier learning approach. Taking the twin support vector machine as an example, a novel decorrelation regularization term is added into the joint problem. Additionally, an effective alternating optimization algorithm has been introduced to address this nonconvex problem. To evaluate the proposed method, a series of experiments are conducted on binary and multi-class classification datasets from the benchmark database repository. Experimental results compared to several parallel and non-parallel hyperplane based classifiers demonstrate the effectiveness of the proposed method.

GAN‐MD: A myocarditis detection using multi‐channel convolutional neural networks and generative adversarial network‐based data augmentation

AbstractMyocarditis is a significant public health concern because of its potential to cause heart failure and sudden death. The standard invasive diagnostic method, endomyocardial biopsy, is typically reserved for cases with severe complications, limiting its widespread use. Conversely, non‐invasive cardiac magnetic resonance (CMR) imaging presents a promising alternative for detecting and monitoring myocarditis, because of its high signal contrast that reveals myocardial involvement. To assist medical professionals via artificial intelligence, the authors introduce generative adversarial networks ‐ multi discriminator (GAN‐MD), a deep learning model that uses binary classification to diagnose myocarditis from CMR images. Their approach employs a series of convolutional neural networks (CNNs) that extract and combine feature vectors for accurate diagnosis. The authors suggest a novel technique for improving the classification precision of CNNs. Using generative adversarial networks (GANs) to create synthetic images for data augmentation, the authors address challenges such as mode collapse and unstable training. Incorporating a reconstruction loss into the GAN loss function requires the generator to produce images reflecting the discriminator features, thus enhancing the generated images' quality to more accurately replicate authentic data patterns. Moreover, combining this loss function with other regularisation methods, such as gradient penalty, has proven to further improve the performance of diverse GAN models. A significant challenge in myocarditis diagnosis is the imbalance of classification, where one class dominates over the other. To mitigate this, the authors introduce a focal loss‐based training method that effectively trains the model on the minority class samples. The GAN‐MD approach, evaluated on the Z‐Alizadeh Sani myocarditis dataset, achieves superior results (F‐measure 86.2%; geometric mean 91.0%) compared with other deep learning models and traditional machine learning methods.

Face Mask Recognition based on YOLOV3

Wearing masks is considered one of the effective ways to prevent epidemics and protect physical health. In densely populated public places, detecting whether the mask is correctly worn has important application value, which can help managers reduce the potential risk of disease transmission. Previous works mainly rely on manual features, which are limited to fully expressing complex scene information. Thanks to the rapid development of artificial intelligence technology, automatic mask detection methods based on deep learning have achieved breakthroughs in speed and accuracy, while mask detection in complex scenes is still an open issue. To alleviate this issue, this paper proposes a mask detection method based on YOLOV3. Specifically, this study introduces regularization methods such as Drop Block and Label Smoothing, as well as data augmentation strategies such as Gridmask and Mosaic. This study further explores the recognition performance for refined scenarios including no mask, standard mask-wearing and incorrect mask-wearing. Extensive experimental results show the effectiveness of our method. By introducing the improved YOLOV5 algorithm to realize the mask detection function, the data structure is optimized through pruning, the data volume is reduced, the calculation speed is accelerated, and the mask detection is well realized.

Trust your source: quantifying source condition elements for variational regularisation methods

Abstract Source conditions are a key tool in regularisation theory that are needed to derive error estimates and convergence rates for ill-posed inverse problems. In this paper, we provide a recipe to practically compute source condition elements as the solution of convex minimisation problems that can be solved with first-order algorithms. We demonstrate the validity of our approach by testing it on two inverse problem case studies in machine learning and image processing: sparse coefficient estimation of a polynomial via LASSO regression and recovering an image from a subset of the coefficients of its discrete Fourier transform. We further demonstrate that the proposed approach can easily be modified to solve the machine learning task of identifying the optimal sampling pattern in the Fourier domain for a given image and variational regularisation method, which has applications in the context of sparsity promoting reconstruction from magnetic resonance imaging data.

Predicting financial distress using multimodal data: An attentive and regularized deep learning method

The proliferation of multimodal data provides a valuable repository of information for financial distress prediction. However, the use of multimodal data faces critical challenges, such as heterogeneity within and among modalities and difficulties in discriminating complementary and redundant information among modalities. To this end, we propose an attentive and regularized deep learning method for predicting financial distress using multimodal data, including financial indicators, current reports, and interfirm networks. Specifically, considering heterogeneity within and among modalities, we design three modality-specific attentions, i.e., ratio-aware, report-aware, and neighbor-aware attentions, for adaptively extracting key information from financial indicators, current reports, and interfirm networks, respectively. Considering difficulties in discriminating complementary and redundant information among modalities, we design a conditional entropy-based regularization to guide the method focusing on complementary information while discarding redundant information during modality fusion. We also propose the use of focal loss to address the class imbalance problem. Empirical evaluation shows that the proposed method significantly outperformed all benchmarked methods in terms of predictive and representation performance. We also provide key findings and implications for stakeholders.

Numerical method for a Cauchy problem for multi-dimensional Helmholtz equation

The main focus of this research is to address the Cauchy problem of the multi-dimensional Helmholtz equation with mixed boundary conditions. This problem is known to be ill-posed according to Hadamard's definition. To tackle this issue, we propose the mollification regularization method based on exponential decay. Using prior rule and posterior rule to create a regular approximation solution and convergence of the solution is also provided. Furthermore, the proposed method is shown to be robust against data disruption noise, making it a reliable approach for solving the Cauchy problem of the multi-dimensional Helmholtz equation with mixed boundary conditions.

Artificial neural network-based cooling capacity estimation of various radiator configurations for power transformers operated in ONAN mode

Power transformers submerged in oil are universally acknowledged as very useful elements in electrical power networks. A fraction of the electrical energy involved in the conversion from high to low voltages is lost as thermal energy, which is produced inside the transformer's windings and core. The effective dissipation of heat is of paramount significance and can be accomplished through the strategic installation of radiators on the tank. This study aims to examine the total cooling capacity of a radiator employed for cooling a power transformer operating in oil, natural air, and natural mode. The investigation is conducted by varying the design parameters, specifically the number of fins per radiator and the radiator length. The study also utilizes computational fluid dynamics results to achieve a substantial convergence with experimental findings. In the context of the verification study conducted for a specific study point, it was found that the simulation outcomes at a mass flow rate of 0.15 kg/s corresponded to about 7.4 % of the experimental results. Additionally, the cooling capacity value obtained was 7.2 %. In the context of machine learning, the performance of the numerical approach was assessed by employing the Bayesian regularization method. The evaluation revealed that the margin of deviation, mean squared error, and coefficient of determination (R2) metrics yielded values of −0.001 %, 1.32E-02, and 0.99930, respectively.

Kantorovich Version of Vector-Valued Shepard Operators

In the present work, in order to approximate integrable vector-valued functions, we study the Kantorovich version of vector-valued Shepard operators. We also display some applications supporting our results by using parametric plots of a surface and a space curve. Finally, we also investigate how nonnegative regular (matrix) summability methods affect the approximation.

High-Precision Calculation Using the Method of Analytical Regularization for the Cut-Off Wave Numbers for Waveguides of Arbitrary Cross Sections with Inner Conductors

A method for the accurate calculation of the cut-off wavenumbers of a waveguide with an arbitrary cross section and a number of inner conductors is demonstrated. Concepts of integral and infinite-matrix (summation) operator-valued functions depending nonlinearly on the frequency spectral parameter provide a secure basis for formulating the spectral problem, and the Method of Analytical Regularization is employed to implement an effective algorithm. The algorithm is based on a mathematically rigorous solution of the homogeneous Dirichlet problem for the Helmholtz equation in the interior of the waveguide, excluding the regions occupied by the inner conductor boundaries. A highly efficient method of calculating the cut-off wavenumbers and the corresponding non-trivial solutions representing the modal distribution is developed. The mathematical correctness of the problem statement, the method, and the ability to calculate the cut-off wavenumbers with any prescribed and proven accuracy provide a secure basis for treating these as “benchmark solutions”. In this paper, we use this new approach to validate previously obtained results against our benchmark solutions. Furthermore, we demonstrate its universality in solving some new problems, which are barely accessible by existing methods.

A new second-order dynamical method for solving linear inverse problems in Hilbert spaces

A new second-order dynamic method (SODM) is proposed for solving ill-posed linear inverse problems in Hilbert spaces. The SODM can be viewed as a combination of Tikhonov regularization and second-order asymptotical regularization methods. As a result, a double-regularization-parameter strategy is adopted. The regularization properties of SODM are demonstrated under both a priori and a posteriori stopping rules. In the context of time discretization, we propose several iterative schemes with different choices of damping parameters. A truncated discrepancy principle is employed as the stop criterion. Finally, numerical experiments are performed to show the efficiency of the SODM: on the whole, compared with the classical Tikhonov method and the first-order dynamical-system method, the SODM leads to more-accurate approximate solutions while requiring fewer-iterative numbers.

Knowledge, Attitude and Practice of Emergency Contraceptive Among Female Preparatory School Students in Awsi Resu Zone(01), Afar, Ethiopia, 2023

BACKGROUND: Students in preparatory schools are more likely to engage in risky sexual behavior, which may lead to a rise in untended pregnancies, unsafe abortions, and STIs. Many of these unplanned pregnancies can be avoided using emergency contraception. Therefore, this study was aimed at assessing the knowledge, attitude, and practice of emergency contraceptives. OBJECTIVES: To assess the knowledge, attitude, and practice of emergency contraception among preparatory school students of Awsiresu zone (01) in Afar, Ethiopia. Methods: A school-based cross-sectional study was conducted in selected preparatory schools in Awsi Resu Zone, Afar, Ethiopia, from October 20 to November 8, 2023, among 419 female students who were selected using the systematic random sampling method. A semi-structured questionnaire was developed and entered into the Kobo toolbox. Data were collected by five female public health officer students using interview-administered methods. A one-day discussion was held among group members and data collectors on how to obtain consent and collect complete data. The collected data was exported to SPSS window version 26 for analysis, and a descriptive analysis was performed for all variables. RESULT: Three hundred ten (74.0%) of the students said that they have heard about emergency contraceptives. 57.7% of the students had a positive attitude towards emergency contraceptives. One hundred (32.3%) of the students said that EC is recommended as a regular contraceptive method, and another one hundred forty-one (50.2%) students said that EC can prevent STIs. Among the ten respondents who had used emergency contraceptives, six (60%) used regular contraceptives. CONCLUSION AND RECOMMENDATIONS: The study findings showed that most of the respondents who had ever heard of emergency contraception had good knowledge and a positive attitude, despite some respondents having misunderstandings about emergency contraceptives. Therefore, the Afar Health Bureau should distribute educational materials such as brochures, posters, and flyers that teach about emergency contraception and arrange programs on a regular basis to provide accurate information concerning the utilization of emergency contraception.