Standard Regularization Research Articles

Finding global solutions of some inverse optimal control problems using penalization and semismooth Newton methods

We present a method to solve a special class of parameter identification problems for an elliptic optimal control problem to global optimality. The bilevel problem is reformulated via the optimal-value function of the lower-level problem. The reformulated problem is nonconvex and standard regularity conditions like Robinson’s CQ are violated. Via a relaxation of the constraints, the problem can be decomposed into a family of convex problems and this is the basis for a solution algorithm. The convergence properties are analyzed. It is shown that a penalty method can be employed to solve this family of problems while maintaining convergence speed. For an example problem, the use of the identity as penalty function allows for the solution by a semismooth Newton method. Numerical results are presented. Difficulties and limitations of our approach to solve a nonconvex problem to global optimality are discussed.

Arabic sentiment analysis using GCL-based architectures and a customized regularization function

Sentiment analysis aims to extract emotions from textual data; with the proliferation of various social media platforms and the flow of data, particularly in the Arabic language, significant challenges have arisen, necessitating the development of various frameworks to handle issues. In this paper, we firstly design an architecture called Gated Convolution Long (GCL) to perform Arabic Sentiment Analysis. GCL can overcome difficulties with lengthy sequence training samples, extracting the optimal features that help improve Arabic sentiment analysis performance for binary and multiple classifications. The proposed method trains and tests in various Arabic datasets; The results are better than the baselines in all cases. GCL includes a Custom Regularization Function (CRF), which improves the performance and optimizes the validation loss. We carry out an ablation study and investigate the effect of removing CRF. CRF is shown to make a difference of up to 5.10% (2C) and 4.12% (3C). Furthermore, we study the relationship between Modern Standard Arabic and five Arabic dialects via a cross-dialect training study. Finally, we apply GCL through standard regularization (GCL+L1, GCL+L2, and GCL+LElasticNet) and our Lnew on two big Arabic sentiment datasets; GCL+Lnew gave the highest results (92.53%) with less performance time.

Information Equivalence among Transformations of Semi‐parametric Nonlinear Panel Data Models*

AbstractThis paper considers transformations of nonlinear semi‐parametric mean functions that yield moment conditions for estimation. Such transformations are said to be information equivalent if they yield the same asymptotic efficiency bound. I derive a unified theory of algebraic equivalence for moment conditions created by a given linear transformation. The main equivalence result states that under standard regularity conditions, transformations that create conditional moment restrictions in a given empirical setting need only to have an equal rank to reach the same efficiency bound. I compare feasible and infeasible transformations of both nonlinear models with multiplicative heterogeneity and linear models with unobserved factor structures.

Generalization of the shortest path approach for superpixel segmentation of omnidirectional images

With the growing use of image capture devices using wide angles and the need for fast and accurate image analysis in computer vision, there is a demand for dedicated under-representation methods. Most decomposition methods segment an image into a small number of irregular homogeneous regions, called superpixels. Nevertheless, these approaches are generally designed to process natural 2D planar images, i.e., captured with a 90o angle view without distortion. In this work, we present SphSPS, a new general decomposition method (for Spherical Shortest Path-based Superpixels)11Available code at: https://github.com/rgiraud/sphsps, that is dedicated to wide 360o omnidirectional or spherical images. The produced superpixels respect both the geometry of the 3D spherical acquisition space, and the boundaries of image objects. To fastly extract relevant clustering features, we generalize the shortest path approach between a pixel and a superpixel center. We demonstrate that considering the geometry of the acquisition space to compute the shortest path enables to jointly improve the segmentation accuracy and the shape regularity of superpixels. To evaluate this regularity property, we propose a generalization of a standard 2D regularity metric to the spherical space, addressing the limitations of the only existing spherical compactness measure. Finally, SphSPS is validated on reference 360o images from the PSD (Panorama Segmentation Dataset) and also on synthetic road omnidirectional images. Our method significantly outperforms both planar and spherical state-of-the-art approaches in terms of segmentation accuracy, robustness to noise and regularity, providing a very interesting tool for superpixel-based applications on 360o images.

Regularised Volterra series models for modelling of nonlinear self-excited forces on bridge decks

Volterra series models are considered an attractive approach for modelling nonlinear aerodynamic forces for bridge decks since they extend the convolution integral to higher dimensions. Optimal identification of nonlinear systems is a challenging task since there are typically many unknown variables that need to be determined, and it is vital to avoid overfitting. Several methods exist for identifying Volterra kernels from experimental data, but a large class of them put restrictions on the system inputs, making them infeasible for section model tests of bridge decks. A least-squares identification method does not restrict the inputs, but the identified model often struggles with noisy (non-smooth) kernels, which is deemed to be unphysical and a sign of overfitting. In this work, regularised least-squares identification is introduced to improve the performance of model identification using least-squares. Standard Tikhonov regularisation and other penalty techniques that impose decaying kernels are also explored. The performance of the methodology is studied using experimental data from wind tunnel tests of a twin deck section. The regularised Volterra models show equal or better results in terms of modelling the self-excited forces, and the regularisation makes the models less prone to overfitting.

Precision aiming performance with the paretic upper limb is associated with center of pressure patterns in individuals with chronic stroke

BackgroundIndividuals with stroke commonly demonstrate upper-limb sensorimotor impairments. Upper-limb tasks occur against a background level of postural control and thus require a flexible postural control system to facilitate performance. Anterior precision aiming tasks, for example, benefit from lower medial-lateral (ML) center of pressure (COP) fluctuations (where increased fluctuations erode performance) relative to anterior-posterior (AP) fluctuations (where increased fluctuations do not strongly influence performance). After stroke, individuals may compensate for upper-limb impairments by increasing trunk movement which increases overall COP fluctuations and thus may make it more difficult to modulate COP in a task-sensitive manner. Research questionDo upper-limb task demands modulate COP movement patterns after stroke? MethodsIn this cross-sectional study, adults with chronic stroke (n = 23) and unilateral upper-limb impairments were immersed in a virtual environment displaying an anterior target. Participants aimed to maintain the position of a virtual laser pointer (via handheld controller) in the target with each hand. COP was concurrently recorded. Mixed effects models and correlations were used to detect differences in COP patterns between limbs and movement planes and evaluate associations between task performance and COP patterns, respectively. ResultsParticipants showed greater COP standard deviation and regularity in the AP compared to the ML direction. The magnitude of difference between AP and ML COP metrics was greater using the nonparetic limb. Task performance was moderately and positively associated with task-sensitive COP patterns (i.e., higher AP:ML ratios of COP metrics) using the paretic upper limb. Participants consistently demonstrated high levels of task performance and task-sensitive COP movement patterns using the nonparetic limb. SignificanceImpairments in postural control after stroke may be related to the upper limb used. It is important to recognize the role of directional COP variability and regularity in the context of a task goal after stroke.

Trimmed sampling algorithm for the noisy generalized eigenvalue problem

Solving the generalized eigenvalue problem is a useful method for finding energy eigenstates of large quantum systems. It uses projection onto a set of basis states which are typically not orthogonal. One needs to invert a matrix whose entries are inner products of the basis states, and the process is unfortunately susceptible to even small errors. The problem is especially bad when matrix elements are evaluated using stochastic methods and have significant error bars. In this work, we introduce the trimmed sampling algorithm in order to solve this problem. Using the framework of Bayesian inference, we sample prior probability distributions determined by uncertainty estimates of the various matrix elements and likelihood functions composed of physics-informed constraints. The result is a probability distribution for the eigenvectors and observables which automatically comes with a reliable estimate of the error and performs far better than standard regularization methods. The method should have immediate use for a wide range of applications involving classical and quantum computing calculations of large quantum systems.Received 14 October 2022Accepted 15 March 2023DOI:https://doi.org/10.1103/PhysRevResearch.5.L022001Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.Published by the American Physical SocietyPhysics Subject Headings (PhySH)Research AreasBose gasesNuclear many-body theoryNuclear structure & decaysQuantum algorithmsAtomic, Molecular & Optical

Spatial gradient consistency for unsupervised learning of hyperspectral demosaicking: application to surgical imaging

PurposeHyperspectral imaging has the potential to improve intraoperative decision making if tissue characterisation is performed in real-time and with high-resolution. Hyperspectral snapshot mosaic sensors offer a promising approach due to their fast acquisition speed and compact size. However, a demosaicking algorithm is required to fully recover the spatial and spectral information of the snapshot images. Most state-of-the-art demosaicking algorithms require ground-truth training data with paired snapshot and high-resolution hyperspectral images, but such imagery pairs with the exact same scene are physically impossible to acquire in intraoperative settings. In this work, we present a fully unsupervised hyperspectral image demosaicking algorithm which only requires exemplar snapshot images for training purposes.MethodsWe regard hyperspectral demosaicking as an ill-posed linear inverse problem which we solve using a deep neural network. We take advantage of the spectral correlation occurring in natural scenes to design a novel inter spectral band regularisation term based on spatial gradient consistency. By combining our proposed term with standard regularisation techniques and exploiting a standard data fidelity term, we obtain an unsupervised loss function for training deep neural networks, which allows us to achieve real-time hyperspectral image demosaicking.ResultsQuantitative results on hyperspetral image datasets show that our unsupervised demosaicking approach can achieve similar performance to its supervised counter-part, and significantly outperform linear demosaicking. A qualitative user study on real snapshot hyperspectral surgical images confirms the results from the quantitative analysis.ConclusionOur results suggest that the proposed unsupervised algorithm can achieve promising hyperspectral demosaicking in real-time thus advancing the suitability of the modality for intraoperative use.

Enhanced damage imaging in three-dimensional composite structures via electrical impedance tomography with mixed and level set regularization

Electrical impedance tomography (EIT) is a promising tool for nondestructive evaluation (NDE) of materials that exhibit stimulus-responsive electrical conductivity. To date, materials practitioners of EIT have used standard regularization methods such as smoothness-promoting operators to solve the ill-posed EIT inverse problem. However, more sophisticated and better performing regularization methods exist. It is essential that these other regularization methods be explored in order for EIT to see more widespread adoption in NDE. To that end, we present two novel regularization techniques – a mixed smoothness/focal method and a level set method – for application to damage imaging via EIT. These new techniques are experimentally validated on two three-dimensional composite components, a carbon black-modified glass fiber/epoxy tube and a carbon fiber/epoxy laminate airfoil subject to multiple damaging impacts. It is shown that both of the new regularization techniques markedly outperform traditional smoothness-only methods. These results are an important step towards adoption of more sophisticated EIT formulations in NDE and transition of EIT from the proof-of-concept phase and into practice for engineered materials.

Adaptive Image Processing: First Order PDE Constraint Regularizers and a Bilevel Training Scheme

A bilevel training scheme is used to introduce a novel class of regularizers, providing a unified approach to standard regularizers TGV^2 and NsTGV^2. Optimal parameters and regularizers are identified, and the existence of a solution for any given set of training imaging data is proved by Gamma -convergence under a conditional uniform bound on the trace constant of the operators and a finite-null-space condition. Some first examples and numerical results are given.

Adversarial Robustness via Fisher-Rao Regularization.

Adversarial robustness has become a topic of growing interest in machine learning since it was observed that neural networks tend to be brittle. We propose an information-geometric formulation of adversarial defense and introduce Fire, a new Fisher-Rao regularization for the categorical cross-entropy loss, which is based on the geodesic distance between the softmax outputs corresponding to natural and perturbed input features. Based on the information-geometric properties of the class of softmax distributions, we derive an explicit characterization of the Fisher-Rao Distance (FRD) for the binary and multiclass cases, and draw some interesting properties as well as connections with standard regularization metrics. Furthermore, we verify on a simple linear and Gaussian model, that all Pareto-optimal points in the accuracy-robustness region can be reached by Fire while other state-of-the-art methods fail. Empirically, we evaluate the performance of various classifiers trained with the proposed loss on standard datasets, showing up to a simultaneous 1% of improvement in terms of clean and robust performances while reducing the training time by 20% over the best-performing methods.

Non-convex sparse regularization via convex optimization for impact force identification

Instead of traditional Tikhonov regularization, sparse regularization methods like ℓ1 regularization have been a popular choice for impact force identification because it can encode sparse prior information into the inverse problem solving. However, ℓ1 regularization tends to have a limited ability to promote sparsity and underestimate high-amplitude components in the solution. Non-convex regularization holds the potential to promote sparsity more efficiently, and then further improves the accuracy of solutions. In this paper, we consider a family of non-convex regularizers with convexity-preserving characteristics to promote sparsity. By virtue of the non-convex regularizers, we propose a non-convex sparse regularization method to simultaneously localize and reconstruct impact events with a highly under-determined sensor setting. To avoid the inherent difficulties concerning non-convex optimization, we turn to the Alternating Direction Method of Multipliers(ADMM) algorithm. ADMM helps separate the primal non-convex problem into a sequence of convex sub-problems, each of which is easy-to-solve. The non-convex exponential penalty is chosen as the representative of this class of regularizers through theoretical analysis. Several simulations and experiments are conducted on a stiffened composite laminated panel to validate the proposed method. The proposed method outperforms the standard ℓ1 regularization in both localization accuracy and time-history reconstruction accuracy.

SPECT reconstruction with a trained regularizer using CT-side information: Application to 177Lu SPECT imaging.

Improving low-count SPECT can shorten scans and support pre-therapy theranostic imaging for dosimetry-based treatment planning, especially with radionuclides like 177Lu known for low photon yields. Conventional methods often underperform in low-count settings, highlighting the need for trained regularization in model-based image reconstruction. This paper introduces a trained regularizer for SPECT reconstruction that leverages segmentation based on CT imaging. The regularizer incorporates CT-side information via a segmentation mask from a pre-trained network (nnUNet). In this proof-of-concept study, we used patient studies with 177Lu DOTATATE to train and tested with phantom and patient datasets, simulating pre-therapy imaging conditions. Our results show that the proposed method outperforms both standard unregularized EM algorithms and conventional regularization with CT-side information. Specifically, our method achieved marked improvements in activity quantification, noise reduction, and root mean square error. The enhanced low-count SPECT approach has promising implications for theranostic imaging, post-therapy imaging, whole body SPECT, and reducing SPECT acquisition times.

Optimization of beam arrangement for tunable diode laser absorption tomography reconstruction based on fractional Tikhonov regularization

Beam arrangement with limited projections based on tunable diode laser absorption spectroscopy is the key to achieving a more accurate two-dimensional reconstruction of the combustion. Using fractional calculus theory, a beam optimization method based on fractional Tikhonov regularization is proposed. The beam arrangement function based on fractional Tikhonov regularization is established by extending the standard Tikhonov regularization to fractional modes. The genetic algorithm is used to analyze the calculation results of different orders in a range of (0, 1), and the beam arrangement is obtained. Using 20 laser beams to scan the characteristic absorption spectrum of H<sub>2</sub>O in the near-infrared band 7185.6 cm<sup>–1</sup>, modeling the calculations in a 10×10 element discrete tomography domain, and comparing the reconstruction results of the five beam arrangements for different Gaussian distribution models, the beam arrangement based on fractional Tikhonov regularization shows more obvious advantages. This design method proposed in this work is valuable for the theoretical study of the optimal design of two-dimensional measurement beams based on the tunable diode laser absorption spectroscopy technique, which can promote the application of this technique in the two-dimensional reconstruction of complex engine combustion and combustion efficiency improvement.

Discriminative restricted Boltzmann machine with trainable sparsity

Discriminative restricted Boltzmann machine (DRBM) is a probabilistic three-layered neural network, consisting of the input, hidden, and output layers, that helps to solve classification problems. This study attempts to improve the generalization property of the DRBM. Regularization methods such as L1 or L2 regularizations can be used to control the representation power of a learning model and suppress over-fitting to a dataset. To control the representation power of the DRBM, an alternative regularization approach is proposed, in which sparse regularization is imposed on the values of the hidden variables of the DRBM. In the resultant model, the sparse regularization controls the effective size of the hidden layer of the DRBM. Unlike standard regularization methods, in the proposed model, parameters that control the sparsity strength are trainable. The method is validated through numerical experiments based on benchmark datasets.

Identification of Time-Varying External Force Using Group Sparse Regularization and Redundant Dictionary.

How to accurately identify unknown time-varying external force from measured structural responses is an important engineering problem, which is critical for assessing the safety condition of the structure. In the context of a few available accelerometers, this paper proposes a novel time-varying external force identification method using group sparse regularization based on the prior knowledge in the redundant dictionary. Firstly, the relationship between time-varying external force and acceleration responses is established, and a redundant dictionary is designed to create a sparse expression of external force. Then, the relevance of atoms in the redundant dictionary is revealed, and this prior knowledge is used to determine the group structures of atoms. As a result, a force identification governing equation is formulated, and the group sparse regularization is reasonably introduced to ensure the accuracy of the identified results. The contribution of this paper is that the group structures of atoms are reasonably determined based on prior knowledge, and the complexity in the process for identifying external force from measured acceleration responses is reduced. Finally, the effectiveness of the proposed method is demonstrated by numerical simulations and an experimental structure. The illustrated results show that, compared with the force identification method based on the standard l1-norm regularization, the proposed method can further improve the identified accuracy of unknown external force and greatly enhance the computational efficiency for the force identification problem.

Localization of magnetocardiographic sources for myocardial infarction cases using deterministic and Bayesian approaches

In this paper, the inverse problems of cardiac sources using analytical and probabilistic methods are solved and discussed. The standard Tikhonov regularization technique is solved initially to estimate the under-determined heart surface potentials from Magnetocardiographic (MCG) signals. The results of the deterministic method subjected to noise in the measurements are discussed and compared with the probabilistic models. Hierarchical Bayesian modeling with fixed Gaussian prior is employed to quantify the uncertainties in source reconstructions. A novel application of Variational Bayesian inference approach has been presented to estimate the heart sources. The reconstruction results of Variational Bayesian model with non-stationary priors are compared with solutions of simplistic Bayesian approach; and the performances are evaluated using Root Mean Square Error (RMSE) and correlation co-efficient metrics. The Bayesian solutions in the study are also extended to localize the MCG sources for two types of Myocardial infarction cases.

Zero-sum semi-Markov games with state-action-dependent discount factors

Semi-Markov model is one of the most general models for stochastic dynamic systems. This paper deals with a two-person zero-sum game for semi-Markov processes. We focus on the expected discounted criterion with state-action-dependent discount factors. The state and action spaces are both Polish spaces, and the reward rate function is ω-bounded. We first construct a fairly general model of semi-Markov games under a given semi-Markov kernel and a pair of strategies. Next, based on the standard regularity condition and the continuity-compactness condition for semi-Markov games, we derive a “drift condition” on the semi-Markov kernel and suppose that the discount factors have a positive lower bound, under which the existence of the value function and an optimal pair of stationary strategies of our semi-Markov game are proved by using a general Shapley equation. Moreover, in the scenario of finite state and action spaces, a value iteration-type algorithm for approximating the value function and an optimal pair of stationary strategies is developed. The convergence and the error bound of the algorithm are also proved. Finally, we conduct numerical examples to demonstrate the main results.

On Fractional Tikhonov Regularization: Application to the Adaptive Network-Based Fuzzy Inference System for Regression Problems

In this article, we introduce a variant of the adaptive network-based fuzzy inference system (ANFIS). The proposed variant does not use backpropagation and grid partitioning, but the least-squares method with fractional Tikhonov regularization. The fractional regularization is a generalization of the standard regularization and is applied here to the learning process of the ANFIS scheme for the first time. This results in a simpler rule base, with a low number of rules, allowing to handle problems with many input variables with relatively low computational time while keeping high accuracy. We present new theoretical results on the fractional Tikhonov regularization. Such results are the basis for a formal discussion on how much the choice of a different architecture, resulting in a different matrix in the least-squares minimization, could affect the accuracy. We perform several numerical experiments on benchmark examples, first to assess the impact of the fractional regularization on the accuracy and then to compare our results against the most recent ones reported in the literature by other ANFIS-like or neuro-fuzzy systems. The numerical results show the good performance of the proposed approach.

Characteristic power series of graph limits

In this note, we show how to obtain a “characteristic power series” of graphons – infinite limits of dense graphs – as the limit of normalized reciprocal characteristic polynomials. This leads to a new characterization of graph quasi-randomness and another perspective on spectral theory for graphons, a complete description of the function in terms of the spectrum of the graphon as a self-adjoint kernel operator. Interestingly, while we apply a standard regularization to classical determinants, it is unclear how necessary this is.