Regularized Loss Function Research Articles

DEN-HMM: Deep Emission Network Based Hidden Markov Model with Time-Evolving Multivariate Observations

A Hidden Markov Model (HMM) is a popular statistical modeling technique for system health state estimation, monitoring, and prognosis. However, most existing HMMs adopted some simple parametric probability distribution as the distribution of observations for a given state and thus cannot capture the intricate dependency of observations on state and possibly other covariates such as time. To address this, we propose a Deep Emission Network-based Hidden Markov Model (DEN-HMM) to capture the complex evolution of multivariate observations with respect to state and time. We also address the challenging issue of state nondiscrimination in DEN-HMM. To overcome this, we propose a regularized loss function that can prevent certain non-discriminative trivial solutions and enhance the state discriminative capabilities of DEN-HMM. The study further demonstrates extensive numerical studies to show the effectiveness of the proposed DEN-HMM, including a case study on steady-state estimation in ultrasonic cavitation-based dispersion processes.

Junction conditions for one-dimensional network hemodynamic model for total cavopulmonary connection using physically informed deep learning technique

Abstract This paper presents a novel methodology utilizing physics-informed neural network (PINN) as a junction condition for a 1D network model of blood flow in total cavopulmonary connection generated by the Fontan procedure. The technique integrates a 3D mesh generation process based on the parameterization of the junction geometry, along with a sophisticated physically regularized neural network architecture. Synthetic datasets are produced using 3D steady Stokes simulations within fixed boundaries. We use a physically informed feedforward neural network that utilizes a physically regularized loss function, which incorporates the principle of mass conservation. Our PINN achieves a tolerance of 6% on the test set. We develop a 1D-PINN multiscale model based on a previously developed method for multiscale 1D–3D simulations. Comparison with 1D–3D Stokes based model and 3D Navier–Stokes based model verifies the 1D-PINN model. In the first and second comparison, the maximum deviations of the averaged pressures and flows do not exceed 1.48% and 12.26%, respectively.

Copyright protection framework for federated learning models against collusion attacks

Federated learning (FL) models are constructed by multiple participants who provide their training datasets and collaborate in joint training. However, training and deployment processes have encountered various challenges in terms of intellectual property protection, such as illegal theft and data leakage. Existing FL protection frameworks focus on each client independently and verify the model ownership. When they are under collusion attacks (i.e., when multiple clients negotiate to steal together), they cannot accurately identify the clients that have stolen the model. To address this challenge, a novel watermarking protection scheme against collusion attacks for federated learning is proposed in this work. It employs anti-collusion coding to design unique watermark information for each client, which can effectively detect colluders. Furthermore, it utilizes a specific regularized loss function for watermark information embedding along with the incorporation of skip connections to embed the watermark information within each batch normalization layer. The experimental results demonstrated that embedding different watermark information into each client did not affect the accuracy of the original task. The accuracy was approximately 100% when identifying the colluders. The embedding and extraction times for the original task were only 1.53% and 0.29%, respectively. Further, it exhibited high robustness against various common attacks, including fine-tuning, shearing, and collusion attacks.

Regularization, early-stopping and dreaming: A Hopfield-like setup to address generalization and overfitting

In this work we approach attractor neural networks from a machine learning perspective: we look for optimal network parameters by applying a gradient descent over a regularized loss function. Within this framework, the optimal neuron-interaction matrices turn out to be a class of matrices which correspond to Hebbian kernels revised by a reiterated unlearning protocol. Remarkably, the extent of such unlearning is proved to be related to the regularization hyperparameter of the loss function and to the training time. Thus, we can design strategies to avoid overfitting that are formulated in terms of regularization and early-stopping tuning. The generalization capabilities of these attractor networks are also investigated: analytical results are obtained for random synthetic datasets, next, the emerging picture is corroborated by numerical experiments that highlight the existence of several regimes (i.e., overfitting, failure and success) as the dataset parameters are varied.

Regularized Loss With Hyperparameter Estimation for Weakly Supervised Single Class Segmentation.

We propose a new image level weakly supervised segmentation approach for datasets with a single object class of interest. Our approach is based on a regularized loss function inspired by the classical Conditional Random Field (CRF) modeling. Our loss models properties of generic objects, and we use it to guide CNN towards segments that are more likely to correspond to the object, thus avoiding the need for pixel precise annotations. Training CNN with regularized loss is a difficult task for gradient descent. We develop an annealing algorithm which is crucial for a successful training. Furthermore, we develop an approach for hyperparameter setting for the most important components of our regularized loss. This is far from trivial, since there is no pixel precise ground truth for guidance. The advantage of our method is that we use a standard CNN architecture and an easy to interpret loss function, derived from classical CRF models. Furthermore, we apply the same loss function for any task/dataset. We first evaluate our approach for salient object segmentation and co-segmentation. These tasks naturally involve one object class of interest. Then we adapt our approach to image level weakly supervised multi-class semantic segmentation. We obtain state-of-the-art results.

Physically Informed Deep Learning Technique for Estimating Blood Flow Parameters in Four-Vessel Junction after the Fontan Procedure

This study introduces an innovative approach leveraging physics-informed neural networks (PINNs) for the efficient computation of blood flows at the boundaries of a four-vessel junction formed by a Fontan procedure. The methodology incorporates a 3D mesh generation technique based on the parameterization of the junction’s geometry, coupled with an advanced physically regularized neural network architecture. Synthetic datasets are generated through stationary 3D Navier–Stokes simulations within immobile boundaries, offering a precise alternative to resource-intensive computations. A comparative analysis of standard grid sampling and Latin hypercube sampling data generation methods is conducted, resulting in datasets comprising 1.1×104 and 5×103 samples, respectively. The following two families of feed-forward neural networks (FFNNs) are then compared: the conventional “black-box” approach using mean squared error (MSE) and a physically informed FFNN employing a physically regularized loss function (PRLF), incorporating mass conservation law. The study demonstrates that combining PRLF with Latin hypercube sampling enables the rapid minimization of relative error (RE) when using a smaller dataset, achieving a relative error value of 6% on the test set. This approach offers a viable alternative to resource-intensive simulations, showcasing potential applications in patient-specific 1D network models of hemodynamics.

Using spatio-temporal graph neural networks to estimate fleet-wide photovoltaic performance degradation patterns.

Accurate estimation of photovoltaic (PV) system performance is crucial for determining its feasibility as a power generation technology and financial asset. PV-based energy solutions offer a viable alternative to traditional energy resources due to their superior Levelized Cost of Energy (LCOE). A significant challenge in assessing the LCOE of PV systems lies in understanding the Performance Loss Rate (PLR) for large fleets of PV systems. Estimating the PLR of PV systems becomes increasingly important in the rapidly growing PV industry. Precise PLR estimation benefits PV users by providing real-time monitoring of PV module performance, while explainable PLR estimation assists PV manufacturers in studying and enhancing the performance of their products. However, traditional PLR estimation methods based on statistical models have notable drawbacks. Firstly, they require user knowledge and decision-making. Secondly, they fail to leverage spatial coherence for fleet-level analysis. Additionally, these methods inherently assume the linearity of degradation, which is not representative of real world degradation. To overcome these challenges, we propose a novel graph deep learning-based decomposition method called the Spatio-Temporal Graph Neural Network for fleet-level PLR estimation (PV-stGNN-PLR). PV-stGNN-PLR decomposes the power timeseries data into aging and fluctuation components, utilizing the aging component to estimate PLR. PV-stGNN-PLR exploits spatial and temporal coherence to derive PLR estimation for all systems in a fleet and imposes flatness and smoothness regularization in loss function to ensure the successful disentanglement between aging and fluctuation. We have evaluated PV-stGNN-PLR on three simulated PV datasets consisting of 100 inverters from 5 sites. Experimental results show that PV-stGNN-PLR obtains a reduction of 33.9% and 35.1% on average in Mean Absolute Percent Error (MAPE) and Euclidean Distance (ED) in PLR degradation pattern estimation compared to the state-of-the-art PLR estimation methods.

Research on the service condition monitoring method of rolling bearings based on adaptive deconvolution and transformer

Aiming at the complexity of the rolling bearing operation state in actual working conditions and the limitation of the feature extraction capability of single-scale model, this paper proposes a fault diagnosis method based on two-channel adaptive deconvolution and transformer module (TADAT). First, a frequency domain dataset is built and multiscale feature information is extracted using two-channel adaptive deconvolution at the beginning stage of the model; the global features of the signal are further extracted by combining them in series with the transformer (multi-head attention). Next, the loss function is optimized by introducing a regularized loss function with label smoothing and L2 regularization penalty term. Finally, the model is validated against an open-source dataset and operating condition data collected from an unbalanced bearing load test bed with an accuracy of over 98.5%. The model also maintains high accuracy under noisy samples. The test shows that the rolling bearing service condition monitoring method based on TADAT feature extraction model proposed by this method is characterized by simple model structure, high diagnostic accuracy and strong generalization performance, which provides a useful reference for the research of rolling bearing service condition monitoring.

Machine learning of independent conservation laws through neural deflation.

We introduce a methodology for seeking conservation laws within a Hamiltonian dynamical system, which we term "neural deflation." Inspired by deflation methods for steady states of dynamical systems, we propose to iteratively train a number of neural networks to minimize a regularized loss function accounting for the necessity of conserved quantities to be in involution and enforcing functional independence thereof consistently in the infinite-sample limit. The method is applied to a series of integrable and nonintegrable lattice differential-difference equations. In the former, the predicted number of conservation laws extensively grows with the number of degrees of freedom, while for the latter, it generically stops at a threshold related to the number of conserved quantities in the system. This data-driven tool could prove valuable in assessing a model's conserved quantities and its potential integrability.

Physics informed and data-based augmented learning in structural health diagnosis

Data-based diagnosis has been extensively addressed in the domain of structural health monitoring. Data exhibit patterns able to infer the existence of damaged areas, and in some cases to locate them. However, optimal predictive maintenance requires the ability to make fine predictions, which can be obtained by interrogating a model describing the systems under scrutiny. Thus, data is not only expected to enable diagnosis, but also to update the nominal model to incorporate the effects of localized damage, from data assimilation. Such a data-driven model enrichment can be performed by using different routes. In the present paper we propose an augmented physics-informed neural network procedure that allows updating the model, while completing the data collected by few sensors, using physics-informed NN and an appropriate regularized loss function.

Can Machine Learning Be Better than Biased Readers?

Background: Training machine learning (ML) models in medical imaging requires large amounts of labeled data. To minimize labeling workload, it is common to divide training data among multiple readers for separate annotation without consensus and then combine the labeled data for training a ML model. This can lead to a biased training dataset and poor ML algorithm prediction performance. The purpose of this study is to determine if ML algorithms can overcome biases caused by multiple readers' labeling without consensus. Methods: This study used a publicly available chest X-ray dataset of pediatric pneumonia. As an analogy to a practical dataset without labeling consensus among multiple readers, random and systematic errors were artificially added to the dataset to generate biased data for a binary-class classification task. The Resnet18-based convolutional neural network (CNN) was used as a baseline model. A Resnet18 model with a regularization term added as a loss function was utilized to examine for improvement in the baseline model. Results: The effects of false positive labels, false negative labels, and random errors (5-25%) resulted in a loss of AUC (0-14%) when training a binary CNN classifier. The model with a regularized loss function improved the AUC (75-84%) over that of the baseline model (65-79%). Conclusion: This study indicated that it is possible for ML algorithms to overcome individual readers' biases when consensus is not available. It is recommended to use regularized loss functions when allocating annotation tasks to multiple readers as they are easy to implement and effective in mitigating biased labels.

Generated Image Editing Method Based on Global-Local Jacobi Disentanglement for Machine Learning.

Accurate semantic editing of the generated images is extremely important for machine learning and sample enhancement of big data. Aiming at the problem of semantic entanglement in generated image latent space of the StyleGAN2 network, we proposed a generated image editing method based on global-local Jacobi disentanglement. In terms of global disentanglement, we extract the weight matrix of the style layer in the pre-trained StyleGAN2 network; obtain the semantic attribute direction vector by using the weight matrix eigen decomposition method; finally, utilize this direction vector as the initialization vector for the Jacobi orthogonal regularization search algorithm. Our method improves the speed of the Jacobi orthogonal regularization search algorithm with the proportion of effective semantic attribute editing directions. In terms of local disentanglement, we design a local contrast regularized loss function to relax the semantic association local area and non-local area and utilize the Jacobi orthogonal regularization search algorithm to obtain a more accurate semantic attribute editing direction based on the local area prior MASK. The experimental results show that the proposed method achieves SOTA in semantic attribute disentangled metrics and can discover more accurate editing directions compared with the mainstream unsupervised generated image editing methods.

Personalised federated learning framework for damage detection in structural health monitoring

Data-driven machine learning models, compared to numerical models, demonstrated promising improvements in detecting damage in structural health monitoring (SHM) applications. In such approaches, sensors’ data are used to train a model either in a centralized model (server) or locally inside each sensor unit node (client). The centralized learning model often leads to computing and privacy issues such as wireless transmission costs and data-sensitive vulnerability, especially in real-time settings. The decentralized model also poses different challenges such as feature correlations and relationships loss in decentralized learning settings. To handle the shortcomings of both models, we propose a new Personalized federated learning (FL) model augmented with tensor data fusion to learn and detect damage in SHM. Our approach employs FL which enables the central machine learning model to gain experience from diverse datasets located at different sensor locations. Furthermore, our proposed model addresses the problems associated non-i.i.d. data by employing the Moreau envelopes as a regularized loss function in the learning process of client’s models. Our methods help in decoupling the client models from the central one which improves personalized in FL. Our experimental evaluation on real structural datasets demonstrates promising damage detection accuracy without the need to transmit the actual data to the centralized learning model. The results also show that the data correlations and relationships from all participating sensors are preserved.

Optimal recovery of unsecured debt via interpretable reinforcement learning

This paper addresses the issue of interpretability and auditability of reinforcement-learning agents employed in the recovery of unsecured consumer debt. To this end, we develop a deterministic policy-gradient method that allows for a natural integration of domain expertise into the learning procedure so as to encourage learning of consistent, and thus interpretable, policies. Domain knowledge can often be expressed in terms of policy monotonicity and/or convexity with respect to relevant state inputs. We augment the standard actor–critic policy approximator using a monotonically regularized loss function which integrates domain expertise into the learning. Our formulation overcomes the challenge of learning interpretable policies by constraining the search to policies satisfying structural-consistency properties. The resulting state-feedback control laws can be readily understood and implemented by human decision makers. This new domain-knowledge enhanced learning approach is applied to the problem of optimal debt recovery which features a controlled Hawkes process and an asynchronous action–feedback relationship.

Classifying Chart Based on Structural Dissimilarities using Improved Regularized Loss Function

Classifying Chart Based on Structural Dissimilarities using Improved Regularized Loss Function

Bearing fault diagnosis based on wavelet sparse convolutional network and acoustic emission compression signals.

A bearing is an important and easily damaged component of mechanical equipment. For early fault diagnosis of ball bearings, acoustic emission signals are more sensitive and less affected by mechanical background noise. To cope with the large amount of data brought by the high sampling frequency and high sampling points of acoustic emission signals, a compressed sensing processing framework is introduced to research data compression and feature extraction, and a wavelet sparse convolutional network is proposed for resolved diagnosis and evaluation. The main research objective of this paper is to maximize the compression rate of the signal under the constraint of ensuring the reconstruction error of the acoustic emission signal, which can reduce the data volume of the acoustic emission signal and reduce the pressure of data analysis for subsequent fault diagnosis. At the same time, a wide convolution kernel based on a continuous wavelet is introduced when designing the neural network, and the energy information of different frequency bands of the signal is extracted by the wavelet convolution kernel to characterize the fault characteristics of the equipment. The energy pooling layer is designed to enhance the deep mining ability of compressed features, and the regularized loss function is introduced to improve the diagnostic accuracy and robustness through feature sparseness. The experimental results show that the method can effectively extract the fault characteristics of the bearing acoustic emission signal, improve the analysis efficiency and accurately classify the bearing faults.

ResNet-LDDMM: Advancing the LDDMM Framework Using Deep Residual Networks.

In deformable registration, the Riemannian framework - Large Deformation Diffeomorphic Metric Mapping, or LDDMM for short - has inspired numerous techniques for comparing, deforming, averaging and analyzing shapes or images. Grounded in flows of vector fields, akin to the equations of motion used in fluid dynamics, LDDMM algorithms solve the flow equation in the space of plausible deformations, i.e., diffeomorphisms. In this work, we make use of deep residual neural networks to solve the non-stationary ODE (flow equation) based on an Euler's discretization scheme. The central idea is to represent time-dependent velocity fields as fully connected ReLU neural networks (building blocks) and derive optimal weights by minimizing a regularized loss function. Computing minimizing paths between deformations, thus between shapes, turns to find optimal network parameters by back-propagating over the intermediate building blocks. Geometrically, at each time step, our algorithm searches for an optimal partition of the space into multiple polytopes, and then computes optimal velocity vectors as affine transformations on each of these polytopes. As a result, different parts of the shape, even if they are close (such as two fingers of a hand), can be made to belong to different polytopes, and therefore be moved in different directions without costing too much energy. Importantly, we show how diffeomorphic transformations, or more precisely bilipshitz transformations, are predicted by our registration algorithm. We illustrate these ideas on diverse registration problems of 3D shapes under complex topology-preserving transformations. We thus provide essential foundations for more advanced shape variability analysis under a novel joint geometric-neural networks Riemannian-like framework, i.e., ResNet-LDDMM.

Robust classification from noisy labels: Integrating additional knowledge for chest radiography abnormality assessment

Chest radiography is the most common radiographic examination performed in daily clinical practice for the detection of various heart and lung abnormalities. The large amount of data to be read and reported, with more than 100 studies per day for a single radiologist, poses a challenge in consistently maintaining high interpretation accuracy. The introduction of large-scale public datasets has led to a series of novel systems for automated abnormality classification. However, the labels of these datasets were obtained using natural language processed medical reports, yielding a large degree of label noise that can impact the performance. In this study, we propose novel training strategies that handle label noise from such suboptimal data. Prior label probabilities were measured on a subset of training data re-read by 4 board-certified radiologists and were used during training to increase the robustness of the training model to the label noise. Furthermore, we exploit the high comorbidity of abnormalities observed in chest radiography and incorporate this information to further reduce the impact of label noise. Additionally, anatomical knowledge is incorporated by training the system to predict lung and heart segmentation, as well as spatial knowledge labels. To deal with multiple datasets and images derived from various scanners that apply different post-processing techniques, we introduce a novel image normalization strategy. Experiments were performed on an extensive collection of 297,541 chest radiographs from 86,876 patients, leading to a state-of-the-art performance level for 17 abnormalities from 2 datasets. With an average AUC score of 0.880 across all abnormalities, our proposed training strategies can be used to significantly improve performance scores.

A novel deep learning neural network for fast-food image classification and prediction using modified loss function

Accurate food image classification is often critical to accurately monitor the dietary assessment to reduce the risk of different heart-related diseases, obesity, diabetes, and other related health conditions. The accuracy and efficiency of image classification results when using traditional deep learning methods were less than optimal. This research aimed at enhancing the classification and prediction accuracy of food images and reducing the processing time by using the Deep Convolutional Neural Network (DCN) algorithm. The solution starts by using the Modified Loss function, the images are fed into the DCN for features extraction through alternating between convolutional layers and pooling layers, then this is followed by a fully connected layer. Finally, the Softmax function is used to classify the images. The result was compared during the classification phase in the DCN. The proposed solution enhanced the accuracy of the classification by using the regularized loss function and lowered the processing time by decreasing the weights of the neurons in the neural network. Probability score is used as the evaluation metric for the accuracy, and total execution time is used as the evaluation metric for the speed of the algorithm. The combination of deep neural network with regularized cross entropy cost function has improved the fast-food images classification by ahcieving better processing time by 40 ~ 50s and accuracy by 5% in average.

Parameter Estimation and Classification via Supervised Learning in the Wireless Physical Layer

Emerging wireless networks possess the potential to achieve levels of connectivity and Quality-of-Service (QoS) that are orders of magnitude higher than today’s networks. Realizing the potential of these networks will require flexible, low cost, and accurate Digital Signal Processing (DSP). Supervised Learning (SL) models employing unknown parameter estimation and classification techniques have experienced widespread use in physical (PHY) layer wireless communication systems since they can achieve low costs via inexpensive forward-pass computations, attain flexible operations due to trainable parameters, and yield accurate results based on the universal approximator attribute. In this tutorial paper, we present a methodical explanation of how SL can be applied to unknown parameter estimation and classification across several different PHY layer components of a wireless communications system. Additionally, via a survey and comparison of popular methods, this paper provides insights on how to perform weight training, weight initialization, loss function regularization, data pre-processing, input feature design, ensemble training, and hyper-parameter validation. In our review of state-of-the-art works, we found significant use of SL algorithms in the following PHY layer applications: Dynamic Spectrum Access (DSA), channel corrections, Automatic Gain Control (AGC), Multiple Input Multiple Output (MIMO) control, Analog to Digital (ADC) conversion, and Automatic Coding and Modulation (ACM). The overarching goal of this survey and tutorial paper is to assist the reader in understanding the motivation and methodologies associated with various SL algorithms applied to PHY layer DSP operations, as well as to provide the reader with the necessary tools and techniques needed for addressing open challenges to be experienced by future wireless networks.