Conventional Loss Function Research Articles

Contrastive Speaker Representation Learning with Hard Negative Sampling for Speaker Recognition.

Speaker recognition is a technology that identifies the speaker in an input utterance by extracting speaker-distinguishable features from the speech signal. Speaker recognition is used for system security and authentication; therefore, it is crucial to extract unique features of the speaker to achieve high recognition rates. Representative methods for extracting these features include a classification approach, or utilizing contrastive learning to learn the speaker relationship between representations and then using embeddings extracted from a specific layer of the model. This paper introduces a framework for developing robust speaker recognition models through contrastive learning. This approach aims to minimize the similarity to hard negative samples-those that are genuine negatives, but have extremely similar features to the positives, leading to potential mistaken. Specifically, our proposed method trains the model by estimating hard negative samples within a mini-batch during contrastive learning, and then utilizes a cross-attention mechanism to determine speaker agreement for pairs of utterances. To demonstrate the effectiveness of our proposed method, we compared the performance of a deep learning model trained with a conventional loss function utilized in speaker recognition with that of a deep learning model trained using our proposed method, as measured by the equal error rate (EER), an objective performance metric. Our results indicate that when trained with the voxceleb2 dataset, the proposed method achieved an EER of 0.98% on the voxceleb1-E dataset and 1.84% on the voxceleb1-H dataset.

Low-light image enhancement using generative adversarial networks

In low-light environments, the amount of light captured by the camera sensor is reduced, resulting in lower image brightness. This makes it difficult to recognize or completely lose details in the image, which affects subsequent processing of low-light images. Low-light image enhancement methods can increase image brightness while better-restoring color and detail information. A generative adversarial network is proposed for low-quality image enhancement to improve the quality of low-light images. This network consists of a generative network and an adversarial network. In the generative network, a multi-scale feature extraction module, which consists of dilated convolutions, regular convolutions, max pooling, and average pooling, is designed. This module can extract low-light image features from multiple scales, thereby obtaining richer feature information. Secondly, an illumination attention module is designed to reduce the interference of redundant features. This module assigns greater weight to important illumination features, enabling the network to extract illumination features more effectively. Finally, an encoder-decoder generative network is designed. It uses the multi-scale feature extraction module, illumination attention module, and other conventional modules to enhance low-light images and improve quality. Regarding the adversarial network, a dual-discriminator structure is designed. This network has a global adversarial network and a local adversarial network. They determine if the input image is actual or generated from global and local features, enhancing the performance of the generator network. Additionally, an improved loss function is proposed by introducing color loss and perceptual loss into the conventional loss function. It can better measure the color loss between the generated image and a normally illuminated image, thus reducing color distortion during the enhancement process. The proposed method, along with other methods, is tested using both synthesized and real low-light images. Experimental results show that, compared to other methods, the images enhanced by the proposed method are closer to normally illuminated images for synthetic low-light images. For real low-light images, the images enhanced by the proposed method retain more details, are more apparent, and exhibit higher performance metrics. Overall, compared to other methods, the proposed method demonstrates better image enhancement capabilities for both synthetic and real low-light images.

Advancing Glaucoma Diagnosis: Employing Confidence-Calibrated Label Smoothing Loss for Model Calibration

ObjectiveThe aim of our research is to enhance the calibration of machine learning models for glaucoma classification through a specialized loss function named Confidence-Calibrated Label Smoothing Loss (CC-LS). This approach is specifically designed to refine model calibration without compromising accuracy by integrating label smoothing and confidence penalty techniques, tailored to the specifics of glaucoma detection. DesignThis study focuses on the development and evaluation of a calibrated deep learning model. ParticipantsThe study employs fundus images from both external datasets-ORIGA (482 normal, 168 glaucoma) and REFUGE (720 normal, 80 glaucoma)-and an extensive internal dataset (4,639 images per category), aiming to bolster the model's generalizability. The model's clinical performance is validated using a comprehensive test set (47,913 normal, 1,629 glaucoma) from the internal dataset. MethodsThe CC-LS loss function seamlessly integrates label smoothing, which tempers extreme predictions to avoid overfitting, with confidence-based penalties. These penalties deter the model from expressing undue confidence in incorrect classifications. Our study aims at training models using the CC-LS and comparing their performance with those trained using conventional loss functions. Main Outcome MeasuresThe model's precision is evaluated using metrics like the Brier score, sensitivity, specificity, and the false positive rate, alongside qualitative heatmap analyses for a holistic accuracy assessment. ResultsPreliminary findings reveal that models employing the CC-LS mechanism exhibit superior calibration metrics, as evidenced by a Brier score of 0.098, along with notable accuracy measures: sensitivity of 81%, specificity of 80%, and weighted accuracy of 80%. Importantly, these enhancements in calibration are achieved without sacrificing classification accuracy. ConclusionsThe Confidence-Calibrated Label Smoothing Loss presents a significant advancement in the pursuit of deploying machine learning models for glaucoma diagnosis. By improving calibration, the CC-LS ensures that clinicians can interpret and trust the predictive probabilities, making AI-driven diagnostic tools more clinically viable. From a clinical standpoint, this heightened trust and interpretability can potentially lead to more timely and appropriate interventions, thereby optimizing patient outcomes and safety.

An Improved Face Mask Detection Simulation Algorithm Based on YOLOv5 Model

This paper proposes an advanced approach for detecting faces with mask occlusion based on YOLOv5 to address various challenges encountered in face detection, including illumination blur and occlusion. The proposed methodology involves the integration of a convolutional block attention module into the backbone network and different network levels in the neck of the YOLOv5s. This approach enables the suppression of irrelevant features and emphasizes the identification of masked facial features. Replacing the conventional loss function with the Focal Loss function addresses the problem of sample imbalance. The enhanced YOLOv5s network was applied to detect mask-occluded faces. Empirical evaluations were conducted on the WIDER Face and AIZOO datasets. The simulation comparison results demonstrate that the proposed method achieves superior real-time detection performance, fulfilling the objective of developing a lightweight detection model.

More realistic degradation trend prediction for gas turbine based on factor analysis and multiple penalty mechanism loss function

Gas turbines play a crucial role in absorbing the volatility of new energy sources such as wind and photovoltaic. Continuous degradation trend prediction for gas turbines is vital for rationalizing maintenance schedules and improving power system stability. Current prediction techniques do not consider the practicality of the prediction results. To address this issue, a prediction framework based on factor analysis and Multiple Penalty Mechanisms (MPM) loss function is proposed. Firstly, factor analysis is used to assess the health index of gas turbines. Secondly, an innovative loss function that incorporates penalties for prediction errors, lag prediction, and fluctuation prediction is proposed to improve forecast usability. A range-adjustable and asymmetric Hyperbolic Cosine with Exponential (CoshE) function is first proposed to address the prediction lag problem. Finally, Long Short Term Memory network is chosen as the predictive model, and dynamic weights are used to optimize the loss function. Experiments on the combustion chamber degradation dataset and C-MAPSS dataset show that the framework proposed performs optimally than the conventional loss functions and the CoshE function is more efficient in the MPM framework. Meanwhile, MPM significantly improves gate recurrent unit and convolutional neural network performance. The method proposed is noteworthy for its superiority and applicability.

A deep learning-based Monte Carlo simulation scheme for stochastic differential equations driven by fractional Brownian motion

Stochastic differential equations (SDEs) are widely used models to describe the evolution of stochastic processes. Among them, SDEs driven by fractional Brownian motion (fBm) have been shown to be capable of describing systems with temporal dependencies. In this paper, we develop a neural network based Monte Carlo methodology in which we can efficiently simulate SDEs that are governed by fBm. Particularly, we focus on large time step simulations. A property of fBm that complicates the development of such Monte Carlo schemes is the long-range temporal correlation. To this end, we build the network based on the encoder–decoder framework and employ the attention mechanism to learn the temporal relationships in the historical paths of such SDEs. In addition, a loss function based on the quantile loss is used, where the quantile levels to be predicted are determined by means of the stochastic collocation method. Experimental results show that this kind of loss function is superior to conventional loss functions in terms of solution accuracy, and the resulting scheme can learn and simulate SDEs driven by fBm accurately and highly efficiently.

Mitigating Aberration-Induced Noise: A Deep Learning-Based Aberration-to-Aberration Approach.

One of the primary sources of suboptimal image quality in ultrasound imaging is phase aberration. It is caused by spatial changes in sound speed over a heterogeneous medium, which disturbs the transmitted waves and prevents coherent summation of echo signals. Obtaining non-aberrated ground truths in real-world scenarios can be extremely challenging, if not impossible. This challenge hinders the performance of deep learning-based techniques due to the domain shift between simulated and experimental data. Here, for the first time, we propose a deep learning-based method that does not require ground truth to correct the phase aberration problem and, as such, can be directly trained on real data. We train a network wherein both the input and target output are randomly aberrated radio frequency (RF) data. Moreover, we demonstrate that a conventional loss function such as mean square error is inadequate for training such a network to achieve optimal performance. Instead, we propose an adaptive mixed loss function that employs both B-mode and RF data, resulting in more efficient convergence and enhanced performance. Finally, we publicly release our dataset, comprising over 180,000 aberrated single plane-wave images (RF data), wherein phase aberrations are modeled as near-field phase screens. Although not utilized in the proposed method, each aberrated image is paired with its corresponding aberration profile and the non-aberrated version, aiming to mitigate the data scarcity problem in developing deep learning-based techniques for phase aberration correction. Source code and trained model are also available along with the dataset at http://code.sonography.ai/main-aaa.

Imbalanced learning for wind turbine blade icing detection via spatio-temporal attention model with a self-adaptive weight loss function

Accurate blade icing detection of wind turbines is of great significance to avoid secondary damages or accidents. Recently, data-driven methods have attracted increasing attention due to the availability of supervisory control and data acquisition (SCADA) data with their low-cost and high volume. However, SCADA data usually present complex characteristics of high dimensionality and strong spatio-temporal correlations among different sensor variables and severe class imbalance. It is challenging to extract effective features for accurate and reliable icing detection from imbalanced SCADA data. To address these challenges, we propose an imbalanced learning method for blade icing detection. First, we propose a spatio-temporal attention model to extract and assign weights to temporal-spatial features based on their contributions to blade icing detection. Then, to address the class imbalance issue, we design a self-adaptive weight (SAW) loss function to divide SCADA data into multiple batches and then classify them during model training. Unlike conventional loss functions, our SAW loss can adaptively assign weights to each data category according to their numbers in the divided batches. The effectiveness of the proposed method is verified on real SCADA datasets. The results demonstrate our designed spatio-temporal attention and SAW loss achieved better classification results and superior self-adaptability.

Adaptive Region-Specific Loss for Improved Medical Image Segmentation.

Defining the loss function is an important part of neural network design and critically determines the success of deep learning modeling. A significant shortcoming of the conventional loss functions is that they weight all regions in the input image volume equally, despite the fact that the system is known to be heterogeneous (i.e., some regions can achieve high prediction performance more easily than others). Here, we introduce a region-specific loss to lift the implicit assumption of homogeneous weighting for better learning. We divide the entire volume into multiple sub-regions, each with an individualized loss constructed for optimal local performance. Effectively, this scheme imposes higher weightings on the sub-regions that are more difficult to segment, and vice versa. Furthermore, the regional false positive and false negative errors are computed for each input image during a training step and the regional penalty is adjusted accordingly to enhance the overall accuracy of the prediction. Using different public and in-house medical image datasets, we demonstrate that the proposed regionally adaptive loss paradigm outperforms conventional methods in the multi-organ segmentations, without any modification to the neural network architecture or additional data preparation.

Federated Learning via Disentangled Information Bottleneck

Existing Federated Learning (FL) algorithms generally suffer from high communication costs and data heterogeneity due to the use of conventional loss function for local model update and the equal consideration of each local model for global model aggregation. In this paper, we propose a novel FL approach to address the above issues. For local model update, we propose a disentangled Information Bottleneck (IB) principle-based loss function. For global model aggregation, we suggest a model selection strategy based on Mutual Information (MI). Particularly, we design a Lagrangian-based loss function using the IB principle and “disentanglement” for maximizing MI between the ground truth and model prediction and minimizing MI between the intermediate representations. We calculate MI ratio between the ground truth and model prediction, and between the original input and ground truth to select the effective models for aggregation. We analyze the theoretical optimal cost of the loss function and manifest optimal convergence rate, and quantify the outlier robustness of the aggregation scheme. Experiments demonstrate the superiority of the proposed FL approach, in terms of testing performance and communication speedup (i.e., 3.00-14.88 times for IID MNIST, 2.5-50.75 times for non-IID MNIST, 1.87-18.40 times for IID CIFAR-10, and 1.24-2.10 times for non-IID MIMIC-III).

Analyzing the impact of activation functions on the performance of the data-driven gait model

Long-term prediction of joint kinematics is an important factor for the development of advanced technology in the area of the prosthetic leg, biped robot, automotive industry, and human-robot collaboration. It is well-defined that the long-term prediction of joint kinematic angle for an uneven surface is a challenging task. In previous work, authors have employed deep learning techniques for kinematic modelling using the collected walking dataset on an uneven surface with multiple speeds. However, it is a challenging task to find the appropriate activation function for deep learning techniques. Therefore, this research work focuses on the suitability of the activation function for a data-driven gait model for multiple slopes and inclines as ground conditions. Twenty-five activation function with their time complexity is extensively studied. In addition, the fusion of standard error from the subject mean trajectory with conventional loss function is also presented. This fusion helps the training algorithm to train the data-driven model to follow the low variance point in the gait cycle more precisely. The result shows that the Sigmoid-weighted Linear Units (SiLU) activation function-based gait model outperforms the others based on the maximum error summary statistic. It is also established that the SiLU function-based gait model trained by fused objective function significantly improved the long-term prediction for two cases: (a) step change slope and (b) continuously varying slope.

Customized deep learning for precipitation bias correction and downscaling

Abstract. Systematic biases and coarse resolutions are major limitations of current precipitation datasets. Many deep learning (DL)-based studies have been conducted for precipitation bias correction and downscaling. However, it is still challenging for the current approaches to handle complex features of hourly precipitation, resulting in the incapability of reproducing small-scale features, such as extreme events. This study developed a customized DL model by incorporating customized loss functions, multitask learning and physically relevant covariates to bias correct and downscale hourly precipitation data. We designed six scenarios to systematically evaluate the added values of weighted loss functions, multitask learning, and atmospheric covariates compared to the regular DL and statistical approaches. The models were trained and tested using the Modern-era Retrospective Analysis for Research and Applications version 2 (MERRA2) reanalysis and the Stage IV radar observations over the northern coastal region of the Gulf of Mexico on an hourly time scale. We found that all the scenarios with weighted loss functions performed notably better than the other scenarios with conventional loss functions and a quantile mapping-based approach at hourly, daily, and monthly time scales as well as extremes. Multitask learning showed improved performance on capturing fine features of extreme events and accounting for atmospheric covariates highly improved model performance at hourly and aggregated time scales, while the improvement is not as large as from weighted loss functions. We show that the customized DL model can better downscale and bias correct hourly precipitation datasets and provide improved precipitation estimates at fine spatial and temporal resolutions where regular DL and statistical methods experience challenges.

Loss Function Design for Data-Driven Predictors to Enhance the Energy Efficiency of Connected and Automated Vehicles

In car-following scenarios, accurate previews of the preceding vehicle’s trajectory are essential for minimizing the energy consumption of a following automated vehicle. This work presents a novel design strategy for data-driven vehicle speed predictors to increase the energy efficiency of the following connected and automated vehicle. The loss function is formulated as a weighted-mean-squared error, where the weights are tuned based on the influence of uncertainty at an individual prediction step on the energy consumption of the automated following vehicle. The efficacy of the proposed loss function is validated by applying it to training representative predictors and testing the predictors in the energy-optimal control of the following vehicle. The energy saving of the optimal controller is evaluated by comparing the electricity consumption of a battery electric vehicle with the human driver following, emulated by an intelligent driver model. Simulation results show that the energy consumption of an electric vehicle is reduced by an average of 12% compared to a human driver following demonstrated by an intelligent driver model, whereas using a conventional loss function, a mean-squared error loss function, reduces by 10%.

Fast and accurate learned multiresolution dynamical downscaling for precipitation

Abstract. This study develops a neural-network-based approach for emulating high-resolution modeled precipitation data with comparable statistical properties but at greatly reduced computational cost. The key idea is to use combination of low- and high-resolution simulations (that differ not only in spatial resolution but also in geospatial patterns) to train a neural network to map from the former to the latter. Specifically, we define two types of CNNs, one that stacks variables directly and one that encodes each variable before stacking, and we train each CNN type both with a conventional loss function, such as mean square error (MSE), and with a conditional generative adversarial network (CGAN), for a total of four CNN variants. We compare the four new CNN-derived high-resolution precipitation results with precipitation generated from original high-resolution simulations, a bilinear interpolater and the state-of-the-art CNN-based super-resolution (SR) technique. Results show that the SR technique produces results similar to those of the bilinear interpolator with smoother spatial and temporal distributions and smaller data variabilities and extremes than the original high-resolution simulations. While the new CNNs trained by MSE generate better results over some regions than the interpolator and SR technique do, their predictions are still biased from the original high-resolution simulations. The CNNs trained by CGAN generate more realistic and physically reasonable results, better capturing not only data variability in time and space but also extremes such as intense and long-lasting storms. The new proposed CNN-based downscaling approach can downscale precipitation from 50 to 12 km in 14 min for 30 years once the network is trained (training takes 4 h using 1 GPU), while the conventional dynamical downscaling would take 1 month using 600 CPU cores to generate simulations at the resolution of 12 km over the contiguous United States.

Objective-sensitive principal component analysis for high-dimensional inverse problems

We introduce a novel approach of data-driven dimensionality reduction for solving high-dimensional optimization problems, including history matching. Objective-Sensitive parameterization of the argument accounts for the corresponding change of objective function value. The result is achieved via an extension of the conventional loss function, which only quantifies approximation error over realizations. This paper contains three instances of such an approach based on Principal Component Analysis (PCA). Gradient-Sensitive PCA (GS-PCA) exploits a linear approximation of the objective function. Two other approaches solve the problem approximately within the framework of stationary perturbation theory (SPT). All the algorithms are verified and tested with a synthetic reservoir model. The results demonstrate improvements in parameterization quality regarding the reveal of the unconstrained objective function minimum. Also, we provide possible extensions and analyze the overall applicability of the Objective-Sensitive approach, which can be combined with modern parameterization techniques beyond PCA.

Machine Learning Prediction of Storm‐Time High‐Latitude Ionospheric Irregularities From GNSS‐Derived ROTI Maps

AbstractThis study presents an image‐based convolutional long short‐term memory (convLSTM) machine learning algorithm to predict storm‐time ionospheric irregularities. Unlike existing methods that are either focused on irregularities at individual locations or treat the irregularity prediction as a classification problem, the convLSTM‐based architecture forecasts an entire regions' ionospheric irregularity occurrence and intensity values. We implemented the convLSTM‐based model with a custom‐designed loss function (convLSTM‐Lc) that includes a dynamic penalty on the difference between the truth and the predicted rate of total electron content index (ROTI) maps. The convLSTM‐Lc is trained with real ROTI data collected during January 1–August 7, 2015 from ∼550 global navigation satellite system receivers located in (45°–90°N, 0°–180°W). Test results show that the convLSTM‐Lc algorithm can forecast irregularity structures more accurately than a convLSTM model that implements conventional loss functions. The model also outperforms the convLSTM‐L1, convLSTM‐L2, and persistence models according to statistical classification metrics with a lead time of up to 60 min.

Deep Low-Contrast Image Enhancement using Structure Tensor Representation

We present a new deep learning framework for low-contrast image enhancement, which trains the network using the multi-exposure sequences rather than explicit ground-truth images. The purpose of our method is to enhance a low-contrast image so as to contain abundant details in various exposure levels. To realize this, we propose to design the loss function using the structure tensor representation, which has been widely used as high-dimensional image contrast. Our loss function penalizes the difference of the structure tensor between the network output and the multi-exposure images in a multi-scale manner. Eventually, the network trained by the loss function produces a high-quality image approximating the overall contrast of the sequence. We provide in-depth analysis on our method and comparison with conventional loss functions. Quantitative and qualitative evaluations demonstrate that the proposed method outperforms the existing state-of-the-art approaches in various benchmarks.

Imbalanced Loss-Integrated Deep-Learning-Based Ultrasound Image Analysis for Diagnosis of Rotator-Cuff Tear.

A rotator cuff tear (RCT) is an injury in adults that causes difficulty in moving, weakness, and pain. Only limited diagnostic tools such as magnetic resonance imaging (MRI) and ultrasound Imaging (UI) systems can be utilized for an RCT diagnosis. Although UI offers comparable performance at a lower cost to other diagnostic instruments such as MRI, speckle noise can occur the degradation of the image resolution. Conventional vision-based algorithms exhibit inferior performance for the segmentation of diseased regions in UI. In order to achieve a better segmentation for diseased regions in UI, deep-learning-based diagnostic algorithms have been developed. However, it has not yet reached an acceptable level of performance for application in orthopedic surgeries. In this study, we developed a novel end-to-end fully convolutional neural network, denoted as Segmentation Model Adopting a pRe-trained Classification Architecture (SMART-CA), with a novel integrated on positive loss function (IPLF) to accurately diagnose the locations of RCT during an orthopedic examination using UI. Using the pre-trained network, SMART-CA can extract remarkably distinct features that cannot be extracted with a normal encoder. Therefore, it can improve the accuracy of segmentation. In addition, unlike other conventional loss functions, which are not suited for the optimization of deep learning models with an imbalanced dataset such as the RCT dataset, IPLF can efficiently optimize the SMART-CA. Experimental results have shown that SMART-CA offers an improved precision, recall, and dice coefficient of 0.604% (+38.4%), 0.942% (+14.0%) and 0.736% (+38.6%) respectively. The RCT segmentation from a normal ultrasound image offers the improved precision, recall, and dice coefficient of 0.337% (+22.5%), 0.860% (+15.8%) and 0.484% (+28.5%), respectively, in the RCT segmentation from an ultrasound image with severe speckle noise. The experimental results demonstrated the IPLF outperforms other conventional loss functions, and the proposed SMART-CA optimized with the IPLF showed better performance than other state-of-the-art networks for the RCT segmentation with high robustness to speckle noise.

Predicting Coordinated Actuated Traffic Signal Change Times using Long Short-Term Memory Neural Networks

Vehicle acceleration and deceleration maneuvers at traffic signals result in significant fuel and energy consumption levels. Green light optimal speed advisory systems require reliable estimates of signal switching times to improve vehicle energy/fuel efficiency. Obtaining these estimates is difficult for actuated signals where the length of each green indication changes to accommodate varying traffic conditions and pedestrian requests. This study details a four-step long short-term memory (LSTM) deep learning based methodology that can be used to provide reasonable switching time estimates from green to red and vice versa while being robust to missing data. The four steps are data gathering, data preparation, machine learning model tuning, and model testing and evaluation. The input to the models includes controller logic, signal timing parameters, time of day, traffic state from detectors, vehicle actuation data, and pedestrian actuation data. The methodology is applied and evaluated on data from an intersection in Northern Virginia. A comparative analysis is conducted between different loss functions including the mean squared error, mean absolute error, and mean relative error used in LSTM and a new loss function that is proposed in this paper. The results show that while the proposed loss function outperforms conventional loss functions in overall absolute error values, the choice of the loss function is dependent on the prediction horizon. Specifically, the proposed loss function is slightly outperformed by the mean relative error for very short prediction horizons (less than 20 s) and the mean squared error for very long prediction horizons (greater than 120 s).

A conditional Triplet loss for few-shot learning and its application to image co-segmentation

Few-shot learning tries to solve the problems that suffer the limited number of samples. In this paper we present a novel conditional Triplet loss for solving few-shot problems using deep metric learning. While the conventional Triplet loss suffers the limitation of random sampling of triplets which leads to slow convergence in training process, our proposed network tries to distinguish between samples so that it improves the training speed. Our main contributions are two-fold. (i) We propose a conditional Triplet loss to train a deep Triplet network for deep metric embedding. The proposed Triplet loss employs a penalty–reward technique to enhance the convergence of standard Triplet loss. (ii) We improve the performance of the existing image co-segmentation model by replacing the conventional loss function by our proposed conditional Triplet loss. To demonstrate the performance of the proposed network, experiments carry out on MNIST and CIFAR. Simulation results are evaluated by AUC and Recall (sensitivity) and indicate that the proposed conditional Triplet network achieves higher accuracy in comparison to state-of-the-arts.