Design Of Loss Function Research Articles

Leveraging Incremental Learning for Dynamic Modulation Recognition

Modulation recognition is an important technology used to correctly identify the modulation modes of wireless signals and is widely used in cooperative and confrontational scenarios. Traditional modulation-recognition algorithms require the assistance of expert experiences, which constrains their applications. With the rapid development of artificial intelligence in recent years, deep learning (DL) is widely advocated for intelligent modulation recognition. Typically, DL-based modulation-recognition algorithms implicitly assume a relatively static scenario in which the signal samples of all the modulation modes can be completely collected in advance. In practical situations, the radio environment is quite dynamic and the signal samples with new modulation modes may appear sequentially, in which the current DL-based modulation-recognition algorithms may require unacceptable time and computing resource consumption to re-train the optimal model from scratch. In this study, we leveraged incremental learning (IL) and designed a novel IL-based modulation-recognition algorithm that consists of an initial stage and multiple incremental stages. The main novelty of the proposed algorithm lies in the new loss function design in each incremental stage, which combines the distillation loss of recognizing old modulation modes and the cross-entropy loss of recognizing new modulation modes. With the proposed algorithm, the knowledge of the signal samples with new modulation modes can be efficiently learned in the current stage without forgetting the knowledge learned in the previous stages. The simulation results demonstrate that the proposed algorithm could achieve a recognition accuracy close to the upper bound with a much lower computing time and it outperformed the existing IL-based benchmarks.

A medical image classification method based on self-regularized adversarial learning.

Deep learning (DL) techniques have been extensively applied in medical image classification. The unique characteristics of medical imaging data present challenges, including small labeled datasets, severely imbalanced class distribution, and significant variations in imaging quality. Recently, generative adversarial network (GAN)-based classification methods have gained attention for their ability to enhance classification accuracy by incorporating realistic GAN-generated images as data augmentation. However, the performance of these GAN-based methods often relies on high-quality generated images, while large amounts of training data are required to train GAN models to achieve optimalperformance. In this study, we propose an adversarial learning-based classification framework to achieve better classification performance. Innovatively, GAN models are employed as supplementary regularization terms to support classification, aiming to address the challenges describedabove. The proposed classification framework, GAN-DL, consists of a feature extraction network (F-Net), a classifier, and two adversarial networks, specifically a reconstruction network (R-Net) and a discriminator network (D-Net). The F-Net extracts features from input images, and the classifier uses these features for classification tasks. R-Net and D-Net have been designed following the GAN architecture. R-Net employs the extracted feature to reconstruct the original images, while D-Net is tasked with the discrimination between the reconstructed image and the original images. An iterative adversarial learning strategy is designed to guide model training by incorporating multiple network-specific loss functions. These loss functions, serving as supplementary regularization, are automatically derived during the reconstruction process and require no additional dataannotation. To verify the model's effectiveness, we performed experiments on two datasets, including a COVID-19 dataset with 13958 chest x-ray images and an oropharyngeal squamous cell carcinoma (OPSCC) dataset with 3255 positron emission tomography images. Thirteen classic DL-based classification methods were implemented on the same datasets for comparison. Performance metrics included precision, sensitivity, specificity, and -score. In addition, we conducted ablation studies to assess the effects of various factors on model performance, including the network depth of F-Net, training image size, training dataset size, and loss function design. Our method achieved superior performance than all comparative methods. On the COVID-19 dataset, our method achieved , , , and in terms of precision, sensitivity, specificity, and -score, respectively. It achieved across all these metrics on the OPSCC dataset. The study to investigate the effects of two adversarial networks highlights the crucial role of D-Net in improving model performance. Ablation studies further provide an in-depth understanding of ourmethodology. Our adversarial-based classification framework leverages GAN-based adversarial networks and an iterative adversarial learning strategy to harness supplementary regularization during training. This design significantly enhances classification accuracy and mitigates overfitting issues in medical image datasets. Moreover, its modular design not only demonstrates flexibility but also indicates its potential applicability to various clinical contexts and medical imaging applications.

Physics Constrained High-Precision Data-Driven Modeling for Multi-Path Ultrasonic Flow Meter in Natural Gas Measurement.

Ultrasonic flow meters are crucial measuring instruments in natural gas transportation pipeline scenarios. The collected flow velocity data, along with the operational conditions data, are vital for the analysis of the metering performance of ultrasonic flow meters and analysis of the flow process. In practical applications, high requirements are placed on the modeling accuracy of ultrasonic flow meters. In response, this paper proposes an ultrasonic flow meter modeling method based on a combination of data learning and industrial physics knowledge. This paper builds ultrasonic flow meter flow velocity prediction models under different working conditions, combining pipeline flow field velocity distribution knowledge for data preprocessing and loss function design. By making full use of the characteristics of the physics and data learning, the prediction results are close to the real acoustic path flow velocity distribution; thus, the model has high accuracy and interpretability. Experiments are conducted to prove that the prediction error of the proposed method can be controlled within 1%, which can meet the needs of ultrasonic flow meter modeling and subsequent performance analysis in actual production.

Multi-objective explainable smart dispatch for integrated energy system based on an explainable MO-RL method

With the rapid development of renewable energy penetration and the transition to high-efficiency, low-carbon-footprint energy systems, renewable-based heat-electricity integrated energy systems (RHE-IES) have become a viable method. The dispatch and operation scheduling of the RHE-IES is more complex than before because a well-operated RHE-IES should be economically favorable, environmentally friendly, and feasible with respect to security constraints. However, it is not easy to balance these trade-offs with uncertainties. In this work, a multi-objective hierarchical deep reinforcement learning (H-DRL) method is proposed to address the dispatch of RHE-IES, and it is also explainable due to the design of the algorithmic structure, the decoupling of rewards, and the analysis of feature importance. First, the dispatch and operation scheduling problem of the RHE-IES is formulated considering economic, environmental, and security constraints and is converted to a Markov decision process. Second, a novel H-DRL method is proposed with a multi-critic, single-actor structure, where each objective is handled by a designated deep critic network with decoupled rewards, and the action value functions of different objectives are hierarchically optimized by the actor network. H-DRL can be trained in a comprehensive environment offline and provide fast online decisions considering the uncertainties of renewable generation. Third, a shapley additive explanations method is carried out utilizing the explainable nature of the H-DRL structure and the loss function design. The simulation results and case studies show the effectiveness of the proposed method in terms of balancing the tradeoffs of different objectives, overcoming uncertainties and enhancing AI interpretability.

Class-Attribute Priors: Adapting Optimization to Heterogeneity and Fairness Objective

Modern classification problems exhibit heterogeneities across individual classes: Each class may have unique attributes, such as sample size, label quality, or predictability (easy vs difficult), and variable importance at test-time. Without care, these heterogeneities impede the learning process, most notably, when optimizing fairness objectives. Confirming this, under a gaussian mixture setting, we show that the optimal SVM classifier for balanced accuracy needs to be adaptive to the class attributes. This motivates us to propose CAP: An effective and general method that generates a class-specific learning strategy (e.g.~hyperparameter) based on the attributes of that class. This way, optimization process better adapts to heterogeneities. CAP leads to substantial improvements over the naive approach of assigning separate hyperparameters to each class. We instantiate CAP for loss function design and post-hoc logit adjustment, with emphasis on label-imbalanced problems. We show that CAP is competitive with prior art and its flexibility unlocks clear benefits for fairness objectives beyond balanced accuracy. Finally, we evaluate CAP on problems with label noise as well as weighted test objectives to showcase how CAP can jointly adapt to different heterogeneities.

Knowledge distillation based on projector integration and classifier sharing

Knowledge distillation can transfer the knowledge from the pre-trained teacher model to the student model, thus effectively accomplishing model compression. Previous studies have carefully crafted knowledge representation, targeting loss function design, and distillation location selection, but there have been few studies on the role of classifiers in distillation. Previous experiences have shown that the final classifier of the model has an essential role in making inferences, so this paper attempts to narrow the gap in performance between models by having the student model directly use the classifier of the teacher model for the final inference, which requires an additional projector to help match features of the student encoder with the teacher's classifier. However, a single projector cannot fully align the features, and integrating multiple projectors may result in better performance. Considering the balance between projector size and performance, through experiments, we obtain the size of projectors for different network combinations and propose a simple method for projector integration. In this way, the student model undergoes feature projection and then uses the classifiers of the teacher model for inference, obtaining a similar performance to the teacher model. Through extensive experiments on the CIFAR-100 and Tiny-ImageNet datasets, we show that our approach applies to various teacher–student frameworks simply and effectively.

LiViT-Net: A U-Net-like, lightweight Transformer network for retinal vessel segmentation

The intricate task of precisely segmenting retinal vessels from images, which is critical for diagnosing various eye diseases, presents significant challenges for models due to factors such as scale variation, complex anatomical patterns, low contrast, and limitations in training data. Building on these challenges, we offer novel contributions spanning model architecture, loss function design, robustness, and real-time efficacy. To comprehensively address these challenges, a new U-Net-like, lightweight Transformer network for retinal vessel segmentation is presented. By integrating MobileViT+ and a novel local representation in the encoder, our design emphasizes lightweight processing while capturing intricate image structures, enhancing vessel edge precision. A novel joint loss is designed, leveraging the characteristics of weighted cross-entropy and Dice loss to effectively guide the model through the task's challenges, such as foreground-background imbalance and intricate vascular structures. Exhaustive experiments were performed on three prominent retinal image databases. The results underscore the robustness and generalizability of the proposed LiViT-Net, which outperforms other methods in complex scenarios, especially in intricate environments with fine vessels or vessel edges. Importantly, optimized for efficiency, LiViT-Net excels on devices with constrained computational power, as evidenced by its fast performance. To demonstrate the model proposed in this study, a freely accessible and interactive website was established (https://hz-t3.matpool.com:28765?token=aQjYR4hqMI), revealing real-time performance with no login requirements.

Loss Function Design for Wrapped Phase Fitting in InSAR Deep Learning Network Training

Loss Function Design for Wrapped Phase Fitting in InSAR Deep Learning Network Training

Towards Food Image Retrieval via Generalization-Oriented Sampling and Loss Function Design

Food computing has increasingly received widespread attention in the multimedia field. As a basic task of food computing, food image retrieval has wide applications, that is, food image retrieval can help users to find the desired food from a large number of food images. Besides, the retrieved information can be applied to establish a richer database for the subsequent food content-related recommendation. Food image retrieval aims to achieve better performance on novel categories. Thus, it is worth studying to transfer the embedding ability from the training set to the unseen test set, that is, the generalization of the model. Food is influenced by various factors, such as culture and geography, leading to great differences between domains, such as Asian food and western food. Therefore, it is challenging to study the generalization of the model in food image retrieval. In this article, we improve the classical metric learning framework and propose a generalization-oriented sampling strategy, which boosts the generalization of the model by maximizing the intra-class distance from a proportion of positive pairs to avoid the excessive distance compression in the embedding space. Considering that the existing optimization process is in an opposite direction to our proposed sampling strategy, we further propose an adaptive gradient assignment policy named gradient-adaptive optimization , which can alleviate the intra-class distance compression during optimization by assigning different gradients to different samples. Extensive evaluation on three popular food image datasets demonstrates the effectiveness of the proposed method. We also experiment on three popular general datasets to prove that solving the problem from the generalization can also improve the performance of general image retrieval. Code is available at https://github.com/Jiajun-ISIA/Generalization-oriented-Sampling-and-Loss .

Effects of Different Full-Reference Quality Assessment Metrics in End-to-End Deep Video Coding

Visual quality assessment is often used as a key performance indicator (KPI) to evaluate the performance of electronic devices. There exists a significant association between visual quality assessment and electronic devices. In this paper, we bring attention to alternative choices of perceptual loss function for end-to-end deep video coding (E2E-DVC), which can be used to reduce the amount of data generated by electronic sensors and other sources. Thus, we analyze the effects of different full-reference quality assessment (FR-QA) metrics on E2E-DVC. First, we select five optimization-suitable FR-QA metrics as perceptual objectives, which are differentiable and thus support back propagation, and use them to optimize an E2E-DVC model. Second, we analyze the rate–distortion (R-D) behaviors of an E2E-DVC model under different loss function optimizations. Third, we carry out subjective human perceptual tests on the reconstructed videos to show the performance of different FR-QA optimizations on subjective visual quality. This study reveals the effects of the competing FR-QA metrics on E2E-DVC and provides a guide for further future study on E2E-DVC in terms of perceptual loss function design.

A method of knowledge distillation based on feature fusion and attention mechanism for complex traffic scenes

Object detectors based on deep learning can run smoothly on a terminal device in complex traffic scenes, and the model compression method has become a research hotspot. Considering student network single learning in the knowledge distillation algorithm, the dependence on loss function design leads to parameter sensitivity and other problems, we propose a new knowledge distillation method with second-order term attention mechanisms and feature fusion of adjacent layers. First, we build a knowledge distillation framework based on YOLOv5 and propose a new attention mechanism in the teacher network backbone to extract the hot map. Then, we combine the hot map features with the next level features through the fusion module. By fusing the useful information of the low convolution layer and the feature map of the high convolution layer to help the student network obtain the final prediction map. Finally, to improve the accuracy of small objects, we add a 160 × 160 detection head and use a transformer encoder block module to replace the convolution network of the head. Sufficient experimental results show that our method achieves state-of-the-art performance. The speed and number of parameters remain unchanged, but the average detection accuracy is 97.4% on the KITTI test set. On the Cityscapes test set, the average detection accuracy reaches 92.7%.

Efficient Carrier Frequency Offset Estimation in Wireless Sensor Networks

The carrier frequency offset (CFO) estimation is a crucial problem in orthogonal frequency division multiplexing (OFDM) systems, especially for the enabling technologies in the Internet of Things, which demand stringent limitations of low cost, low power consumption, and wide ranges, like wireless sensor networks. In this letter, we propose an efficient CFO estimation method for OFDM systems with in- and quadrature-phase (IQ) imbalance, which combines the emerging multitask learning (MTL) with the channel residual energy (CRE) to reduce the average inferring time greatly. The MTL estimator can extract the correlated features between the CFO and the IQ imbalance so as to mitigate the intractable coupling effect. On the other hand, the CRE technique can guide the neural network and loss function design to improve the estimation accuracy and the training efficiency. Numerical results demonstrate that our proposal achieves high estimation accuracy while the inferring time is significantly reduced compared with other methods.

Iterative deblending using unsupervised learning with double-deep neural networks

Simultaneous source acquisition technology can greatly improve seismic acquisition efficiency. However, due to continuous shooting and serious crosstalk noise of the adjacent sources in seismic data, simultaneous source data cannot be directly used in conventional data processing procedures. Therefore, simultaneous source data need to be deblended to obtain the conventional shot record. Under densely sampled sources, we have developed a novel unsupervised deep learning (UDL) method based on the double-deep neural networks for iterative inversion deblending of simultaneous source data. Our UDL, which is mainly composed of the residual neural network (R-net) and the U-net neural network, has excellent nonlinear optimization ability. The total loss function design can optimize our UDL in the correct direction and avoid the problem of overfitting. By minimizing the total loss function, the R-net and U-net branches of the UDL can extract the coherent effective signals of all sources and suppress the crosstalk noise. The most prominent advantage of our UDL method is that it does not require label data, and the training data set does not contain raw unblended data, thus solving the problem of missing training data sets. The examples with two synthetic and one field data set are used to prove the effectiveness of iterative inversion deblending of simultaneous source data based on our UDL method when sources are within a small distance of each other. By comparing our UDL method with the traditional curvelet-based and contourlet-based methods, the superiority of our method in the quality of separation results is demonstrated.

Physics-Driven Deep Learning Methods for Fast Quantitative Magnetic Resonance Imaging: Performance improvements through integration with deep neural networks

Quantitative magnetic resonance imaging (qMRI) aims to obtain quantitative biophysical parameters based on physical models derived from MR spin magnetization evolution. This requires the acquisition of multiple MR images, resulting in very long scan times. Recently, deep learning (DL) has shown significant potential for reconstructing undersampled MR data. When applying DL to fast qMRI, physical priors can be integrated into the DL framework to improve the performance of qMRI further. This article introduces the physical models of qMRI and four ways to integrate these models into a deep neural network (NN), namely, training sample generation, contrast image prediction, loss function design, and network architecture design. The core concepts and methods for each category are presented. The associated signal processing issues regarding the generalization and reliability of physics-driven DL-based fast qMRI methods are also discussed.

Loss Function Design for Data-Driven Predictors to Enhance the Energy Efficiency of Connected and Automated Vehicles

In car-following scenarios, accurate previews of the preceding vehicle’s trajectory are essential for minimizing the energy consumption of a following automated vehicle. This work presents a novel design strategy for data-driven vehicle speed predictors to increase the energy efficiency of the following connected and automated vehicle. The loss function is formulated as a weighted-mean-squared error, where the weights are tuned based on the influence of uncertainty at an individual prediction step on the energy consumption of the automated following vehicle. The efficacy of the proposed loss function is validated by applying it to training representative predictors and testing the predictors in the energy-optimal control of the following vehicle. The energy saving of the optimal controller is evaluated by comparing the electricity consumption of a battery electric vehicle with the human driver following, emulated by an intelligent driver model. Simulation results show that the energy consumption of an electric vehicle is reduced by an average of 12% compared to a human driver following demonstrated by an intelligent driver model, whereas using a conventional loss function, a mean-squared error loss function, reduces by 10%.

Balanced masking strategy for multi-label image classification

Data imbalance is an essential issue in multi-label image classification that may reduce the model’s generalization ability. Unlike oversampling or undersampling in single-label image classification, simply removing or repeating some images in multi-label classification will influence labels in other categories due to complex label dependencies. Current methods usually tackle this issue from either data resampling or loss function design. This paper proposes a balanced masking strategy to solve the imbalance problem in feature space for graph-based methods. By selectively manipulating the label node embedding of the graph, the model can gain the priors of the majority of samples to maintain intra-class and inter-class balance simultaneously, thus performing better and more stable. Extensive experiments verify the adaptability and effectiveness of our method, which can be easily applied to any graph-based multi-label model to achieve highly competitive performance.

CR Loss: Improving Biometric Using ClassRoom Learning Approach

Abstract One of the important factors in deep feature learning is their loss function design which highly influences the network performance. In this paper, we proposed a classroom (CR) learning approach along with arcface loss for contactless palmprint recognition to obtain high-level discriminative features without any extra load made to the network architecture. CR loss allows the network to learn the best possible feature representations for palmprint images. To validate our concept, we performed extensive experimental evaluations on various popular benchmark palmprint databases where our methods outperform the state-of-the-art methods. We also introduced a challenging contactless palmprint database called Harbin Institute of Technology-Network & Information Security Research Center contactless palmprint database version 1.0 (HIT-NIST-V1), as a new contribution to this domain. The result proves that the proposed CR loss consistently outperforms the SOTA methods for all the considered databases and especially for HIT-NIST-V1.

Improved sales time series predictions using deep neural networks with spatiotemporal dynamic pattern acquisition mechanism

The ability to predict product sales is invaluable for improving many of the routine decisions essential for the running of an enterprise. One significant challenge of sales prediction is that it is hard to dynamically capture changing dependent patterns along the sales time line, because sales are often influenced by complicated and changeable market environment. To address this issue, we model sales prediction as a task of multivariate time series (MTS) prediction, and propose a Spatiotemporal Dynamic Pattern Acquisition Mechanism (SDPA), which comprises four components, described below: (1) In the processing of input data: A Spatiotemporal Dynamic Kernel (SDK) component is designed for MTS to effectively capture different dependent correlation patterns during different time periods. (2) In terms of model design: A Simultaneous Regression (SR) component is proposed to dynamically detect stable correlations by using co-integration based dynamic programming over different time periods. (3) A novel Hierarchical Attention (HA) component is designed to incorporate SDK to detect spatiotemporal attention patterns from the captured dynamic correlations. (4) In the design of loss function, A Change Sensitive and Alignment component (DC) is proposed to provide more future information based on future trend correlations for better model training. The four components are incorporated into a unified framework by considering Homovariance Uncertainty (HU). This is referred to as SDPANet and contributes to model training and sales prediction. Extensive experiments were conducted on two real-world datasets: Galanz and Cainiao, and experimental results show that the proposed method achieves statistically significant improvements compared to the most state-of-the-art baselines, with average 41.5% reduction on RMAE, average 39.5% reduction on RRSE and average 46% improvement on CORR. Experiments are also conducted on two new datasets, which are Traffic and Exchange-Rate from other fields, to further verify the effectiveness of the proposed model. Case studies show that the model is capable of capturing dynamic changing patterns and of predicting future sales trends with greater accuracy.

Heterogeneous data fusion and loss function design for tooth point cloud segmentation

Heterogeneous data fusion and loss function design for tooth point cloud segmentation

Impact of loss functions on semantic segmentation in far‐field monitoring

AbstractAlthough previous research laid the foundation for vision‐based monitoring systems using convolutional neural networks (CNNs), too little attention has been paid to the challenges associated with data imbalance and varying object sizes in far‐field monitoring. To fill the knowledge gap, this paper investigates various loss functions to design a customized loss function to address the challenges. Scaffold installation operations recorded by camcorders were selected as the subject of analysis in a far‐field surveillance setting. It was confirmed that the data imbalance between the workers, hardhats, harnesses, straps, and hooks caused poor performances especially for small size objects. This problem was mitigated by employing a region‐based loss and Focal loss terms in the loss function of segmentation models. The findings illustrate the importance of the loss function design in improving performance of CNN models for far‐field construction site monitoring.