Adaptive Loss Function Research Articles

SMWE-GFPNNet: A high-precision and robust method for forest fire smoke detection

Smoke is an early manifestation of forest fire. Accurate identification of smoke from forest fires is crucial for the prevention and control of forest fires, which helps protect the ecological environment and the safety of people. The texture features of smoke are complex and prone to detection omissions. The forest environment is complex, and smoke-like objects in the forest often interfere with smoke recognition. The concentration of smoke at the edge is thin, which easily leads to edge omission. In response to these problems, we propose a high-precision edge focused forest fire smoke detection network. To begin, in response to the problem of detection omission, we present a Swin multidimensional window extractor (SMWE) that enhances information exchange between windows in both horizontal and vertical dimensions to extract global texture features from images with smoke. Then, the guillotine feature pyramid network (GFPN) is suggested, along with a new guillotine convolution method for reducing redundant feature information from a feature fusion perspective, thereby improving the anti-interference ability of the model. Finally, taking into account the thinness and irregularity of the smoke near the borders, a contour adaptive loss function is suggested to minimize the boundary blur caused by down-sampling the feature map in the network. The experimental and application results show that SMWE-GFPNNet accomplishes 80.92 % of the mAP, 90.01 % of the mAP50, and 83.38 % of the mAP75 on the Forest Fire Smoke Complex Background Detection Dataset. Excellent in anti-interference ability and accuracy.

A recurrent positional encoding circular attention mechanism network for biomedical image segmentation

Deep-learning-based medical image segmentation techniques can assist doctors in disease diagnosis and rapid treatment. However, existing medical image segmentation models do not fully consider the dependence between feature segments in the feature extraction process, and the correlated features can be further extracted. Therefore, a recurrent positional encoding circular attention mechanism network (RPECAMNet) is proposed based on relative positional encoding for medical image segmentation. Multiple residual modules are used to extract the primary features of the medical images, which are thereafter converted into one-dimensional data for relative positional encoding. The recursive former is used to further extract features from medical images, and decoding is performed using deconvolution. An adaptive loss function is designed to train the model and achieve accurate medical-image segmentation. Finally, the proposed model is used to conduct comparative experiments on the synapse and self-constructed kidney datasets to verify the accuracy of the proposed model for medical image segmentation.

Thyroid ultrasound diagnosis improvement via multi-view self-supervised learning and two-stage pre-training

Thyroid nodule classification and segmentation in ultrasound images are crucial for computer-aided diagnosis; however, they face limitations owing to insufficient labeled data. In this study, we proposed a multi-view contrastive self-supervised method to improve thyroid nodule classification and segmentation performance with limited manual labels. Our method aligns the transverse and longitudinal views of the same nodule, thereby enabling the model to focus more on the nodule area. We designed an adaptive loss function that eliminates the limitations of the paired data. Additionally, we adopted a two-stage pre-training to exploit the pre-training on ImageNet and thyroid ultrasound images. Extensive experiments were conducted on a large-scale dataset collected from multiple centers. The results showed that the proposed method significantly improves nodule classification and segmentation performance with limited manual labels and outperforms state-of-the-art self-supervised methods. The two-stage pre-training also significantly exceeded ImageNet pre-training.

A novel autoencoder for structural anomalies detection in river tunnel operation

Anomaly diagnosis is essential to prevent disasters and ensure long-term stable operation of tunnels. However, the diversity and scarcity of abnormal data make it difficult to identify outliers, especially to diagnose structural anomalies from poor-quality data. Therefore, an adaptive loss function improved autoencoder (AdaAE) model is proposed for anomaly detection, which is robust to poor-quality data and can accurately determine the anomaly source of river tunnel. To expand the abnormal dataset, numerical simulation of structure under extreme conditions and Gaussian noise are adopted to construct structural damage data and disturbance data respectively. The proposed model is then instantiated on the prepared dataset. Finally, the reliability and the advantage of the proposed model are verified by ablation study. The research results indicate that the detection ability of AdaAE model is greatly improved to that of the widely used methods. This model is suitable to poor quality dataset, and the accuracy to detect structural anomalies from pollution data sets is more than 90%. As a case study, the AdaAE model is applied to the Wuhan Yangtze River tunnel to detect anomalies of segment strain. This study would play a role in preventing structural diseases and promoting intelligent management during tunnel operation.

Adaptive deep neural networks for solving corner singular problems

Deep neural networks (DNNs) for numerical solutions to partial differential equations (PDEs) have exhibited their remarkable merits of meshless methods, dimensionless features, and nonlinear approximation powers. The majority of conventional DNNs are mainly concentrated on the low- or high-dimensional PDE with smooth solutions. In singular problems the DNN does not behave so efficiently as in the smooth problem because of low regularities of the solution. In this paper, we propose a novel adaptive DNN for the PDE with corner singular solutions. In the corner singular problem gradients of solution vary rapidly around some particular vertices or lines. This poses a difficulty in development of DNN solvers. We design adaptive DNN techniques for the corner singularity, which incorporates (a) an adaptive loss function that assigns radial weights to local losses, (b) an adaptive activation function that varies the choice of activation functions from neuron to neuron, and (c) an adaptive sampling strategy that selectively focuses on sampling points near the singular locations. The adaptive DNN with these adaptive schemes greatly enhances approximation accuracy and computation efficiency. This is verified by a comprehensive series of numerical experiments of 2D and 3D corner singular problems. The comparison analyzes with the existing DNN solvers are also made to demonstrate effectiveness of the new method. The L2 relative errors of the proposed adaptive DNN are reduced by an order of magnitude with respect to several conventional DNN methods.

Research on a Classification Method for Strip Steel Surface Defects Based on Knowledge Distillation and a Self-Adaptive Residual Shrinkage Network

Different types of surface defects will occur during the production of strip steel. To ensure production quality, it is essential to classify these defects. Our research indicates that two main problems exist in the existing strip steel surface defect classification methods: (1) they cannot solve the problem of unbalanced data using few-shot in reality, (2) they cannot meet the requirement of online real-time classification. To solve the aforementioned problems, a relational knowledge distillation self-adaptive residual shrinkage network (RKD-SARSN) is presented in this work. First, the data enhancement strategy of Cycle GAN defective sample migration is designed. Second, the self-adaptive residual shrinkage network (SARSN) is intended as the backbone network for feature extraction. An adaptive loss function based on accuracy and geometric mean (Gmean) is proposed to solve the problem of unbalanced samples. Finally, a relational knowledge distillation model (RKD) is proposed, and the functions of GUI operation interface encapsulation are designed by combining image processing technology. SARSN is used as a teacher model, its generalization performance is transferred to the lightweight network ResNet34, and it is conveniently deployed as a student model. The results show that the proposed method can improve the deployment efficiency of the model and ensure the real-time performance of the classification algorithms. It is superior to other mainstream algorithms for fine-grained images with unbalanced data classification.

Attention and feature transfer based knowledge distillation

Existing knowledge distillation (KD) methods are mainly based on features, logic, or attention, where features and logic represent the results of reasoning at different stages of a convolutional neural network, and attention maps symbolize the reasoning process. Because of the continuity of the two in time, transferring only one of them to the student network will lead to unsatisfactory results. We study the knowledge transfer between the teacher-student network to different degrees, revealing the importance of simultaneously transferring knowledge related to the reasoning process and reasoning results to the student network, providing a new perspective for the study of KD. On this basis, we proposed the knowledge distillation method based on attention and feature transfer (AFT-KD). First, we use transformation structures to transform intermediate features into attentional and feature block (AFB) that contain both inference process information and inference outcome information, and force students to learn the knowledge in AFBs. To save computation in the learning process, we use block operations to align the teacher-student network. In addition, in order to balance the attenuation ratio between different losses, we design an adaptive loss function based on the loss optimization rate. Experiments have shown that AFT-KD achieves state-of-the-art performance in multiple benchmark tests.

A Primary task driven adaptive loss function for multi-task speech emotion recognition

Although Speech Emotion Recognition (SER) is becoming an active research area, the state-of-the-art performance is limited by the scarcity of emotional datasets. The introduction of Multi-Task Learning (MTL) which jointly trains multiple relative tasks, can utilize more emotion-related information within limited data, enhancing the SER performance but also increasing the training difficulty. In this paper, we propose a primary-task driven adaptive loss function named as PTDA-Loss to adaptively adjust the task weights in the loss function, aiming to more reasonably allocate the learning resources in the training process and further improve the SER performance. We observe that excessive concern with the auxiliary task can lead to insufficient learning resources for SER, not conducive to SER performance improvement. Inspired by this finding, in PTDA-Loss, we retain the advantage of SER in the whole training process. Meanwhile, the learning resources for the auxiliary task will be appropriately but not over compressed, ensuring the effective learning of useful information from the auxiliary task. To improve the model’s learning ability, furthermore, we first employ several dynamic convolutional layers as the backbone network. Regarding gender recognition and speaker recognition as the auxiliary task respectively, ablation experiments on the IEMOCAP, MSP-IMPROV and MELD datasets prove the rationality and effectiveness of PTDA-Loss. Additionally, comparative experiments demonstrate that our method achieves state-of-the-art performance on the three datasets, further enhancing the SER accuracy.

Dual teacher model for semi-supervised ABUS tumor segmentation

Automated Breast Ultrasound (ABUS) has emerged as a crucial modality for the detection of breast tumors in clinical settings. However, obtaining accurate pixel-level annotations of medical images for ABUS tumor segmentation is a labor-intensive and expensive process, which hinders the performance of deep learning methods. In this work, we propose a novel semi-supervised tumor segmentation model, namely the Dual-Teacher model, which effectively leverages a small set of labeled images and a large number of unlabeled images for improved segmentation. The Dual-Teacher model extends the original mean teacher (MT) model, MT model only utilizes commonly used consistency criteria, by incorporating an auxiliary teacher branch the Dual-Teacher model captures complementary information from the teacher model and guides the training of the student model. We further introduce a pseudo-label update strategy that combines self-training and consistency constraints to improve the accuracy of pseudo-labels. To optimize the training of the Dual-Teacher model, we design an adaptive loss function that considers the pseudo-labels generated by both teacher models. Our proposed method is evaluated on an ABUS dataset comprising 107 patients with 170 volumes, totaling 3736 2D labeled slices. Experimental results demonstrate that the Dual-Teacher model achieves a Dice similarity coefficient (DSC) of 74.6 %, precision of 75.3 % and recall of 73.9 %, indicating promising performance compared to existing methods.

Learning Person-Specific Cognition From Facial Reactions for Automatic Personality Recognition

This paper proposes to recognise the true (self-reported) personality traits from the target subject's cognition simulated from facial reactions. This approach builds on the following two findings in cognitive science: (i) human cognition partially determines expressed behaviour and is directly linked to true personality traits; and (ii) in dyadic interactions, individuals' nonverbal behaviours are influenced by their conversational partner's behaviours. In this context, we hypothesise that during a dyadic interaction, a target subject's facial reactions are driven by two main factors: their internal (person-specific) cognitive process, and the externalised nonverbal behaviours of their conversational partner. Consequently, we propose to represent the target subject's (defined as the listener) person-specific cognition in the form of a person-specific CNN architecture that has unique architectural parameters and depth, which takes audio-visual non-verbal cues displayed by the conversational partner (defined as the speaker) as input, and is able to reproduce the target subject's facial reactions. Each person-specific CNN is explored by the Neural Architecture Search (NAS) and a novel adaptive loss function, which is then represented as a graph representation for recognising the target subject's true personality. Experimental results not only show that the produced graph representations are well associated with target subjects' personality traits in both human-human and human-machine interaction scenarios, and outperform the existing approaches with significant advantages, but also demonstrate that the proposed novel strategies help in learning more reliable personality representations.

μPEWFace: Parallel ensemble of weighted deep convolutional neural networks with novel loss functions for face-based authentication

Training Deep Convolutional Neural Networks (DCNN) with large-scale face images takes a significant amount of processing resources and the tuning/optimization time cost for face-based authentication. It must continue to improve its accuracy and speed during the matching phase. In this study, we propose a method, μPEWFace, that inherits not only the benefits of recent DCNNs with adaptive loss functions, such as MagFace, ElasticFace, and AdaFace, for boosting accuracy but also the means to reduce matching time. Consequently, our method expands on the weighted voting mechanism that leverages suboptimal trained models to improve the discriminative capabilities of face-based authentication, as opposed to searching for the best optimal model. In order to boost the efficiency of face-based authentication, we also propose performing the matching phase for each model in parallel. To demonstrate the speed and accuracy of our method, we conduct exhaustive experiments using a variety of well-known benchmarks, including LFW, CFP-FP, AgeDB-30, CALFW, CPLFW, and IJB-B. The experimental findings demonstrate that the proposed method for face-based authentication achieves state-of-the-art results and exhibits remarkable performance.© 2023 Published by Elsevier Ltd.

OWAdapt: An adaptive loss function for deep learning using OWA operators

In this paper, we propose a novel adaptive loss function for enhancing deep learning performance in classification tasks. Specifically, we redefine the cross-entropy loss to effectively address class-level noise conditions, including the challenging problem of class imbalance. Our approach introduces aggregation operators to improve classification accuracy. The rationale behind our proposed method lies in the iterative up-weighting of class-level components within the loss function, focusing on those with larger errors. To achieve this, we employ the ordered weighted average (OWA) operator and combine it with an adaptive scheme for gradient-based learning. The main finding is that our method outperforms other commonly used loss functions, such as the standard cross-entropy or focal loss, across various binary and multiclass classification tasks. Furthermore, we explore the influence of hyperparameters associated with the OWA operators and propose a default configuration that performs well across different experimental settings.

Hyperspectral and multispectral remote sensing image fusion using SwinGAN with joint adaptive spatial-spectral gradient loss function

ABSTRACT Hyperspectral remote sensing image (HSI) fusion with multispectral remote sensing images (MSI) improves data resolution. However, current fusion algorithms focus on local information and overlook long-range dependencies. The parameter of network tuning prioritizes global optimization, neglecting spatial and spectral constraints, and limiting spatial and spectral reconstruction capabilities. This study introduces SwinGAN, a fusion network combining Swin Transformer, CNN, and GAN architectures. SwinGAN's generator employs a detail injection framework to separately extract HSI and MSI features, fusing them to generate spatial residuals. These residuals are injected into the supersampled HSI to produce the final image, while a pure CNN architecture acts as the discriminator, enhancing the fusion quality. Additionally, we introduce a new adaptive loss function that improves image fusion accuracy. The loss function uses L1 loss as the content loss, and spatial and spectral gradient loss functions are introduced to improve the spatial representation and spectral fidelity of the fused images. Our experimental results on several datasets demonstrate that SwinGAN outperforms current popular algorithms in both spatial and spectral reconstruction capabilities. The ablation experiments also demonstrate the rationality of the various components of the proposed loss function.

Semi-FCMNet: Semi-Supervised Learning for Forest Cover Mapping from Satellite Imagery via Ensemble Self-Training and Perturbation

Forest cover mapping is of paramount importance for environmental monitoring, biodiversity assessment, and forest resource management. In the realm of forest cover mapping, significant advancements have been made by leveraging fully supervised semantic segmentation models. However, the process of acquiring a substantial quantity of pixel-level labelled data is prone to time-consuming and labour-intensive procedures. To address this issue, this paper proposes a novel semi-supervised-learning-based semantic segmentation framework that leverages limited labelled and numerous unlabelled data, integrating multi-level perturbations and model ensembles. Our framework incorporates a multi-level perturbation module that integrates input-level, feature-level, and model-level perturbations. This module aids in effectively emphasising salient features from remote sensing (RS) images during different training stages and facilitates the stability of model learning, thereby effectively preventing overfitting. We also propose an ensemble-voting-based label generation strategy that enhances the reliability of model-generated labels, achieving smooth label predictions for challenging boundary regions. Additionally, we designed an adaptive loss function that dynamically adjusts the focus on poorly learned categories and dynamically adapts the attention towards labels generated during both the student and teacher stages. The proposed framework was comprehensively evaluated using two satellite RS datasets, showcasing its competitive performance in semi-supervised forest-cover-mapping scenarios. Notably, the method outperforms the fully supervised approach by 1–3% across diverse partitions, as quantified by metrics including mIoU, accuracy, and mPrecision. Furthermore, it exhibits superiority over other state-of-the-art semi-supervised methods. These results indicate the practical significance of our solution in various domains, including environmental monitoring, forest management, and conservation decision-making processes.

Improved Detector Based on Yolov5 for Typical Targets on the Sea Surfaces

Detection of targets on sea surfaces is an important area of application that can bring great benefits to the management and control systems in marine environments. However, there are few open-source datasets accessible for the purpose of object detection on seas and rivers. In this paper, a study is conducted on the improved detection algorithms based on the YOLOv5 model. The dataset for the tests contains ten categories of typical objects that are commonly seen in the contexts of seas, including ships, devices, and structures. Multiple augmentation methods are employed in the pre-processing of the input data, which are verified to be effective in enhancing the generalization ability of the algorithm. Moreover, a new form of the loss function is proposed that highlights the effects of the high-quality boxes during training. The results demonstrate that the adapted loss function contributes to a boost in the model performance. According to the ablation studies, the synthesized methods raise the inference accuracy by making up for several shortcomings of the baseline model for the detection tasks of single or multiple targets from varying backgrounds.

Automatic classification of company’s document stream: Comparison of two solutions

Documents are essential nowadays and present everywhere. In order to manage the vast amount of documents managed by companies, a first step consists in automatically determining the type of the document (its class). Even if automatic classification has been widely studied in the state of the art, the strongly imbalanced context and industrial constraints bring new challenges which were not studied till now: how to classify as many documents as possible with the highest precision, in an imbalanced context and with some classes missing during training?To this end, this paper proposes to study two different solutions to address these issues. The first is a multimodal neural network reinforced by an attention model and an adapted loss function that is able to classify a great variety of documents. The second is a combination method that uses a cascade of systems to offer a gradual solution for each issue. These two options provide good results as well in ideal context than in imbalanced context. This comparison outlines the limitations and the future challenges.

Diverse local facial behaviors learning from enhanced expression flow for microexpression recognition

Facial microexpression recognition (FMER) has recently gained increasing attention in the interrogation and clinical diagnosis fields. However, accurate FMER suffers from not only the low-intensity muscle movements and a short duration of microexpressions but also limited training data. To address the abovementioned issues, this paper employs facial u/v/s images to explore short durationlow-intensity muscle movements, which can be extracted from the expression flow. Specifically, we propose a motion detail enhancement method to disentangle microexpression-irrelevant motions from the expression flow. Then, a novel data augmentation strategy based on different u-v weights is introduced to generate diverse s components, resolving the limited scale issue. Afterward, the enhanced facial u/v/s images are fed to a novel local-diverse facial microexpression recognition network (LD-FMERN) for microexpression-related feature extraction, where a spatial-channel modulator is used to refine the extracted features. A locally diverse feature mining strategy further enhances the refined features, forcing the network to focus on small and diverse facial regions. Following the law of human vision, we propose an adaptive loss function, which comprises a supervised cross-entropy loss and a self-supervised local-diverse loss, to optimize the network. Extensive quantitative and qualitative evaluations on benchmark datasets, CASMEII, SAMM, and MMEW, demonstrate the improvements. Comparisons with state-of-the-art FMER methods reveal the superiority of the proposed method.

An adapted loss function for composite quantile regression with censored data

An adapted loss function for composite quantile regression with censored data

An Adaptive Hybrid Model for Wind Power Prediction Based on the IVMD-FE-Ad-Informer.

Accurate wind power prediction can increase the utilization rate of wind power generation and maintain the stability of the power system. At present, a large number of wind power prediction studies are based on the mean square error (MSE) loss function, which generates many errors when predicting original data with random fluctuation and non-stationarity. Therefore, a hybrid model for wind power prediction named IVMD-FE-Ad-Informer, which is based on Informer with an adaptive loss function and combines improved variational mode decomposition (IVMD) and fuzzy entropy (FE), is proposed. Firstly, the original data are decomposed into K subsequences by IVMD, which possess distinct frequency domain characteristics. Secondly, the sub-series are reconstructed into new elements using FE. Then, the adaptive and robust Ad-Informer model predicts new elements and the predicted values of each element are superimposed to obtain the final results of wind power. Finally, the model is analyzed and evaluated on two real datasets collected from wind farms in China and Spain. The results demonstrate that the proposed model is superior to other models in the performance and accuracy on different datasets, and this model can effectively meet the demand for actual wind power prediction.

A cloud-oriented siamese network object tracking algorithm with attention network and adaptive loss function

Aiming at solving the problems of low success rate and weak robustness of object tracking algorithms based on siamese network in complex scenes with occlusion, deformation, and rotation, a siamese network object tracking algorithm with attention network and adaptive loss function (SiamANAL) is proposed. Firstly, the multi-layer feature fusion module for template branch (MFFMT) and the multi-layer feature fusion module for search branch (MFFMS) are designed. The modified convolutional neural networks (CNN) are used for feature extraction through the fusion module to solve the problem of features loss caused by too deep network. Secondly, an attention network is introduced into the SiamANAL algorithm to calculate the attention of template map features and search map features, which enhances the features of object region, reduces the interference of background region, and improves the accuracy of the algorithm. Finally, an adaptive loss function combined with pairwise Gaussian loss function and cross entropy loss function is designed to increase inter-class separation and intra-class compactness of classification branches and improve the accuracy rate of classification and the success rate of regression. The effectiveness of the proposed algorithm is verified by comparing it with other popular algorithms on two popular benchmarks, the visual object tracking 2018 (VOT2018) and the object tracking benchmark 100 (OTB100). Extensive experiments demonstrate that the proposed tracker achieves competitive performance against state-of-the-art trackers. The success rate and precision rate of the proposed algorithm SiamANAL on OTB100 are 0.709 and 0.883, respectively. With the help of cloud computing services and data storage, the processing performance of the proposed algorithm can be further improved.