Joint Loss Function Research Articles

Remote Sensing Image Fusion Method Based on Progressive Cascaded Deep Residual Network

With the rapid development of deep learning in recent years, it has shown excellent performance in various image and video processing tasks. In addition, it also has a great role in promoting the spatio-temporal fusion of remote sensing images. The reconstructed image can give people a good visual experience. The invention relates to a remote sensing image fusion method based on a progressive cascade deep residual network and provides an end-to-end progressive cascade deep residual network model for remote sensing image fusion. The use of the MSE loss function may cause oversmoothing of the fused image, so a new joint loss function is defined to capture finer spatial information to improve the spatial resolution of the fused image. Resize-convolution is used to replace the transposed convolution to eliminate the checkerboard effect in the fused image caused by the transposed convolution. Through the experiments on the remote sensing image fusion simulation and real datasets of multiple satellites, the data results of the proposed algorithm are more than 5.25% better than those of the comparative algorithm in the average quantification. The calculation time and system resource occupation are also reduced, which has important theoretical significance and application value in the field of artificial intelligence and image processing. It will play a certain role in promoting the theoretical research and application of remote sensing image fusion.

Camouflaged Object Detection Based on Ternary Cascade Perception

Camouflaged object detection (COD), in a broad sense, aims to detect image objects that have high degrees of similarity to the background. COD is more challenging than conventional object detection because of the high degree of “fusion” between a camouflaged object and the background. In this paper, we focused on the accurate detection of camouflaged objects, conducting an in-depth study on COD and addressing the common detection problems of high miss rates and low confidence levels. We proposed a ternary cascade perception-based method for detecting camouflaged objects and constructed a cascade perception network (CPNet). The innovation lies in the proposed ternary cascade perception module (TCPM), which focuses on extracting the relationship information between features and the spatial information of the camouflaged target and the location information of key points. In addition, a cascade aggregation pyramid (CAP) and a joint loss function have been proposed to recognize camouflaged objects accurately. We conducted comprehensive experiments on the COD10K dataset and compared our proposed approach with other seventeen-object detection models. The experimental results showed that CPNet achieves optimal results in terms of six evaluation metrics, including an average precision (AP)50 that reaches 91.41, an AP75 that improves to 73.04, and significantly higher detection accuracy and confidence.

Multi-task melody extraction using feature optimization and CRNN-CRF

At present, mainstream melody extraction mostly uses deep learning methods, but there are still problems: such as incomplete network architecture, lack of research on the importance of input features for melody extraction, etc. Based on the previous issues, to predict the melody more accurately, we firstly use phase correction after short-time Fourier transform (STFT) feature extraction method according to audio features. Secondly, we use a weight-based and multi-task network feature optimization mechanism to assign weights to the input features of the main and auxiliary networks, to fully capture the important parts in the time sequence of the melodic. Thirdly, to make use of the context information of features and label sequences, we further propose a multi-task pitch extraction network based on Convolutional Recurrent Neural Network-Conditional Random Field (CRNN-CRF) to decode the optimal label sequences. Besides, we use the joint loss function of the previous two tasks to train and optimize the model to improve the accuracy of pitch prediction. After testing and evaluating the various datasets before visual comparison, the experimental results have demonstrated the superiority of our proposed method.

An Unmixing-Based Multi-Attention GAN for Unsupervised Hyperspectral and Multispectral Image Fusion

Hyperspectral images (HSI) frequently have inadequate spatial resolution, which hinders numerous applications for the images. High resolution multispectral image (MSI) has been fused with HSI to reconstruct images with both high spatial and high spectral resolutions. In this paper, we propose a generative adversarial network (GAN)-based unsupervised HSI-MSI fusion network. In the generator, two coupled autoencoder nets decompose HSI and MSI into endmembers and abundances for fusing high resolution HSI through the linear mixing model. The two autoencoder nets are connected by a degradation-generation (DG) block, which further improves the accuracy of the reconstruction. Additionally, a coordinate multi-attention net (CMAN) is designed to extract more detailed features from the input. Driven by the joint loss function, the proposed method is straightforward and easy to execute in an end-to-end training manner. The experimental results demonstrate that the proposed strategy outperforms the state-of-art methods.

Dual Attention and Question Categorization-Based Visual Question Answering

Visual question answering (VQA) aims at predicting an answer to a natural language question associated with an image. This work focuses on two important issues pertaining to VQA, which is a complex multimodal AI task: First, the task of answer prediction in a large output answer space, and second, to obtain enriched representation through cross-modality interactions. This work aims to address these two issues by proposing a dual attention (DA) and question categorization (QC)-based visual question answering model (DAQC-VQA). DAQC-VQA has three main network modules: First, a novel dual attention mechanism that helps toward the objective of obtaining an enriched cross-domain representation of the two modalities; second, a question classifier subsystem for identifying input (natural language) question category. The second module of question categorizer helps in reducing the answer search space; and third, a subsystem for predicting answer depending on the question category. All component networks of DAQC-VQA are trained in an end-to-end manner with a joint loss function. The performance of DAQC-VQA is evaluated on two widely used VQA datasets, viz., TDIUC and VQA2.0. Experimental results demonstrate competitive performance of DAQC-VQA against the recent state-of-the-art VQA models. An ablation analysis indicates that the enriched representation obtained using the proposed dual-attention mechanism helps improve performance.

A low-light image enhancement method based on AUDD-GAN

A large number of images taken under conditions of insufficient lighting, often with overall darkness, low contrast, noise and inconspicuous details, thus affecting people’s observation of the image content and the subsequent use of the image. This paper presents a novel method, Attention U-Net Dual Discriminator-Generative Adversarial Networks (AUDD-GAN), which combines attention mechanism, U-Net network structure and the dual discriminator. First, the method generates a normalized attention map to guide the subsequent modules to perform illumination enhancement to prevent overexposure or underexposure problems; second, the U-Net generative network extracts characteristics of diverse levels to acquire richer detail information; then, a dual discriminator is used as the discriminative network, where PatchGAN is used as the local discriminator to achieve better enhancement effects; ultimately, we combine multiple loss functions as the joint loss function. AUDD-GAN is verified to enhance the global contrast of the image while enhancing the detail features more effectively.

LightMobileBert: A secondary lightweight model based on MobileBert

MobileBert is a generic lightweight model suffering from a large network depth and parameter cardinality. Therefore, this paper proposes a secondary lightweight model entitled LightMobileBert, which retains the bottom 12 Transformers structure of the pre-trained MobileBert and utilizes the tensor decomposition technique to process the model to skip pre-training and further reduce the parameters. At the same time, the joint loss function is constructed based on the improved Supervised Contrastive Learning loss function and the Cross-Entropy loss function to improve performance and stability. Finally, the LMBert_Adam optimizer, an improved Bert_Adam optimizer, is used to optimize the model. The experimental results demonstrate that LightMobileBert has a comparatively higher performance than MobileBert and other popular models while requiring 57% fewer network parameters than MobileBert, confirming that LightMobileBert retains a higher performance while being lightweight.

DeepGpgs: a novel deep learning framework for predicting arginine methylation sites combined with Gaussian prior and gated self-attention mechanism.

Protein arginine methylation is an important posttranslational modification (PTM) associated with protein functional diversity and pathological conditions including cancer. Identification of methylation binding sites facilitates a better understanding of the molecular function of proteins. Recent developments in the field of deep neural networks have led to a proliferation of deep learning-based methylation identification studies because of their fast and accurate prediction. In this paper, we propose DeepGpgs, an advanced deep learning model incorporating Gaussian prior and gated attention mechanism. We introduce a residual network channel to extract the evolutionary information of proteins. Then we combine the adaptive embedding with bidirectional long short-term memory networks to form a context-shared encoder layer. A gated multi-head attention mechanism is followed to obtain the global information about the sequence. A Gaussian prior is injected into the sequence to assist in predicting PTMs. We also propose a weighted joint loss function to alleviate the false negative problem. We empirically show that DeepGpgs improves Matthews correlation coefficient by 6.3% on the arginine methylation independent test set compared with the existing state-of-the-art methylation site prediction methods. Furthermore, DeepGpgs has good robustness in phosphorylation site prediction of SARS-CoV-2, which indicates that DeepGpgs has good transferability and the potential to be extended to other modification sites prediction. The open-source code and data of the DeepGpgs can be obtained from https://github.com/saizhou1/DeepGpgs.

Vehicle Tracking Algorithm Based on Deep Learning in Roadside Perspective

Traffic intelligence has become an important part of the development of various countries and the automobile industry. Roadside perception is an important part of the intelligent transportation system, which mainly realizes the effective perception of road environment information by using sensors installed on the roadside. Vehicles are the main road targets in most traffic scenes, so tracking a large number of vehicles is an important subject in the field of roadside perception. Considering the characteristics of vehicle-like rigid targets from the roadside view, a vehicle tracking algorithm based on deep learning was proposed. Firstly, we optimized a DLA-34 network and designed a block-N module, then the channel attention and spatial attention modules were added in the front of the network to improve the overall feature extraction ability and computing efficiency of the network. Next, the joint loss function was designed to improve the intra-class and inter-class discrimination ability of the tracking algorithm, which can better discriminate objects of similar appearance and the color of vehicles, alleviate the IDs problem and improve algorithm robustness and the real-time performance of the tracking algorithm. Finally, the experimental results showed that the method had a good tracking effect for the vehicle tracking task from the roadside perspective and could meet the practical application demands of complex traffic scenes.

Deep Learning from Noisy Labels with Some Adjustments of a Recent Method

In this paper we have used JoCoR, a fairly recent method for learning with label noise, that makes use of two neural networks with a joint loss function using an additional contrastive loss to increase the agreement between them. This method can be extended to more than two networks in a straightforward way. We have carried out experiments on the CIFAR-10 and CIFAR-100 datasets (contaminated by synthetic label noise) with this kind of extension using several contrastive losses. We have concluded that it makes a significant improvement if we use a third network, especially when we use Kullback-Leibler terms for all possible pairs of softmax outputs. Further extension also means some kind of improvement, but in the case of the CIFAR datasets, those were not so significant, maybe except the cases with lower ratio of label noise.

Towards Unified All-Neural Beamforming for Time and Frequency Domain Speech Separation

Recently, frequency domain all-neural beamforming methods have achieved remarkable progress for multichannel speech separation. In parallel, the integration of time domain network structure and beamforming also gains significant attention. This study proposes a novel all-neural beamforming method in time domain and makes an attempt to unify the all-neural beamforming pipelines for time domain and frequency domain multichannel speech separation. The proposed model consists of two modules: separation and beamforming. Both modules perform temporal-spectral-spatial modeling and are trained from end-to-end using a joint loss function. The novelty of this study lies in two folds. Firstly, a time domain directional feature conditioned on the direction of the target speaker is proposed, which can be jointly optimized within the time domain architecture to enhance target signal estimation. Secondly, an all-neural beamforming network in time domain is designed to refine the pre-separated results. This module features with parametric time-variant beamforming coefficient estimation, without explicitly following the derivation of optimal filters that may lead to an upper bound. The proposed method is evaluated on simulated reverberant overlapped speech data derived from the AISHELL-1 corpus. Experimental results demonstrate significant performance improvements over frequency domain state-of-the-arts, ideal magnitude masks and existing time domain neural beamforming methods.

Vision Transformer With Contrastive Learning for Remote Sensing Image Scene Classification

Remote sensing images (RSIs) are characterized by complex spatial layouts and ground object structures. ViT can be a good choice for scene classification owing to the ability to capture long-range interactive information between patches of input images. However, due to the lack of some inductive biases inherent to CNNs, such as locality and translation equivariance, ViT cannot generalize well when trained on insufficient amounts of data. Compared with training ViT from scratch, transferring a large-scale pretrained one is more cost-efficient with better performance even when the target data are small scale. In addition, the cross-entropy (CE) loss is frequently utilized in scene classification yet has low robustness to noise labels and poor generalization performances for different scenes. In this article, a ViT-based model in combination with supervised contrastive learning (CL) is proposed, named ViT-CL. For CL, supervised contrastive (SupCon) loss, which is developed by extending the self-supervised contrastive approach to the fully supervised setting, can explore the label information of RSIs in embedding space and improve the robustness to common image corruption. In ViT-CL, a joint loss function that combines CE loss and SupCon loss is developed to prompt the model to learn more discriminative features. Also, a two-stage optimization framework is introduced to enhance the controllability of the optimization process of the ViT-CL model. Extensive experiments on the AID, NWPU-RESISC45, and UCM datasets verified the superior performance of ViT-CL, with the highest accuracies of 97.42%, 94.54%, and 99.76% among all competing methods, respectively.

Multi-Branch and Progressive Network for Low-Light Image Enhancement

Low-light images incur several complicated degradation factors such as poor brightness, low contrast, color degradation, and noise. Most previous deep learning-based approaches, however, only learn the mapping relationship of single channel between the input low-light images and the expected normal-light images, which is insufficient enough to deal with low-light images captured under uncertain imaging environment. Moreover, too deeper network architecture is not conducive to recover low-light images due to extremely low values in pixels. To surmount aforementioned issues, in this paper we propose a novel multi-branch and progressive network (MBPNet) for low-light image enhancement. To be more specific, the proposed MBPNet is comprised of four different branches which build the mapping relationship at different scales. The followed fusion is performed on the outputs obtained from four different branches for the final enhanced image. Furthermore, to better handle the difficulty of delivering structural information of low-light images with low values in pixels, a progressive enhancement strategy is applied in the proposed method, where four convolutional long short-term memory networks (LSTM) are embedded in four branches and an recurrent network architecture is developed to iteratively perform the enhancement process. In addition, a joint loss function consisting of the pixel loss, the multi-scale perceptual loss, the adversarial loss, the gradient loss, and the color loss is framed to optimize the model parameters. To evaluate the effectiveness of proposed MBPNet, three popularly used benchmark databases are used for both quantitative and qualitative assessments. The experimental results confirm that the proposed MBPNet obviously outperforms other state-of-the-art approaches in terms of quantitative and qualitative results. The code will be available at https://github.com/kbzhang0505/MBPNet.

Automatic Detection of CFRP Subsurface Defects via Thermal Signals in Long Pulse and Lock-In Thermography

Thermography is widely used to detect delamination defects in carbon fiber reinforced plastics (CFRP). This paper proposes a model to detect defects automatically by extracting the thermal signal characteristics of CFRP materials. An optically excited thermography system is constructed for pulsed and lock-in thermography experiments to compare thermal signal data sets in different excitation modes. A multi-task joint loss function is defined to train the model for defect detection and depth prediction. The effects of different attention modules (AM) are analyzed to improve the model performance. By comparing the effects of traditional thermography processing methods and methods based on Convolutional Neural Network (CNN), it is found that the proposed model can detect defects with minimum aspect ratio (ratio of short side to depth) of 2.5, and the relative error percentage in depth prediction is below 10%.

Finger vein recognition method based on ant colony optimization and improved EfficientNetV2.

Deep learning is an important technology in the field of image recognition. Finger vein recognition based on deep learning is one of the research hotspots in the field of image recognition and has attracted a lot of attention. Among them, CNN is the most core part, which can be trained to get a model that can extract finger vein image features. In the existing research, some studies have used methods such as combination of multiple CNN models and joint loss function to improve the accuracy and robustness of finger vein recognition. However, in practical applications, finger vein recognition still faces some challenges, such as how to solve the interference and noise in finger vein images, how to improve the robustness of the model, and how to solve the cross-domain problem. In this paper, we propose a finger vein recognition method based on ant colony optimization and improved EfficientNetV2, using ACO to participate in ROI extraction, fusing dual attention fusion network (DANet) with EfficientNetV2, and conducting experiments on two publicly available databases, and the results show that the recognition rate using the proposed method on the FV-USM dataset reaches The results show that the proposed method achieves a recognition rate of 98.96% on the FV-USM dataset, which is better than other algorithmic models, proving that the method has good recognition rate and application prospects for finger vein recognition.

An Adaptive Domain Adaptation Method for Rolling Bearings’ Fault Diagnosis Fusing Deep Convolution and Self-Attention Networks

Intelligent fault diagnosis methods based on deep learning have attracted significant attention in recent years. However, it still faces many challenges, including complex and variable working conditions, noise interference, and insufficient valid data samples. Therefore, a novel deep transfer learning bearing fault diagnosis model is designed in this work by fusing time-frequency analysis, residual network (ResNet) and self-attention mechanism (SAM). A multiscale time-frequency feature map (MTFFM) and global statistical feature matrix (GSFM) of vibration signals are first constructed using wavelet packet transform (WPT). A deep feature extraction network combining ResNet and SAM networks is then designed to realize the fused extraction of local and global time-frequency features. Finally, we construct a joint loss function by the combination of multi-kernel maximum mean discrepancy (MK-MMD) and the domain adversarial neural network (DANN) to optimize the depth feature extraction network, which improves the cross-domain invariance and fault state discrimination of depth features. The proposed optimization method fully exploits the advantages of high-dimensional spatial distribution difference evaluation and gradient inversion adversarial strategy. Its effectiveness is demonstrated through variable working condition transfer fault diagnosis tasks, showing superior performance compared with other intelligent fault diagnosis methods.

Learning Domain and Pose Invariance for Thermal-to-Visible Face Recognition

Interest in thermal to visible face recognition has grown significantly over the last decade due to advancements in thermal infrared cameras and analytics beyond the visible spectrum. Despite large discrepancies between thermal and visible spectra, existing approaches bridge domain gaps by either synthesizing visible faces from thermal faces or by learning the cross-spectrum image representations. These approaches typically work well with frontal facial imagery collected at varying ranges and expressions, but exhibit significantly reduced performance when matching thermal faces with varying poses to frontal visible faces. We propose a novel Domain and Pose Invariant Framework that simultaneously learns domain and pose invariant representations. Our proposed framework is composed of modified networks for extracting the most correlated intermediate representations from off-pose thermal and frontal visible face imagery, a sub-network to jointly bridge domain and pose gaps, and a joint-loss function comprised of cross-spectrum and pose-correction losses. We demonstrate efficacy and advantages of the proposed method by evaluating on three thermal-visible datasets: ARL Visible-to-Thermal Face, ARL Multimodal Face, and Tufts Face. Although DPIF focuses on learning to match off-pose thermal to frontal visible faces, we also show that DPIF enhances performance when matching frontal thermal face images to frontal visible face images.

Two-stage deep neural network with joint loss and multi-level representations for defect detection

In recent years, segmentation-based two-stage neural networks have emerged in the defect detection domain. Although these neural networks have shown great success in surface defect detection, they suffer from blurred edges, noise, and imbalanced data, which degrade the detection performance. In this study, we address these limitations by presenting a new two-stage defect detection framework. To solve the problem of defect detection with blurred edges, we propose leveraging multilevel representations during segmentation. Furthermore, we employ a joint loss function that combines the binary cross-entropy loss and Dice loss to reduce the influence of noise. Meanwhile, the joint loss function that contributes to our model is more robust for imbalanced data and is better for segmenting small-sized defects. To verify the performance of our proposed model, comprehensive experiments were conducted on two surface defect datasets: the Kolektor surface-defect dataset (KSDD) (http://www.vicos.si/Downloads/KolektorSDD) and the Deutsche Arbeitsgemeinschaft für Mustererkennung dataset (DAGM) (https://hci.iwr.uni-heidelberg.de/content/weakly-supervised-learning-industrial-optical-inspection). By performing more accurate segmentation of defects, our model shows superior results over the compared networks; it provided a 100% average precision on KSDD and 100% mean accuracy on DAGM, which demonstrates the effectiveness of our model.

Coupled adversarial learning for fusion classification of hyperspectral and LiDAR data

Hyperspectral image (HSI) provides rich spectral–spatial information and the light detection and ranging (LiDAR) data reflect the elevation information, which can be jointly exploited for better land-cover classification. However, due to different imaging mechanisms, HSI and LiDAR data always present significant image difference, current pixel-wise feature fusion classification methods relying on concatenation or weighted fusion are not effective. To achieve accurate classification result, it is important to extract and fuse similar high-order semantic information and complementary discriminative information contained in multimodal data. In this paper, we propose a novel coupled adversarial learning based classification (CALC) method for fusion classification of HSI and LiDAR data. In specific, a coupled adversarial feature learning (CAFL) sub-network is first trained, to effectively learn the high-order semantic features from HSI and LiDAR data in an unsupervised manner. On one hand, the proposed CAFL sub-network establishes an adversarial game between dual generators and discriminators, so that the learnt features can preserve detail information in HSI and LiDAR data, respectively. On the other hand, by designing weight-sharing and linear fusion structure in the dual generators, we can simultaneously extract similar high-order semantic information and modal-specific complementary information. Meanwhile, a supervised multi-level feature fusion classification (MFFC) sub-network is trained, to further improve the classification performance via adaptive probability fusion strategy. In brief, the low-level, mid-level and high-level features learnt by the CAFL sub-network lead to multiple class estimation probabilities, which are then adaptively combined to generate a final accurate classification result. Both the CAFL and MFFC sub-networks are collaboratively trained by optimizing a designed joint loss function, which consists of unsupervised adversarial loss and supervised classification loss. Overall, by optimizing the joint loss function, the proposed CALC network is pushed to learn highly discriminative fusion features from multimodal data, leading to higher classification accuracies. Extensive experiments on three well-known HSI and LiDAR data sets demonstrate the superior classification performance by the proposed CALC method than several state-of-the-art methods. The source code of the proposed method will be made publicly available at https://github.com/Ding-Kexin/CALC.

Coarse-to-fine feature representation based on deformable partition attention for melanoma identification

In the histopathological melanoma image diagnosis system, manual identification of super-scale slides with dense cells is tedious, time-consuming, and subjective. To deal with this problem, we propose an automatic identification network based on the deformable partition attention to identify lots of dense slides as an assistant. A coarse-to-fine strategy is adopted in feature representation and qualitative identification to improve the identification accuracy of melanomas and nevi. First of all, because it is difficult to extract features in the lesion area with blurred boundaries and uneven distribution, we develop a deformable partition attention module, which integrates the advantage of the attention mechanism and deformable convolution. The module overcomes the limitation of rectangular convolution and gradually refines the channel and spatial features, which enriches feature representation by combining global and local features. Secondly, to address the problem of difficult convergence and poor recognition rate caused by the excessive non-aligned distance between benign-malignant and benign subcategories, we propose a progressive architecture via a coarse sub-network closely followed by a fine sub-network. Moreover, to further increase the inter-class differences and reduce the intra-class disparities, we propose a joint loss function to mine hard samples, which effectively improves the identification performance. Experimental results on the clinical dataset show that the proposed algorithm has higher sensitivity and specificity and outperforms state-of-the-art deep neural networks.