Joint Loss Research Articles

Coupled adversarial learning for fusion classification of hyperspectral and LiDAR data

Hyperspectral image (HSI) provides rich spectral–spatial information and the light detection and ranging (LiDAR) data reflect the elevation information, which can be jointly exploited for better land-cover classification. However, due to different imaging mechanisms, HSI and LiDAR data always present significant image difference, current pixel-wise feature fusion classification methods relying on concatenation or weighted fusion are not effective. To achieve accurate classification result, it is important to extract and fuse similar high-order semantic information and complementary discriminative information contained in multimodal data. In this paper, we propose a novel coupled adversarial learning based classification (CALC) method for fusion classification of HSI and LiDAR data. In specific, a coupled adversarial feature learning (CAFL) sub-network is first trained, to effectively learn the high-order semantic features from HSI and LiDAR data in an unsupervised manner. On one hand, the proposed CAFL sub-network establishes an adversarial game between dual generators and discriminators, so that the learnt features can preserve detail information in HSI and LiDAR data, respectively. On the other hand, by designing weight-sharing and linear fusion structure in the dual generators, we can simultaneously extract similar high-order semantic information and modal-specific complementary information. Meanwhile, a supervised multi-level feature fusion classification (MFFC) sub-network is trained, to further improve the classification performance via adaptive probability fusion strategy. In brief, the low-level, mid-level and high-level features learnt by the CAFL sub-network lead to multiple class estimation probabilities, which are then adaptively combined to generate a final accurate classification result. Both the CAFL and MFFC sub-networks are collaboratively trained by optimizing a designed joint loss function, which consists of unsupervised adversarial loss and supervised classification loss. Overall, by optimizing the joint loss function, the proposed CALC network is pushed to learn highly discriminative fusion features from multimodal data, leading to higher classification accuracies. Extensive experiments on three well-known HSI and LiDAR data sets demonstrate the superior classification performance by the proposed CALC method than several state-of-the-art methods. The source code of the proposed method will be made publicly available at https://github.com/Ding-Kexin/CALC.

EXTRAARTICULAR MANIFESTATIONS OF RHEUMATOID ARTHRITIS

Rheumatoid arthritis (RA) is a systemic autoimmune disease. The main characteristic of RA is persistent joint inammation resulting in joint damage and loss of function. Rheumatoid arthritis may lead to a variety of extra articular manifestations. Extra-articular organ involvement in RA is more frequently seen in patients with severe, active disease and is associated with increased mortality. The occurrence of these systemic manifestations is a major predictor of mortality in patients with RA. [1] Our case series attempts to provide a glimpse into the various extra articular manifestations of Rheumatoid arthritis. In total, 3 patients have been included for this case series. All 3 patients were a known case of Rheumatoid arthritis who presented with extra articular manifestations at MGM hospital, Kamothe.

Coarse-to-fine feature representation based on deformable partition attention for melanoma identification

In the histopathological melanoma image diagnosis system, manual identification of super-scale slides with dense cells is tedious, time-consuming, and subjective. To deal with this problem, we propose an automatic identification network based on the deformable partition attention to identify lots of dense slides as an assistant. A coarse-to-fine strategy is adopted in feature representation and qualitative identification to improve the identification accuracy of melanomas and nevi. First of all, because it is difficult to extract features in the lesion area with blurred boundaries and uneven distribution, we develop a deformable partition attention module, which integrates the advantage of the attention mechanism and deformable convolution. The module overcomes the limitation of rectangular convolution and gradually refines the channel and spatial features, which enriches feature representation by combining global and local features. Secondly, to address the problem of difficult convergence and poor recognition rate caused by the excessive non-aligned distance between benign-malignant and benign subcategories, we propose a progressive architecture via a coarse sub-network closely followed by a fine sub-network. Moreover, to further increase the inter-class differences and reduce the intra-class disparities, we propose a joint loss function to mine hard samples, which effectively improves the identification performance. Experimental results on the clinical dataset show that the proposed algorithm has higher sensitivity and specificity and outperforms state-of-the-art deep neural networks.

Correlation of clinical joint function with histopathological features of knee joint fibrosis

Objective: To investigate the correlation between the joint function and the histologic grading after total knee arthroplasty, to aid in the early diagnosis and prognostication of arthrofibrosis. Methods: A total of 29 patients including 22 females and 7 males were enrolled retrospectively from October 2015 to October 2020. These patients had a mean age of 63 years (range 41 to 79 years) and underwent total knee revision in Jishuitan Hospital due to joint contraction or loss of range of motion. Histologic assessment was carried out by utilizing immunohistochemistry (IHC) and the Masson staining to evaluate the fibrosis and inflammation of the samples. Results: By light microscopy, early stage arthrofibrosis showed massive proliferation of myofibroblasts and fibroblasts with SMA expression by IHC. In late stage arthrofibrosis, hyaline degeneration occured with extensive hyperplasia of fibrosis-related tissue. The arthrofibrosis samples appeared blue with Masson staining. Lymphocytes showed perivascular distribution. The arthrofibrosis tissue was mostly grade 3 (26 samples) in histologic assessment, moderate grade (25 samples) in ALVAL score, and grade 1 (23 samples) in lymphocyte grading. Fibrosis grading showed an overwhelming correlation with range of motion (ROM) of the joint. The ALVAL score was highly correlated with the WOMAC score. There was also a direct correlation between inflammatory cell infiltration and pain. The fibrosis grading joint with ALVAL score showed a good predictive value of joint function after joint replacement surgery. Conclusions: The histologic assessment score is closely correlated to the joint function with predictive values for the prognosis after joint replacement surgery.

Densely connected convolutional transformer for single image dehazing

Image Dehazing is an important low-level vision task that aims to remove the haze from an image. In this paper, we proposed Densely Connected Convolutional Transformer (DCCT) for single image dehazing. DCCT is an efficient architecture that combines the multi-head Performer with the local dependencies. To prevent loss of information between features at different levels, we propose a learnable connection layer that is used to fuse features at different levels across the entire architecture. We guide the training of DCCT through a joint loss considering a supervised metric learning approach that allows us to consider both negative and positive features for a multi-image perceptual loss. We validate the design choices and the effectiveness of the proposed DCCT through ablation studies. Through comparison with the representative techniques, we establish that the proposed DCCT is highly competitive with the state of the art.

Multi-focus image fusion: Transformer and shallow feature attention matters

The depth of field (DOF) of camera equipment is generally limited, so it is very difficult to get a fully focused image with all the objects clear after taking only one shot. A way to obtain a fully focused image is to use a multi-focus image fusion method, which fuses multiple images with different focusing depths into one image. However, most of the existing methods focus too much on the fusion accuracy of a single pixel, ignoring the integrity of the target and the importance of shallow features, resulting in internal errors and boundary artifacts, which need to be repaired after a long time of post-processing. In order to solve these problems, we propose a cascade network based on Transformer and attention mechanism, which can directly obtain the decision map and fusion result of focusing/defocusing region through end-to-end processing of source image, avoiding complex post-processing. For improving the fusion accuracy, this paper introduces the joint loss function, which can optimize the network parameters from three aspects. Furthermore, In order to enrich the shallow features of the network, a global attention module with shallow features is designed. Extensive experiments were conducted, including a large number of ablation experiments, 6 objective measures and a variety of subjective visual comparisons. Compared with 9 state-of-the-art methods, the results show that the proposed network structure can improve the quality of multi-focus fusion images and the performance is optimal.

Dense Residual Fusion and Spatial Local Filtering Low Light Dehazing Algorithm Based on Attention Mechanism

Image dehazing under low-light condition is easy to produce color distortion, patches and artifacts. To adapt to low-light condition, a dense residual fusion and spatial local filtering dehazing algorithm based on attention mechanism is proposed. Firstly, dense residual block is proposed to increase the depth of the neural network, so that the network can learn more advanced feature information. Then, the spatial and channel attention mechanism is introduced to filter and screen the features, so that the network can distinguish the uneven illumination areas and solve the problems of color distortion. The model adopts the method of spatial local filter enhancement to improve the contrast, clarity and visibility of dehazing results. Finally, a learning of joint loss function constraint network is designed to avoid error amplification and learning hybrid degradation of series structure. In the PyTorch environment, the night urban synthetic haze dataset (NHR) is used for testing and compared with the existing dehazing algorithms such as FFANet and GridDehaze. The experimental results show that compared with other dehazing algorithms, proposed method improves PSNR by 8.01‒14.16 dB and increases SSIM by 0.10‒0.36. In addition, proposed method solves the problems of color distortion, plaque and artifact.

RPPG Signal Measurement Based on Time-do-main Attention Network

The measurement of heart rate without touching the object being measured is referred to as “non-contact heart rate estimation.” Telemedicine, education, and a variety of other fields can all benefit from this approach, which is capable of achieving continuous heart rate measurement under certain conditions. The video segmentation network and the backbone network are cascaded in this paper’s cascaded heart rate estimation model. Adding temporal attention based on the existing 3D convolutional network model, the video segmentation network divides the video into multiple segments and fuses the relevant temporal features. It then sends the fusion results to the backbone network with the temporal attention mechanism. Mechanisms and modules like spatiotemporal convolution to make heart rate measurement more reliable and use a custom joint loss function to cut out other kinds of periodic signals’ influence. The trial results show the way that the proposed calculation can successfully lessen movement ancient rarity and light change clamor.Additionally, the dataset UBFC-rPPG demonstrates the model’s efficacy.

BANet: Boundary-Assistant Encoder-Decoder Network for Semantic Segmentation

Recently, boundary information has gained great attraction for semantic segmentation. This paper presents a novel encoder-decoder network, called BANet, for accurate semantic segmentation, where boundary information is employed as an additional assistance for producing more consistent segmentation outputs. BANet is composed of three components: the pre-trained backbone using dilated-ResNet101, semantic flow branch (SFB) and boundary flow branch (BFB) for semantic segmentation and boundary detection, respectively. More specifically, to delineate more accurate object shapes and boundaries, a global attention block (GAB) is designed in SFB as global guidance for high-level feature. On the other hand, BFB directly extracts features on boundaries, avoiding the unexpected interference from the non-boundary parts. Finally, we adopt a joint loss function to further optimize the segmentation results and boundary outputs synchronously. Moreover, compared with previous state-of-the-art methods, e.g., non-local block and ASPP module, our BFB leverages detection accuracy and computational efficiency in a lightweight fashion. To evaluate BANet, we have conducted extensive experiments on several semantic segmentation datasets: Cityscapes, PASCAL Context, and ADE20K. The experimental results show that, with the aid of boundary information, BANet is able to produce more consistent segmentation predictions with accurately delineated object shapes and boundaries, leading to the state-of-the-art performance on Cityscapes, and competitive results on PASCAL Context and ADE20K with respect to recent semantic segmentation networks.

Annotator-dependent uncertainty-aware estimation of gait relative attributes

In this paper, we describe an uncertainty-aware estimation framework for gait relative attributes. We specifically design a two-stream network model that takes a pair of gait videos as input. It then outputs a corresponding pair of Gaussian distributions of gait absolute attribute scores and annotator-dependent gait relative attribute label distributions. Moreover, we propose a differentiable annotator-independent uncertainty layer to estimate the gait relative attribute score distribution from the absolute distributions then map it to a relative attribute label distribution using the computation of cumulative distribution functions. Furthermore, we propose another annotator-dependent uncertainty layer to estimate the uncertainty on the gait relative attribute labels in terms of a set of trainable transition matrices. Finally, we design a joint loss function on the relative attribute label distribution to learn the model parameters. Experiments on two gait relative attribute datasets demonstrated the effectiveness of the proposed method against baselines in quantitative and qualitative evaluations.

Infrared and visible image fusion using a guiding network to leverage perceptual similarity

In infrared (IR) and visible image fusion, visual appearance of fused images produced by an end-to-end fusion network relies on a loss function that defines the desired properties of the fusion results. The previous approaches mainly focus on measuring the similarity of an output image with respective IR and visible images using pixel-wise loss functions such as mean square error or structural similarity. This can lead, however, to preserve only local information in the fused images, and it may lose semantic information in the scene. In this work, we propose a new joint loss function for the image fusion, which considers visual quality and subsequent applications simultaneously. Our joint loss function guides the fusion network by transferring the knowledge of a pre-trained network to leverage perceptual similarity. The contrast loss function serves to prevent the reduction of global and local image contrasts in the fused image. The conventional data fidelity is also considered in the proposed function. Then, the joint loss function is used to train our U-Net like fusion network, a hybrid of convolutional and transformer blocks. Experimental results show that the proposed method preserves detailed scene structure from the both of source images and demonstrates the better quantitative results. Furthermore, the proposed fusion schemes can be used for combining multi-exposure and multi-focus image pairs. To show the further effectiveness of IR and visible image fusion, we apply the results of fusion methods for downstream tasks so that the proposed method improves the cases of only using original modalities.

Toward underwater image enhancement: new dataset and white balance priors-based fusion network

Raw underwater images are degraded due to light attenuation and particle scattering. It is challenging to obtain authentic reference images. Hence, existing underwater datasets may be short of paired and trainable images. Many data-driven approaches are therefore difficult to implement. To address the above issues, we mainly make two contributions. First, a paired artificial underwater dataset (i.e., PAU-Wild) is constructed, including more than 3500 pairs of images of seven types. PAU-Wild is the first attempt to simulate real-world underwater images with external filters. Second, to enhance underwater images, a convolutional neural network using multistage fusion is proposed. Our model introduces two white balance (WB) priors into the fusion strategy. In addition, a joint loss function is designed to preserve content features and edge features. The experimental results demonstrate that our method is effective and even outperforms several state-of-the-art methods.

MT-GCNN: Multi-Task Learning with Gated Convolution for Multiple Transmitters Localization in Urban Scenarios.

With the advance of the Internet of things (IoT), localization is essential in varied services. In urban scenarios, multiple transmitters localization is faced with challenges such as nonline-of-sight (NLOS) propagation and limited deployment of sensors. To this end, this paper proposes the MT-GCNN (Multi-Task Gated Convolutional Neural Network), a novel multiple transmitters localization scheme based on deep multi-task learning, to learn the NLOS propagation features and achieve the localization. The multi-task learning network decomposes the problem into a coarse localization task and a fine correction task. In particular, the MT-GCNN uses an improved gated convolution module to extract features from sparse sensing data more effectively. In the training stage, a joint loss function is proposed to optimize the two branches of tasks. In the testing stage, the well-trained MT-GCNN model predicts the classified grids and corresponding biases jointly to improve the overall performance of localization. In the urban scenarios challenged by NLOS propagation and sparse deployment of sensors, numerical simulations demonstrate that the proposed MT-GCNN framework has more accurate and robust performance than other algorithms.

Increased joint loading induces subchondral bone loss of the temporomandibular joint via the RANTES-CCRs-Akt2 axis.

Early-stage temporomandibular joint osteoarthritis (TMJOA) is characterized by excessive subchondral bone loss. Emerging evidence suggests that TMJ disc displacement is involved, but the pathogenic mechanism remains unclear. Here, we established a rat model of TMJOA that simulated disc displacement with a capacitance-based force-sensing system to directly measure articular surface pressure in vivo. Micro-CT, histological staining, immunofluorescence staining, IHC staining, and Western blot were used to assess pathological changes and underlying mechanisms of TMJOA in the rat model in vivo as well as in RAW264.7 cells in vitro. We found that disc displacement led to significantly higher pressure on the articular surface, which caused rapid subchondral bone loss via activation of the RANTES-chemokine receptors-Akt2 (RANTES-CCRs-Akt2) axis. Inhibition of RANTES or Akt2 attenuated subchondral bone loss and resulted in improved subchondral bone microstructure. Cytological studies substantiated that RANTES regulated osteoclast formation by binding to its receptor CCRs and activating the Akt2 pathway. The clinical evidence further supported that RANTES was a potential biomarker for predicting subchondral bone loss in early-stage TMJOA. Taken together, this study demonstrates important functions of the RANTES-CCRs-Akt2 axis in the regulation of subchondral bone remodeling and provides further knowledge of how disc displacement causes TMJOA.

Treatment of congenital hip dislocation before the walking age

The worst type of hip developmental dysplasia, known as congenital hip dislocation (CHD), is characterized by acetabular cavity, proximal femoral segment, and ligamentus capsule apparatus dysmorphisms that result in partial or total loss of the hip joint's relationship. We provide the following example: Hip dislocation has been diagnosed in a male infant 2 months old. The patient underwent progressive abduction followed by longitudinal skin traction using the Morel technique. After performing an artrography on the hip while under general anesthesia, which revealed a reducible and stable hip, we continued with spica cast immobilization in a human position. To promote proper joint development, improve standing posture, enhance gait, and correct pelvic and spinal imbalances, the treatment aims to reduce joint dislocation and rebuild joint relationships. To gradually clean the structures and lower the risk of distant avascular necrosis (AVN) of the femoral head development, slow and gradual traction is applied to Morel's bed.

A Transferable Learning Classification Model and Carbon Sequestration Estimation of Crops in Farmland Ecosystem

Frequent agricultural activities in farmland ecosystems bring challenges to crop information extraction from remote sensing (RS) imagery. The accurate spatiotemporal information of crops serves for regional decision support and ecological assessment, such as disaster monitoring and carbon sequestration. Most traditional machine learning algorithms are not appropriate for prediction classification due to the lack of historical ground samples and poor model transfer capabilities. Therefore, a transferable learning model including spatiotemporal capability was developed based on the UNet++ model by integrating feature fusion and upsampling of small samples for Sentinel-2A imagery. Classification experiments were conducted for 10 categories from 2019 to 2021 in Xinxiang City, Henan Province. The feature fusion and upsampling methods improved the performance of the UNet++ model, showing lower joint loss and higher mean intersection over union (mIoU) values. Compared with the UNet, DeepLab V3+, and the pyramid scene parsing network (PSPNet), the improved UNet++ model exhibits the best performance, with a joint loss of 0.432 and a mIoU of 0.871. Moreover, the overall accuracy and macro F1 values of prediction classification results based on the UNet++ model are higher than 83% and 58%, respectively. Based on the reclassification rules, about 3.48% of the farmland was damaged in 2021 due to continuous precipitation. The carbon sequestration of five crops (including corn, peanuts, soybean, rice, and other crops) is estimated, with a total carbon sequestration of 2460.56, 2549.16, and 1814.07 thousand tons in 2019, 2020, and 2021, respectively. The classification accuracy indicates that the improved model exhibits a better feature extraction and transferable learning capability in complex agricultural areas. This study provides a strategy for RS semantic segmentation and carbon sequestration estimation of crops based on a deep learning network.

Flood Detection in Dual-Polarization SAR Images Based on Multi-Scale Deeplab Model

The proliferation of massive polarimetric Synthetic Aperture Radar (SAR) data helps promote the development of SAR image interpretation. Due to the advantages of powerful feature extraction capability and strong adaptability for different tasks, deep learning has been adopted in the work of SAR image interpretation and has achieved good results. However, most deep learning methods only employ single-polarization SAR images and ignore the water features embedded in multi-polarization SAR images. To fully exploit the dual-polarization SAR data and multi-scale features of SAR images, an effective flood detection method for SAR images is proposed in this paper. In the proposed flood detection method, a powerful Multi-Scale Deeplab (MS-Deeplab) model is constructed based on the dual-channel MobileNetV2 backbone and the classic DeeplabV3+ architecture to improve the ability of water feature extraction in SAR images. Firstly, the dual-channel feature extraction backbone based on the lightweight MobileNetV2 separately trains the dual-polarization SAR images, and the obtained training parameters are merged with the linear weighting to fuse dual-polarization water features. Given the multi-scale space information in SAR images, then, a multi-scale feature fusion module is introduced to effectively utilize multi-layer features and contextual information, which enhances the representation of water features. Finally, a joint loss function is constructed based on cross-entropy and a dice coefficient to deal with the imbalanced categorical distribution in the training dataset. The experimental results on the time series of Sentinel-1A SAR images show that the proposed method for flood detection has a strong ability to locate water boundaries and tiny water bodies in complex scenes. In terms of quantitative assessment, MS-Deeplab can achieve a better performance compared with other mainstream semantic segmentation models, including PSPNet, Unet and the original DeeplabV3+ model, with a 3.27% intersection over union (IoU) and 1.69% pixel accuracy (PA) improvement than the original DeeplabV3+ model.

Bibliometric and visualization analysis of mesenchymal stem cells and rheumatoid arthritis (from 2012 to 2021).

BackgroundRheumatoid arthritis (RA) is a chronic autoimmune disease that can lead to joint deformity and loss of function. Recent studies have shown great progress in the research of mesenchymal stem cells (MSCs) in RA. However, thus far, there have been no bibliometric or visualization analyses in this field. This bibliometric analysis provides a comprehensive overview of the general information and research hotspots of MSCs and RA.MethodsArticles relevant to MSCs and RA, published between 2012 and 2021, were searched using the Web of Science Core Collection database. Irrelevant publications were excluded from the analysis. Bibliometric and visualization analyses were conducted using VOSviewer, CiteSpace, and Scimago Graphica.ResultsA total of 577 articles were analyzed. The annual number of publications increased from 2012 to 2017 and plateaued from 2017 to 2021. China and the USA had the largest number of publications. Collaboration among different organizations mainly occurs between institutes of the same country. Stem Cell Research and Therapy and Frontiers in Immunology were the most popular journals in this field. All the top 20 co-cited authors had a positive co-citation relationship. The top references indicate that MSCs can contribute to RA research and treatment mainly via immunomodulation. From 2012 to 2021, “collagen-induced arthritis,” “immunomodulation,” and “therapy” were some of the keywords associated with MSCs and RA, while “extracellular vesicles” showed a strong keyword burst from 2019 to 2021.ConclusionMSCs and RA have been widely studied in different countries and institutions and by different authors over the last ten years. China and the USA had the largest number of publications. Different types of journals provide admirable sources for researchers. Some keywords, including immunomodulation and extracellular vesicles, may be hot spots in the near future. There will be more basic research and clinical translation of MSCs and RA, and substantial new treatments for RA will soon be developed.

Alzheimer's Disease Prediction Algorithm Based on Group Convolution and a Joint Loss Function.

Alzheimer's disease (AD) can effectively predict by 18F-fluorodeoxyglucose positron emission tomography (18F-FDG PET) of the brain, but current PET images still suffer from indistinct lesion features, low signal-to-noise ratios, and severe artefacts, resulting in poor prediction accuracy for patients with mild cognitive impairment (MCI) and unclear lesion features. In this paper, an AD prediction algorithm based on group convolution and a joint loss function is proposed. First, a group convolutional backbone network based on ResNet18 is designed to extract lesion features from multiple channels, which makes the expression ability of the network improved to a great extent. Then, a hybrid attention mechanism is presented, which enables the network to focus on target regions and learn feature weights, so as to enhance the network's learning ability of the lesion regions that are relevant to disease diagnosis. Finally, a joint loss function, that avoids the overfitting phenomenon, increases the generalization of the model, and improves prediction accuracy by adding a regularization loss function to the conventional cross-entropy function, is proposed. Experiments conducted on the public Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset show that the algorithm we proposed gives a prediction accuracy improvement of 2.4% over that of the current AD prediction algorithm, thus proving the effectiveness and availability of the new algorithm.

Multi-task thyroid tumor segmentation based on the joint loss function

Medical image segmentation is one of the important topics in the field of medical image, which has attracted the great attention of researchers in recent years. However, there are few studies on the segmentation methods of thyroid tumors, which are determined by the characteristics of the thyroid and the mode of B-mode ultrasound imaging. Therefore, the segmentation of B-mode ultrasound images of thyroid tumors and the study of benign and malignant classification technology have important value and significance. In this paper, we propose a multi-task segmentation framework based on the joint loss function. First, based on our previous work, Feature Fusion Attention Network to Medical Image Segmentation (FFANet), we add a classification branch to expand it into a multi-task image segmentation framework. Then, we design the corresponding joint loss function and explore the weight coefficients between the two loss functions in the segmentation and classification tasks. Finally, through the experiment on thyroid tumor segmentation and classification dataset, the dice coefficient of the multi-task framework proposed in this paper is 0.935, and the classification accuracy is 0.790. It achieves competitive performance compared to existing methods.