Mixed Loss Research Articles

A mixed loss joint approach-based deep learning strategy to enhance SNR and resolution of arterial spin labelling MRI

A mixed loss joint approach-based deep learning strategy to enhance SNR and resolution of arterial spin labelling MRI

Mitigating Aberration-Induced Noise: A Deep Learning-Based Aberration-to-Aberration Approach.

One of the primary sources of suboptimal image quality in ultrasound imaging is phase aberration. It is caused by spatial changes in sound speed over a heterogeneous medium, which disturbs the transmitted waves and prevents coherent summation of echo signals. Obtaining non-aberrated ground truths in real-world scenarios can be extremely challenging, if not impossible. This challenge hinders the performance of deep learning-based techniques due to the domain shift between simulated and experimental data. Here, for the first time, we propose a deep learning-based method that does not require ground truth to correct the phase aberration problem and, as such, can be directly trained on real data. We train a network wherein both the input and target output are randomly aberrated radio frequency (RF) data. Moreover, we demonstrate that a conventional loss function such as mean square error is inadequate for training such a network to achieve optimal performance. Instead, we propose an adaptive mixed loss function that employs both B-mode and RF data, resulting in more efficient convergence and enhanced performance. Finally, we publicly release our dataset, comprising over 180,000 aberrated single plane-wave images (RF data), wherein phase aberrations are modeled as near-field phase screens. Although not utilized in the proposed method, each aberrated image is paired with its corresponding aberration profile and the non-aberrated version, aiming to mitigate the data scarcity problem in developing deep learning-based techniques for phase aberration correction. Source code and trained model are also available along with the dataset at http://code.sonography.ai/main-aaa.

Blind Image Despeckling Using a Multiscale Attention-Guided Neural Network

Coherent imaging systems have been applied in the detection of target of interest, natural resource exploration, ailment diagnosis, etc. However, it is easy to generate speckle-degraded images due to the coherent interference of reflected echoes, restricting these practical applications. Speckle noise is a granular interference that affects the observed reflectivity. It is often modeled as multiplicative noise with a negative exponential distribution. This non-linear property makes despeckling of imaging data an intractable problem. To enhance despeckling performance, we propose to blindly remove speckle noise using an intelligent computing-enabled multi-scale attention-guided neural network (termed MSANN). In particular, we first introduce the logarithmic transformation to convert the multiplicative speckle noise model to an additive version. Our MSANN, essentially a feature pyramid network (FPN), is then exploited to restore degraded images in the logarithmic domain. To enhance the generalization ability of MSANN, a multi-scale feature enhancement attention module (FEAM) is incorporated into MSANN to extract multi-scale features for improving imaging quality. A multi-scale mixed loss function is further presented to increase network robustness during training. The final despeckled images are naturally equivalent to the exponential versions of the output of MSANN. Experimental results have shown that MSANN has the capacity of effectively removing speckle noise while preserving essential structures. It can achieve superior despeckling results in terms of visual quality and quantitative measures.

DTR-GAN: An Unsupervised Bidirectional Translation Generative Adversarial Network for MRI-CT Registration

Medical image registration is a fundamental and indispensable element in medical image analysis, which can establish spatial consistency among corresponding anatomical structures across various medical images. Since images with different modalities exhibit different features, it remains a challenge to find their exact correspondence. Most of the current methods based on image-to-image translation cannot fully leverage the available information, which will affect the subsequent registration performance. To solve the problem, we develop an unsupervised multimodal image registration method named DTR-GAN. Firstly, we design a multimodal registration framework via a bidirectional translation network to transform the multimodal image registration into a unimodal registration, which can effectively use the complementary information of different modalities. Then, to enhance the quality of the transformed images in the translation network, we design a multiscale encoder–decoder network that effectively captures both local and global features in images. Finally, we propose a mixed similarity loss to encourage the warped image to be closer to the target image in deep features. We extensively evaluate methods for MRI-CT image registration tasks of the abdominal cavity with advanced unsupervised multimodal image registration approaches. The results indicate that DTR-GAN obtains a competitive performance compared to other methods in MRI-CT registration. Compared with DFR, DTR-GAN has not only obtained performance improvements of 2.35% and 2.08% in the dice similarity coefficient (DSC) of MRI-CT registration and CT-MRI registration on the Learn2Reg dataset but has also decreased the average symmetric surface distance (ASD) by 0.33 mm and 0.12 mm on the Learn2Reg dataset.

Blend U-Net: Redesigning Skip Connections to Obtain Multiscale Features for Lung CT Images Segmentation.

Lung cancer is a pervasive and persistent issue worldwide, with the highest morbidity and mortality among all cancers for many years. In the medical field, computer tomography (CT) images of the lungs are currently recognized as the best way to help doctors detect lung nodules and thus diagnose lung cancer. U-Net is a deep learning network with an encoder-decoder structure, which is extensively employed for medical image segmentation and has derived many improved versions. However, these advancements do not utilize various feature information from all scales, and there is still room for future enhancement. In this study, we proposed a new model called Blend U-Net, which incorporates nested structures, redesigned long and short skip connections, and deep supervisions. The nested structures and the long and short skip connections combined characteristic information of different levels from feature maps in all scales, while the deep supervision learning hierarchical representations from all-scale concatenated feature maps. Additionally, we employed a mixed loss function to obtain more accurate results. We evaluated the performance of the Blend U-Net against other architectures on the publicly available Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI) dataset. Moreover, the accuracy of the segmentation was verified by using the dice coefficient. Blend U-Net with a boost of 0.83 points produced the best outcome in a number of baselines. Based on the results, our method achieves superior performance in terms of dice coefficient compared to other methods and demonstrates greater proficiency in segmenting lung nodules of varying sizes.

SLAPP: Subgraph-level attention-based performance prediction for deep learning models

The intricacy of the Deep Learning (DL) landscape, brimming with a variety of models, applications, and platforms, poses considerable challenges for the optimal design, optimization, or selection of suitable DL models. One promising avenue to address this challenge is the development of accurate performance prediction methods. However, existing methods reveal critical limitations. Operator-level methods, proficient at predicting the performance of individual operators, often neglect broader graph features, which results in inaccuracies in full network performance predictions. On the contrary, graph-level methods excel in overall network prediction by leveraging these graph features but lack the ability to predict the performance of individual operators. To bridge these gaps, we propose SLAPP, a novel subgraph-level performance prediction method. Central to SLAPP is an innovative variant of Graph Neural Networks (GNNs) that we developed, named the Edge Aware Graph Attention Network (EAGAT). This specially designed GNN enables superior encoding of both node and edge features. Through this approach, SLAPP effectively captures both graph and operator features, thereby providing precise performance predictions for individual operators and entire networks. Moreover, we introduce a mixed loss design with dynamic weight adjustment to reconcile the predictive accuracy between individual operators and entire networks. In our experimental evaluation, SLAPP consistently outperforms traditional approaches in prediction accuracy, including the ability to handle unseen models effectively. Moreover, when compared to existing research, our method demonstrates a superior predictive performance across multiple DL models.

MSRL-Net: An Automatic Segmentation of Intracranial Hemorrhage for CT Images Based on the U-Net Framework

Intracranial hemorrhage (ICH) is a hemorrhagic disease occurring in the ventricle or brain, but we found that the U-Net network has poor segmentation performance for small lesion areas. In order to improve the segmentation accuracy, a new convolutional neural network called MSRL-Net is proposed in this paper to accurately segment the lesion regions in the CT images of intracranial hemorrhage. Specifically, to avoid the problem of missing information in the downsampling process, we propose a strategy combining MaxPool and SoftPool. In addition, the mixed loss function is used to optimize the unbalance of medical images. Finally, at the bottleneck layer, an MRHDC module is designed to represent the rich spatial information in the underlying features, in order to obtain multi-scale features with different receptive fields. Our model achieves 0.712 average Dice on a dataset. The experimental results show that this model has a good segmentation effect and potential clinical prospects.

HMARNET—A Hierarchical Multi‐Attention Residual Network for Gleason scoring of prostate cancer

AbstractManual delineation of prostate cancer (PCa) from whole slide images (WSIs) demands requires pathologists with adequate domain knowledge. This process is generally strenuous and may be subjected to poor inter‐pathologist reproducibility. Accurate Gleason scoring is an important step in the computer‐aided diagnosis of PCa. This work proposes a novel lightweight convolutional neural networks (CNN) to extract significant hierarchical features from the histopathology images. It learns meticulous attention‐guided feature representations through the convolutional layers for precise scoring of Gleason grades. The Hierarchical Multi‐Attention Residual (HMAR) block extracts attention‐guided features and fuses the resulting feature maps from multiple levels with a reduced number of trainable parameters. We have also proposed a new lightweight channel‐attention module, enhanced channel attention (ECA) to extract inter‐channel features from different receptive fields. With the presence of different attention mechanisms, the network learns to focus on more significant features rather than unnecessary ones. Additionally, mixed FocalOHEM loss is proposed to optimize the CNN and efficiently minimize the error. While the focal loss helps to address the class imbalance present in the TCGA‐PRAD dataset, the online hard example mining (OHEM) loss focuses on optimizing hard negative samples. It achieved an accuracy of 89.19% with a Kappa score of 86% on the TCGA‐PRAD dataset.

DSKCA-UNet: Dynamic selective kernel channel attention for medical image segmentation.

U-Net has attained immense popularity owing to its performance in medical image segmentation. However, it cannot be modeled explicitly over remote dependencies. By contrast, the transformer can effectively capture remote dependencies by leveraging the self-attention (SA) of the encoder. Although SA, an important characteristic of the transformer, can find correlations between them based on the original data, secondary computational complexity might retard the processing rate of high-dimensional data (such as medical images). Furthermore, SA is limited because the correlation between samples is overlooked; thus, there is considerable scope for improvement. To this end, based on Swin-UNet, we introduce a dynamic selective attention mechanism for the convolution kernels. The weight of each convolution kernel is calculated to fuse the results dynamically. This attention mechanism permits each neuron to adaptively modify its receptive field size in response to multiscale input information. A local cross-channel interaction strategy without dimensionality reduction was introduced, which effectively eliminated the influence of downscaling on learning channel attention. Through suitable cross-channel interactions, model complexity can be significantly reduced while maintaining its performance. Subsequently, the global interaction between the encoder features is used to extract more fine-grained features. Simultaneously, the mixed loss function of the weighted cross-entropy loss and Dice loss is used to alleviate category imbalances and achieve better results when the sample number is unbalanced. We evaluated our proposed method on abdominal multiorgan segmentation and cardiac segmentation datasets, achieving Dice similarity coefficient and 95% Hausdorff distance metrics of 80.30 and 14.55%, respectively, on the Synapse dataset and Dice similarity coefficient metrics of 90.80 on the ACDC dataset. The experimental results show that our proposed method has good generalization ability and robustness, and it is a powerful tool for medical image segmentation.

UN-PUNet for phase unwrapping from a single uneven and noisy ESPI phase pattern.

The wrapped phase patterns of objects with varying materials exhibit uneven gray values. Phase unwrapping is a tricky problem from a single wrapped phase pattern in electronic speckle pattern interferometry (ESPI) due to the gray unevenness and noise. In this paper, we propose a convolutional neural network (CNN) model named UN-PUNet for phase unwrapping from a single wrapped phase pattern with uneven grayscale and noise. UN-PUNet leverages the benefits of a dual-branch encoder structure, a multi-scale feature fusion structure, a convolutional block attention module, and skip connections. Additionally, we have created an abundant dataset for phase unwrapping with varying degrees of unevenness, fringe density, and noise levels. We also propose a mixed loss function MS_SSIM+L2. Employing the proposed dataset and loss function, we can successfully train the UN-PUNet, ultimately realizing effective and robust phase unwrapping from a single uneven and noisy wrapped phase pattern. We evaluate the performance of our method on both simulated and experimental ESPI wrapped phase patterns, comparing it with DLPU, VUR-Net, and PU-M-Net. The unwrapping performance is assessed quantitatively and qualitatively. Furthermore, we conduct ablation experiments to evaluate the impact of different loss functions and the attention module utilized in our method. The results demonstrate that our proposed method outperforms the compared methods, eliminating the need for pre-processing, post-processing procedures, and parameter fine-tuning. Moreover, our method effectively solves the phase unwrapping problem while preserving the structure and shape, eliminating speckle noise, and addressing uneven grayscale.

Pore Structure Identification Method for Pervious Concrete Based on Improved UNet and Fusion Algorithm

This paper aims to establish an automatic and accurate pore identification method for pervious concrete. The residual module and mixed loss functions were introduced to the original UNet network to obtain the improved UNet. CT scanning was conducted on the six groups of pervious concrete samples with different aggregate sizes to obtain the initial dataset. The initial dataset was marked and enhanced, and then the pore recognition model was trained. The influence of image brightness and contrast on pore identification was analyzed. The fusion algorithm was used to improve the robustness of the model. The results show that during model training, R-UNet began to converge 20 epochs earlier than UNet and the loss value was smaller. Moreover, the maximum increase of mIoU and mDice was 10.3% and 11.7% respectively, and the maximum decrease of mHD was 14.1%. The fusion algorithm could improve the segmentation accuracy of pores in brightness anomaly images. Compared with threshold segmentation method, the method proposed in this paper could improve the accuracy of pore edge segmentation and the “fine pores” identification, and reduced the pore identification defects. The value of mHD was decreased by 48.7%–72.4%, and the efficiency of pore identification was greatly improved.

A Low-Illumination Enhancement Method Based on Structural Layer and Detail Layer.

Low-illumination image enhancement technology is a topic of interest in the field of image processing. However, while improving image brightness, it is difficult to effectively maintain the texture and details of the image, and the quality of the image cannot be guaranteed. In order to solve this problem, this paper proposed a low-illumination enhancement method based on structural and detail layers. Firstly, we designed an SRetinex-Net model. The network is mainly divided into two parts: a decomposition module and an enhancement module. Second, the decomposition module mainly adopts the SU-Net structure, which is an unsupervised network that decomposes the input image into a structural layer image and detail layer image. Afterward, the enhancement module mainly adopts the SDE-Net structure, which is divided into two branches: the SDE-S branch and the SDE-D branch. The SDE-S branch mainly enhances and adjusts the brightness of the structural layer image through Ehnet and Adnet to prevent insufficient or overexposed brightness enhancement in the image. The SDE-D branch is mainly denoised and enhanced with textural details through a denoising module. This network structure can greatly reduce computational costs. Moreover, we also improved the total variation optimization model as a mixed loss function and added structural metrics and textural metrics as variables on the basis of the original loss function, which can well separate the structure edge and texture edge. Numerous experiments have shown that our structure has a more significant impact on the brightness and detail preservation of image restoration.

A Salient Object Detection Method Based on Boundary Enhancement.

Visual saliency refers to the human's ability to quickly focus on important parts of their visual field, which is a crucial aspect of image processing, particularly in fields like medical imaging and robotics. Understanding and simulating this mechanism is crucial for solving complex visual problems. In this paper, we propose a salient object detection method based on boundary enhancement, which is applicable to both 2D and 3D sensors data. To address the problem of large-scale variation of salient objects, our method introduces a multi-level feature aggregation module that enhances the expressive ability of fixed-resolution features by utilizing adjacent features to complement each other. Additionally, we propose a multi-scale information extraction module to capture local contextual information at different scales for back-propagated level-by-level features, which allows for better measurement of the composition of the feature map after back-fusion. To tackle the low confidence issue of boundary pixels, we also introduce a boundary extraction module to extract the boundary information of salient regions. This information is then fused with salient target information to further refine the saliency prediction results. During the training process, our method uses a mixed loss function to constrain the model training from two levels: pixels and images. The experimental results demonstrate that our salient target detection method based on boundary enhancement shows good detection effects on targets of different scales, multi-targets, linear targets, and targets in complex scenes. We compare our method with the best method in four conventional datasets and achieve an average improvement of 6.2% on the mean absolute error (MAE) indicators. Overall, our approach shows promise for improving the accuracy and efficiency of salient object detection in a variety of settings, including those involving 2D/3D semantic analysis and reconstruction/inpainting of image/video/point cloud data.

X-Linked Gusher Disease DFNX2 in Children, a Rare Inner Ear Dysplasia with Mixed Hearing and Vestibular Loss.

Conductive hearing losses are typically present in disorders of the external/middle ear. However, there is a rare group of inner ear conditions called third windows that can also generate a conductive hearing loss. This is due to an abnormal connection between the middle and the inner ear or between the inner ear and the cranial cavity. X-linked gusher disorder is an extremely rare congenital inner ear dysplastic syndrome with an abnormal connection due to a characteristic incomplete cochlear partition type III and an incomplete internal auditory meatus fundus. The disorder is inherited in an X-linked fashion due to the mutation of the POU3F4 gene. We present two siblings diagnosed with the condition and their long-term follow-ups. They both presented audiovestibular symptoms and showed progressive mixed losses and bilateral vestibular weakness. They were treated with cochlear implant, digital amplification and with vestibular rehabilitation. Significant others around them were involved in their journey with the medical team, and in both, a very favourable outcome was achieved. This is the first time that we have reported evolving audiovestibular function with vestibular quantification in X-linked gusher disorder and emphasize on the multidisciplinary holistic approach to manage these children effectively.

Multi-branch Context Awareness Network for Prostate MRI Segmentation

Prostate image segmentation is a precondition for the diagnosis of prostate diseases and subsequent treatment. However, the blurry organ edges and low image contrast make accurate segmentation difficult. In order to overcome the difficulties, this paper proposes an innovative Multi-branch Context Awareness Network (MBCA-Net) for prostate MRI segmentation. MBCA-Net uses the 3D UNet as the backbone network with an encoder-decoder framework. To improve the feature extraction capacity of the network, a multi-branch residual module with different convolution kernels is proposed. To better make use of extracted features and reduce semantic ambiguity, a context awareness module is used to fuse low-level features in the encoder and high-level features in the decoder in a full process manner. To improve the stability of the model during training, a mixed loss function is adopted. Experiments on the public dataset PROMISE12 show that our model’s Dice coefficient is 90.19%. Compared with other advanced techniques, the proposed model has better segmentation results.

Audiologic Consequences of Gunshot Wounds to the Temporal Bone Show No Relation to Concomitant Neurologic or Vascular Injuries.

To evaluate audiologic consequences of gunshot wounds (GSWs) to the temporal bone (TB), and to correlate hearing outcomes with neurologic and vascular injuries adjacent to the temporal bone. Retrospective case series. University-based level-one trauma center. Retrospective review of 35 patients surviving TB ballistic injury, 2012 to 2021. Main outcomes were audiologic results. Demographics, concomitant injuries, CT, and interventions were reviewed. Mean age was 30.7 years; 80% male. Seventeen patients (48.6%) underwent audiologic testing. Mean pure tone average (PTA) was 75 ± 35 dB, bone line average 41 ± 26 dB, and speech discrimination score (SDS) 60 ± 43%. Nineteen (54.3%) demonstrated facial nerve injury (FNI), who were more likely to show SNHL especially anacusis, though their mean PTA and SDS were not statistically different from those without (P = .30 and .47, respectively). Radiographic review of those with sensorineural loss (SNHL, 6/17) revealed otic capsule-disrupting fracture (n = 2), pneumolabyrinth (n = 2), intracranial hemorrhage (n = 3). Those with mixed loss (6/17) showed otic capsule-sparing fracture (n = 6), EAC injury (n = 5), ossicular discontinuity (n = 2), and intracranial hemorrhage (n = 4). Two with mastoid tip fractures alone had normal audiograms. Audiometric outcomes were not predicted by concomitant CSF leak, spinal injuries, vascular injuries, cranial neuropathies, or traumatic brain injury. All patterns of hearing loss-conductive, sensorineural, mixed and normal-may be seen following TB ballistic injuries. Trauma severe enough to disrupt the facial nerve is more likely to cause anacusis. However, all should be formally evaluated, since ballistic injuries complicated by neurologic or vascular damage do not necessarily correlate with worse audiologic outcomes, while patients with minimal fractures may demonstrate losses.

Conditional temporal GAN for intent-aware vessel trajectory prediction in the precautionary area

Accurate vessel trajectory prediction is crucial for ensuring maritime traffic safety and efficiency, particularly in precaution areas characterized by multi-waterway branch merging and frequent traffic conflicts. However, predicting trajectories in these areas poses significant challenges due to the uncertain future intents of different directional branches resulting in diverse motion patterns and multiple possible paths. To address this issue and minimize the prediction error, we propose a conditional temporal generative adversarial network (CTGAN). Specifically, in this method, the trajectory generator is developed to capture the inherently dynamic of ship motions and outputs the future trajectory proposals, while the intent classifier is designed to evaluate whether the trajectory proposals are consistent with the hidden intention. With the adversarial training strategy, the trajectory generator and intent classifier form a closed loop, feedback informative signals to each other that enables generating the intent-constrained trajectory. In addition, a mixed adversarial loss function was designed to capture the spatial–temporal dependencies among the vessel motions for producing consistent trajectories that complied with plausible ship dynamics. Experiments on extensive naturalistic vessel trajectory data demonstrate that compared with the baseline methods, the proposed model achieves comparable or better prediction performance.

Weakly supervised volumetric prostate registration for MRI-TRUS image driven by signed distance map

Image registration is a fundamental step for MRI-TRUS fusion targeted biopsy. Due to the inherent representational differences between these two image modalities, though, intensity-based similarity losses for registration tend to result in poor performance. To mitigate this, comparison of organ segmentations, functioning as a weak proxy measure of image similarity, has been proposed. Segmentations, though, are limited in their information encoding capabilities. Signed distance maps (SDMs), on the other hand, encode these segmentations into a higher dimensional space where shape and boundary information are implicitly captured, and which, in addition, yield high gradients even for slight mismatches, thus preventing vanishing gradients during deep-network training. Based on these advantages, this study proposes a weakly-supervised deep learning volumetric registration approach driven by a mixed loss that operates both on segmentations and their corresponding SDMs, and which is not only robust to outliers, but also encourages optimal global alignment. Our experimental results, performed on a public prostate MRI-TRUS biopsy dataset, demonstrate that our method outperforms other weakly-supervised registration approaches with a dice similarity coefficient (DSC), Hausdorff distance (HD) and mean surface distance (MSD) of 87.3 ± 11.3, 4.56 ± 1.95 mm, and 0.053 ± 0.026 mm, respectively. We also show that the proposed method effectively preserves the prostate gland's internal structure.

Hybrid Domain Consistency Constraints-Based Deep Neural Network for Facial Expression Recognition.

Facial expression recognition (FER) has received increasing attention. However, multiple factors (e.g., uneven illumination, facial deflection, occlusion, and subjectivity of annotations in image datasets) probably reduce the performance of traditional FER methods. Thus, we propose a novel Hybrid Domain Consistency Network (HDCNet) based on a feature constraint method that combines both spatial domain consistency and channel domain consistency. Specifically, first, the proposed HDCNet mines the potential attention consistency feature expression (different from manual features, e.g., HOG and SIFT) as effective supervision information by comparing the original sample image with the augmented facial expression image. Second, HDCNet extracts facial expression-related features in the spatial and channel domains, and then it constrains the consistent expression of features through the mixed domain consistency loss function. In addition, the loss function based on the attention-consistency constraints does not require additional labels. Third, the network weights are learned to optimize the classification network through the loss function of the mixed domain consistency constraints. Finally, experiments conducted on the public RAF-DB and AffectNet benchmark datasets verify that the proposed HDCNet improved classification accuracy by 0.3-3.84% compared to the existing methods.

ASARU-Net: superimposed U-Net with residual squeeze-and-excitation layer for road crack segmentation

Accurate segmentation of road cracks is vital for identifying cracks in pavements for greater traffic safety. However, owing to blurred boundaries, low contrast between cracks and the surrounding environments, changes in color and shape, most existing segmentation methods face significant challenges in obtaining receptive fields and extracting image feature information. To overcome these issues, we constructed a new framework, named ASARU-Net, to analyze and segment road cracks. We utilized a superimposed U-Net architecture instead of the original 3 × 3 convolution layer to improve the receptive field and enhance segmentation performance. Then, we employed a convolution block in the last layer of the decoding path to obtain more discriminative features. Moreover, the residual mechanism was integrated into a spatial squeeze-and-excitation layer and convolutional block attention module, which improved sensitivity and prediction accuracy. A mixed loss integrating binary cross-entropy and Jaccard loss was used to ensure more balanced segmentation. The proposed method was applied on CRACK500 image database, and achieved a superior performance with Dice, Jaccard, and accuracy values of 79.83%, 69.92%, and 96.94%, respectively. The quantitative and qualitative experimental results show that our method achieves high-performance road crack segmentation and can adapt to cracks with complex background and more interference.