Decoding Stage Research Articles

Echocardiographic mitral valve segmentation model

Segmentation of mitral valve is not only important for clinical diagnosis, but also has far-reaching impact on prevention and prognosis of the disease by experts and doctors.In this paper, the multi-channel cross fusion transformer based U-Net network model (MCCT-UNet) is proposed according to the classical U-Net architecture. First, the jump connection part of MCCT-UNet is designed by using a multi-channel cross-fusion based attention mechanism module (MCCT) instead of the original jump connection, and this module fuses the feature maps from different scales in different stages of the encoder. Second, the optimization of the feature fusion method is proposed in the decoding stage by designing the cross-compression excitation sub-module (C-SENet) to replace the simple feature splicing, and the C-SENet is used to bridge the inconsistency of the semantic hierarchy by effectively combining the deeper information in the encoding stage with the shallower information. This two modules can establish a close connection between the encoder and decoder by exploring multi-scale global contextual information to solve the semantic divide problem, thus it significantly improves the segmentation performance of the network. The experimental results show that the improvement is effective, and the MCCT-UNet model outperforms the other 9 network models. Specifically, the MCCT-UNet achieved a Dice coefficient of 0.8734, an IoU of 0.7854, and an accuracy of 0.9977, demonstrating significant improvements over the compared models.

A Multi-Hierarchical Complementary Feature Interaction Network for Accelerated Multi-Modal MR Imaging

Magnetic resonance (MR) imaging is widely used in the clinical field due to its non-invasiveness, but the long scanning time is still a bottleneck for its popularization. Using the complementary information between multi-modal imaging to accelerate imaging provides a novel and effective MR fast imaging solution. However, previous technologies mostly use simple fusion methods and fail to fully utilize their potential sharable knowledge. In this study, we introduced a novel multi-hierarchical complementary feature interaction network (MHCFIN) to realize joint reconstruction of multi-modal MR images with undersampled data and thus accelerate multi-modal imaging. Firstly, multiple attention mechanisms are integrated with a dual-branch encoder–decoder network to represent shared features and complementary features of different modalities. In the decoding stage, the multi-modal feature interaction module (MMFIM) acts as a bridge between the two branches, realizing complementary knowledge transfer between different modalities through cross-level fusion. The single-modal feature fusion module (SMFFM) carries out multi-scale feature representation and optimization of the single modality, preserving better anatomical details. Extensive experiments are conducted under different sampling patterns and acceleration factors. The results show that this proposed method achieves obvious improvement compared with existing state-of-the-art reconstruction methods in both visual quality and quantity.

Multiple myeloma segmentation net (MMNet): an encoder-decoder-based deep multiscale feature fusion model for multiple myeloma segmentation in magnetic resonance imaging.

Patients with multiple myeloma (MM), a malignant disease involving bone marrow plasma cells, shows significant susceptibility to bone degradation, impairing normal hematopoietic function. The accurate and effective segmentation of MM lesion areas is crucial for the early detection and diagnosis of myeloma. However, the presence of complex shape variations, boundary ambiguities, and multiscale lesion areas, ranging from punctate lesions to extensive bone damage, presents a formidable challenge in achieving precise segmentation. This study thus aimed to develop a more accurate and robust segmentation method for MM lesions by extracting rich multiscale features. In this paper, we propose a novel, multiscale feature fusion encoding-decoding model architecture specifically designed for MM segmentation. In the encoding stage, our proposed multiscale feature extraction module, dilated dense connected net (DCNet), is employed to systematically extract multiscale features, thereby augmenting the model's sensing field. In the decoding stage, we propose the CBAM-atrous spatial pyramid pooling (CASPP) module to enhance the extraction of multiscale features, enabling the model to dynamically prioritize both channel and spatial information. Subsequently, these features are concatenated with the final output feature map to optimize segmentation outcomes. At the feature fusion bottleneck layer, we incorporate the dynamic feature fusion (DyCat) module into the skip connection to dynamically adjust feature extraction parameters and fusion processes. We assessed the efficacy of our approach using a proprietary dataset of MM, yielding notable advancements. Our dataset comprised 753 magnetic resonance imaging (MRI) two-dimensional (2D) slice images of the spinal regions from 45 patients with MM, along with their corresponding ground truth labels. These images were primarily obtained from three sequences: T1-weighted imaging (T1WI), T2-weighted imaging (T2WI), and short tau inversion recovery (STIR). Using image augmentation techniques, we expanded the dataset to 3,000 images, which were employed for both model training and prediction. Among these, 2,400 images were allocated for training purposes, while 600 images were reserved for validation and testing. Our method showed increase in the intersection over union (IoU) and Dice coefficients by 7.9 and 6.7 percentage points, respectively, as compared to the baseline model. Furthermore, we performed comparisons with alternative image segmentation methodologies, which confirmed the sophistication and efficacy of our proposed model. Our proposed multiple myeloma segmentation net (MMNet), can effectively extract multiscale features from images and enhance the correlation between channel and spatial information. Furthermore, a systematic evaluation of the proposed network architecture was conducted on a self-constructed, limited dataset. This endeavor holds promise for offering valuable insights into the development of algorithms for future clinical applications.

End‐to‐end speech‐denoising deep neural network based on residual‐attention gated linear units

AbstractIn this letter, an improved gated linear unit (GLU) structure for end‐to‐end (E2E) speech enhancement is proposed. In the U‐Net structure, which is widely used as the foundational architecture for E2E deep neural network‐based speech denoising, the input noisy speech signal undergoes multiple layers of encoding and is compressed into essential potential representative information at the bottleneck. The latent information is then transmitted to the decoder stage for the restoration of the target clean speech. Among these approaches, CleanUNet, a prominent state‐of‐the‐art (SOTA) method, enhances temporal attention in latent space by employing multi‐head self‐attention. However, unlike the approach of applying the attention mechanism to the potentially compressed representative information of the bottleneck layer, the proposed method instead assigns the attention module to the GLU of each encoder/decoder block layer. The proposed method is validated by measuring short‐term objective speech intelligibility and sound quality. The objective evaluation results indicated that the proposed method using residual‐attention GLU outperformed existing methods using SOTA models such as FAIR‐denoiser and CleanUNet across signal‐to‐noise ratios ranging from 0 to 15 dB.

RAF-Unet: A Remote Sensing Identification Method for Forest Land Information with Modified Unet

Abstract Carrying out remote sensing refinement identification of forest land in complex environment is of great significance for timely mapping of forest distribution. Aiming at the problem that remote sensing images have bias in the extraction of forest land information data, based on the semantic segmentation algorithm Unet, combining the ResNet50 deep learning network, the attention mechanism module and the feature pyramid structure, we construct RAF-Unet (ResNet+Attention+FPN+Unet) to improve the extraction of forest land information data. The ResNet50 classification network is used as the encoder of the Unet network to extract the feature maps at five different scales; then, the attention mechanism module is introduced in the decoder stage of the Unet network to extract the key task goal information by learning the weight values of the features; finally, the feature pyramid structure is used in the output stage of the encoder to fuse the information from the shallow network and the deep network to extract the remote sensing forest land information in the image. The results show that the RAF-Unet algorithm outperforms the Unet algorithm in all the indexes, with a precision of 95.24%, a recall of 91.80%, an F1-score value of 93.49%, an intersection over union of 87.63%, and an accuracy of 93.68%; the validity of the modules is verified by the ablation experiments, and the ResNet network, the attention mechanism, and the feature pyramid structure are all effective in improve the classification effect. It helps the forestry department to better manage and dynamically monitor forestry information, which is of great significance to the scientific development, utilization and protection of forest land resources.

Feature differences reduction and specific features preserving network for RGB-T salient object detection

In RGB-T salient object detection, effective utilization of the different characteristics of RGB and thermal modalities is essential to achieve accurate detection. Most of the previous methods usually only focus on reducing the differences between modalities, which may ignore the specific features that are crucial for salient object detection, leading to suboptimal results. To address the above issue, an RGB-T SOD network that simultaneously considers the reduction of modality differences and the preservation of specific features is proposed. Specifically, we construct a modality differences reduction and specific features preserving module (MDRSFPM) which aims to bridge the gap between modalities and enhance the specific features of each modality. In MDRSFPM, the dynamic vector generated by the interaction of RGB and thermal features is used to reduce modality differences, and then a dual branch is constructed to deal with the RGB and thermal modalities separately, employing a combination of channel-level and spatial-level operations to preserve their respective specific features. In addition, a multi-scale global feature enhancement module (MGFEM) is proposed to enhance global contextual information to provide guidance information for the subsequent decoding stage, so that the model can more easily localize the salient objects. Furthermore, our approach includes a fully fusion and gate module (FFGM) that utilises dynamically generated importance maps to selectively filter and fuse features during the decoding process. Extensive experiments demonstrate that our proposed model surpasses other state-of-the-art models on three publicly available RGB-T datasets remarkably. Our code will be released at https://github.com/JOOOOKII/FRPNet.

CDSG-SAM: A cross-domain self-generating prompt few-shot brain tumor segmentation pipeline based on SAM

In clinical practice, accurate segmentation of brain tumor regions is essential for patient treatment and survival. Accurate brain tumor segmentation is an important task and it is based on a large amount of labeled data. Clinical data are from different cities and device protocols, which leads to the poor generalization ability of the model. In addition, the small amount of clinical data also leads to the decline of the segmentation performance. In this paper, a cross-domain self-generating prompt few-shot brain tumor segmentation pipeline called CDSG-SAM is developed. This pipeline is based on the Segment Anything Model (SAM), which integrates a new Cross-domain self-attention (CDS) Adapter module and Self-Generating (SG) Prompt module. The SAM-based segmentation model is more suitable for few-shot medical image segmentation. A dynamic fuzzy support mask decoder module (DFSMD) is proposed in the decoder stage to enhance the accuracy of brain tumor segmentation edges. Furthermore, a Joint Loss function (JL) is designed to comprehensively consider multiple aspects of performance indicators to improve the accuracy and robustness of the model. Experimental results evaluated on the BraTS2021 glioma dataset, Clinical glioma brain tumor data (CBTD), and BraTS2023 metastatic tumor dataset show that the proposed pipeline has achieved excellent performance with an average dice coefficient of 0.918, 0.828, and 0.868 which outperforms than existing methods.

DCSS-UNet: UNet based on State Space Model for Polyp Segmentation

Early and accurate segmentation of medical images can provide valuable information for medical treatment. In recent years, the automatic and accurate segmentation of polyps in colonoscopy images has received extensive attention from the research community of artificial intelligence and computer vision. Many researchers have conducted in-depth research on models based on CNN and Transformer. However, CNN have limited ability to model remote dependencies, which makes it challenging to fully utilize semantic information in images. On the other hand, the complexity of the secondary computation poses a challenge to the transformer. Recently, state-space models (SSMS), such as Mamba, have been recognized as a promising approach. They not only show superior performance in remote interaction, but also maintain linear computational complexity. Inspired by Mamba, we propose DCSS-UNet, where we utilize visual state space (VSS) blocks in VMamba to capture a wide range of contextual information. In the Skip connection phase, we propose Skip Connects Feature Attention modules(SFA) to better communicate information from the encoder. In the decoder stage, we innovatively combined the Temporal Fusion Attention Module(TFAM) to effectively fuse the feature information. In addition, we introduced a custom Loss calculation method, Tversky Loss, for the model to achieve faster convergence and improve segmentation along polyp boundaries. Our model was trained on the Kvasir-SEG and CVC-ClinicDB datasets, and validated on datasets Kvasir-SEG, CVC-ColonDB, CVC-300, and ETIS. The results show that the model achieves good segmentation accuracy and generalization performance with a low number of parameters. We are 6.1% ahead in the Kavirs-SEG dataset and 3.1% ahead in the CVC-ClinicDB dataset compared to VM-UNet.

Multimodal Data-Driven Segmentation of Bone Metastasis Lesions in SPECT Bone Scans Using Deep Learning

Background: Patients with malignant tumors often develop bone metastases. SPECT bone scintigraphy is an effective tool for surveying bone metastases due to its high sensitivity, low-cost equipment, and radiopharmaceutical. However, the low spatial resolution of SPECT scans significantly hinders manual analysis by nuclear medicine physicians. Deep learning, a promising technique for automated image analysis, can extract hierarchal patterns from images without human intervention. Objective: To enhance the performance of deep learning-based segmentation models, we integrate textual data from diagnostic reports with SPECT bone scans, aiming to develop an automated analysis method that outperforms purely unimodal data-driven segmentation models. Methods: We propose a dual-path segmentation framework to extract features from bone scan images and diagnostic reports separately. In the first path, an encoder-decoder network is employed to learn hierarchical representations of features from SPECT bone scan images. In the second path, the Chinese version of the MacBERT model is utilized to develop a text encoder for extracting features from diagnostic reports. The extracted textual features are then fused with image features during the decoding stage in the first path, enhancing the overall segmentation performance. Results: Experimental evaluation conducted on real-world clinical data demonstrated the superior performance of the proposed segmentation model. Our model achieved a 0.0209 increase in the DSC (Dice Similarity Coefficient) score compared to the well-known U-Net model. Conclusions: The proposed multimodal data-driven method effectively identifies and isolates metastasis lesions in SPECT bone scans, outperforming existing classical deep learning models. This study demonstrates the value of incorporating textual data in the deep learning-based segmentation of lowresolution SPECT bone scans.

Eliminating quantization errors in classification-based sound source localization

Sound Source Localization (SSL) involves estimating the Direction of Arrival (DOA) of sound sources. Since the DOA estimation output space is continuous, regression might be more suitable for DOA, offering higher precision. However, in practice, classification often outperforms regression, exhibiting greater robustness. Conversely, classification’s drawback is inherent quantization error. Within the classification paradigm, the DOA output space is discretized into several intervals, each treated as a class. These classes show strong inter-class correlations, being inherently ordered, with higher similarity as intervals grow closer. Nevertheless, this characteristic has not been fully exploited. To address this, we propose Unbiased Label Distribution (ULD) to eliminate quantization error in training targets. Furthermore, we introduce Weighted Adjacent Decoding (WAD) to overcome quantization error during the decoding stage. Finally, we tailor two loss functions for the soft labels: Negative Log Absolute Error (NLAE) and Mean Squared Error without activation (MSE(wo)). Experimental results show our approach surpasses classification quantization limits, achieving state-of-the-art performance. Our code and supplementary material are available at https://github.com/linfeng-feng/ULD.

Stage-Aware Interaction Network for Point Cloud Completion

Point cloud completion aims to restore full shapes of objects from partial scans, and a typical network pipeline is AutoEncoder, which has coarse-to-fine refinement modules. Although existing approaches using this kind of architecture achieve promising results, they usually neglect the usage of shallow geometry features in partial inputs and the fusion of multi-stage features in the upsampling process, which prevents network performances from further improving. Therefore, in this paper, we propose a new method with dense interactions between different encoding and decoding steps. First, we introduce the Decoupled Multi-head Transformer (DMT), which implements and integrates semantic prediction and resolution upsampling in a unified network module, which serves as a primary ingredient in our pipeline. Second, we propose an Encoding-aware Coarse Decoder (ECD) that compactly makes the top–down shape-decoding process interact with the bottom–up feature-encoding process to utilize both shallow and deep features of partial inputs for coarse point cloud generation. Third, we design a Stage-aware Refinement Group (SRG), which comprehensively understands local semantics from densely connected features across different decoding stages and gradually upsamples point clouds based on them. In general, the key contributions of our method are the DMT for joint semantic-resolution generation, the ECD for multi-scale feature fusion-based shape decoding, and the SRG for stage-aware shape refinement. Evaluations on two synthetic and three real-world datasets illustrate that our method achieves competitive performances compared with existing approaches.

A 3D hierarchical cross-modality interaction network using transformers and convolutions for brain glioma segmentation in MR images.

Precise glioma segmentation from multi-parametric magnetic resonance (MR) images is essential for brain glioma diagnosis. However, due to the indistinct boundaries between tumor sub-regions and the heterogeneous appearances of gliomas in volumetric MR scans, designing a reliable and automated glioma segmentation method is still challenging. Although existing 3D Transformer-based or convolution-based segmentation networks have obtained promising results via multi-modal feature fusion strategies or contextual learning methods, they widely lack the capability of hierarchical interactions between different modalities and cannot effectively learn comprehensive feature representations related to all glioma sub-regions. To overcome these problems, in this paper, we propose a 3D hierarchical cross-modality interaction network (HCMINet) using Transformers and convolutions for accurate multi-modal glioma segmentation, which leverages an effective hierarchical cross-modality interaction strategy to sufficiently learn modality-specific and modality-shared knowledge correlated to glioma sub-region segmentation from multi-parametric MR images. In the HCMINet, we first design a hierarchical cross-modality interaction Transformer (HCMITrans) encoder to hierarchically encode and fuse heterogeneous multi-modal features by Transformer-based intra-modal embeddings and inter-modal interactions in multiple encoding stages, which effectively captures complex cross-modality correlations while modeling global contexts. Then, we collaborate an HCMITrans encoder with a modality-shared convolutional encoder to construct the dual-encoder architecture in the encoding stage, which can learn the abundant contextual information from global and local perspectives. Finally, in the decoding stage, we present a progressive hybrid context fusion (PHCF) decoder to progressively fuse local and global features extracted by the dual-encoder architecture, which utilizes the local-global context fusion (LGCF) module to efficiently alleviate the contextual discrepancy among the decoding features. Extensive experiments are conducted on two public and competitive glioma benchmark datasets, including the BraTS2020 dataset with 494 patients and the BraTS2021 dataset with 1251 patients. Results show that our proposed method outperforms existing Transformer-based and CNN-based methods using other multi-modal fusion strategies in our experiments. Specifically, the proposed HCMINet achieves state-of-the-art mean DSC values of 85.33% and 91.09% on the BraTS2020 online validation dataset and the BraTS2021 local testing dataset, respectively. Our proposed method can accurately and automatically segment glioma regions from multi-parametric MR images, which is beneficial for the quantitative analysis of brain gliomas and helpful for reducing the annotation burden of neuroradiologists.

MH2AFormer: An Efficient Multiscale Hierarchical Hybrid Attention With a Transformer for Bladder Wall and Tumor Segmentation.

Achieving accurate bladder wall and tumor segmentation from MRI is critical for diagnosing and treating bladder cancer. However, automated segmentation remains challenging due to factors such as comparable density distributions, intricate tumor morphologies, and unclear boundaries. Considering the attributes of bladder MRI images, we propose an efficient multiscale hierarchical hybrid attention with a transformer (MH2AFormer) for bladder cancer and wall segmentation. Specifically, a multiscale hybrid attention and transformer (MHAT) module in the encoder is designed to adaptively extract and aggregate multiscale hybrid feature representations from the input image. In the decoder stage, we devise a multiscale hybrid attention (MHA) module to generate high-quality segmentation results from multiscale hybrid features. Combining these modules enhances the feature representation and guides the model to focus on tumor and wall regions, which helps to solve bladder image segmentation challenges. Moreover, MHAT utilizes the Fast Fourier Transformer with a large kernel (e.g., 224 × 224) to model global feature relationships while reducing computational complexity in the encoding stage. The model performance was evaluated on two datasets. As a result, the model achieves relatively best results regarding the intersection over union (IoU) and dice similarity coefficient (DSC) on both datasets (Dataset A: IoU = 80.26%, DSC = 88.20%; Dataset B: IoU = 89.74%, DSC = 94.48%). These advantageous outcomes substantiate the practical utility of our approach, highlighting its potential to alleviate the workload of radiologists when applied in clinical settings.

Research on identification of nucleus-shaped anomaly regions in space electric field

The presence of nucleus-shaped anomalous regions in the power spectrum image of the electric field VLF frequency band has been discovered in previous studies. To detect and analyze these nucleus-shaped abnormal areas and improve the recognition rate of nucleus-shaped abnormal areas, this paper proposes a new nucleus-shaped abnormal area detection model ODM_Unet (Omni-dimensional Dynamic Mobile U-net) based on U-net network. Firstly, the power spectrum image data used for training is created and labeled to form a dataset of nucleus-shaped anomalous regions; Secondly, the ODConv (Omni-dimensional Dynamic Convolution) module with embedded attention mechanism was introduced to improve the encoder, extracting nucleus-shaped anomaly region information from four dimensions and focusing on the features of different input data; An SDI (Semantics and Detail Infusion) module is introduced between the encoder and decoder to solve the problem of detail semantic loss in high-level images caused by the reduction of downsampling image size; In the decoder stage, the SCSE (Spatial and Channel Squeeze-and-Excitation) attention module is introduced to more finely adjust the feature maps output through the SDI module. The experimental results show that compared with the current popular semantic segmentation algorithms, the ODM_Unet model has the best detection performance in nucleus-shaped anomaly areas. Using this model to detect data from November 2021 to October 2022, it was found that the frequency of nucleus-shaped anomaly areas is mostly between 0 and 12.5KHz, with geographic spatial distribution ranging from 40° to 70° south and north latitudes, and magnetic latitude spatial distribution ranging from 58° to 80° south and north latitudes. This method has reference significance for detecting other types of spatial electromagnetic field disturbances.

Imbalanced segmentation for abnormal cotton fiber based on GAN and multiscale residual U-Net

The scale of white foreign fibers in bobbin yarn is small, resulting in multiple types of data imbalance in the dataset. These imbalances include a severe imbalance of foreign fiber pixels compared to background pixels and an imbalance in the size target scale. Consequently, conventional semantic segmentation networks struggle to segment these fibers effectively. First, in tackling the scarcity of white foreign fiber instances within bobbin yarn samples, this research utilizes original foreign fiber images to train the DCGAN and generate adequate training samples. Secondly, a multiscale residual U-Net is constructed to extract foreign fiber features from different scales. The network is encouraged to learn semantic features at each scale and each layer of the decoding stage. This overcomes the problem of scale imbalance in the foreign fiber dataset and enhances the model's capability to extract weak semantic information from small targets. Thirdly, a weighted binary cross-entropy loss function is integrated into the network's training phase to rectify the class imbalance and refine segmentation performance. This function adjusts the weighting of foreign fiber pixel data, thereby addressing the disproportionate distribution between foreign fibers and background pixels within the dataset. Finally, the proposed method is experimentally validated using a dataset of white foreign fibers. The experimental results show that the proposed method achieves better results in the critical evaluation metrics, as evidenced by the accuracy of 97.52 %, the MIoU of 95.26 %, the DICE coefficient of 81.29 %, and the F1 Score of 84.92 %. These statistics demonstrate the method's efficacy in achieving high-precision segmentation of white foreign fibers.

Multi-scale and multi-path cascaded convolutional network for semantic segmentation of colorectal polyps

Colorectal polyps are structural abnormalities of the gastrointestinal tract that can potentially become cancerous in some cases. The study introduces a novel framework for colorectal polyp segmentation named the Multi-Scale and Multi-Path Cascaded Convolution Network (MMCC-Net), aimed at addressing the limitations of existing models, such as inadequate spatial dependence representation and the absence of multi-level feature integration during the decoding stage by integrating multi-scale and multi-path cascaded convolutional techniques and enhances feature aggregation through dual attention modules, skip connections, and a feature enhancer. MMCC-Net achieves superior performance in identifying polyp areas at the pixel level. The Proposed MMCC-Net was tested across six public datasets and compared against eight SOTA models to demonstrate its efficiency in polyp segmentation. The MMCC-Net's performance shows Dice scores with confidence interval ranging between 77.43 ± 0.12, (77.08, 77.56) and 94.45 ± 0.12, (94.19, 94.71) and Mean Intersection over Union (MIoU) scores with confidence interval ranging from 72.71 ± 0.19, (72.20, 73.00) to 90.16 ± 0.16, (89.69, 90.53) on the six databases. These results highlight the model's potential as a powerful tool for accurate and efficient polyp segmentation, contributing to early detection and prevention strategies in colorectal cancer.

A novel hybrid decoding neural network for EEG signal representation

In this paper, we proposed a novel hybrid decoding model that combines the superiority of CNNs and multi-head self-attention mechanisms, called HCANN, to finely characterizing EEG features. Depthwise separable convolution with multi-scale factors efficiently decouples temporal relevant information between brain-computer interface (BCI) tasks and EEG signals. Multi-head mechanism adaptively modified for EEG focuses on brain spatial activation patterns and extracts complementary spatial representation information from multiple subspaces. The proposed HCANN decodes the intent information of EEG recorded by three BCI paradigms, including one active and two passive BCI paradigms: rapid serial visual presentation, motor imagery, and imagined speech. We evaluated HCANN by comparing with the current state-of-the-art methods. The experimental results demonstrated that HCANN can effectively decode EEG and improves classification performance for all three BCI tasks. In addition, the visualization of spatial-temporal features at different decoding stages demonstrated that the proposed HCANN gradually extracts effective features related to the BCI tasks. The code of HCANN is publicly available at https://github.com/youshuoji/HCANN.

A multi-scale context-aware and batch-independent lightweight network for green tide extraction from SAR images

ABSTRACT The outbreaks of green tide have caused severe harm to the marine environment and human society. Synthetic Aperture Radar (SAR) plays an important role in green tide monitoring by virtue of its high resolution and cloud-free nature. The existing green tide extraction methods still face challenges in identifying multi-scale green tide patches due to noise interference, uneven greyscale and blurred boundaries in SAR images. Meanwhile, the practical application of deep learning methods with high precision is limited due to the complexity of the model and the large amount of computation. Therefore, we propose a multi-scale context-aware and batch-independent lightweight green tide extraction network called MBL-Net. A novel lightweight heterogeneous backbone is designed to extract multi-scale discriminative features and improve segmentation efficiency by using multi-scale selection kernel (MSK) modules and lightweight stages. Meanwhile, Triplet attention module is introduced to improve the internal consistency of the green tide region and suppress the effect of speckle noise. Then, the mixed pooling-based channel prior module (MCPM) is used to expand the receptive field of the network and extract the fine green tide structure by fusing multi-scale features. In addition, Filter Response Normalisation (FRN) is innovatively applied for feature normalization in the decoding stage, eliminating batch dependency. In order to verify the effectiveness of the proposed method, a dataset is built using the Sentinel-1 images of the Yellow Sea, China, from 2019 to 2021. The experimental results show that the proposed method achieves an overall accuracy of 98.59% with 0.970 G FLOPs and 3.525 M parameters, which ensures high precision and improves green tide detection efficiency. Compared with several representative networks, this method can capture more details of green tide with fewer parameters and faster calculation speed.

MaMfi-Net: Multi-attention and multi-feature interaction network in skin lesion segmentation

Segmenting skin lesions from dermoscopic images accurately and reliably is essential for disease diagnosis and treatment. However, because lesions vary in size, shape, and color, it remains difficult to detect lesion boundaries and local features. Furthermore, manually portraying lesions takes time and requires the expertise of a skilled physician. To address this issue, a segmentation network with multi-attention mechanisms and multi-feature interactions (MaMfi-Net) is proposed, which can accurately capture the main area of lesions at the encoding stage and efficiently supplement detailed information at the decoding stage to synthetically improve skin lesions' prediction capability. We conducted extensive experiments on two publicly available datasets, including ISIC-2016 and ISIC-2018. Experiment findings show that our method outperforms several cutting-edge approaches, with segmentation results that are closer to the true lesions and superior prediction ability for some challenging images. Meanwhile, the metrics have a lower margin of error.

Dual-branch deep cross-modal interaction network for semantic segmentation with thermal images

Semantic segmentation using RGB (Red-Green-Blue) images and thermal datas is an indispensable component of autonomous driving. The key to RGB-Thermal (RGB and Thermal) semantic segmentation is achieving the interaction and fusion of features between RGB and thermal images. Therefore, we propose a dual-branch deep cross-modal interaction network (DCIT) based on Encoder–Decoder structure. This framework consists of two parallel networks for feature extraction from RGB and Thermal data. Specifically, in each feature extraction stage of the Encoder, we design a Cross Feature Regulation Modules (CFRM) to align and correct modality specific features by reducing the inter-modality feature differences and eliminating intra-modality noise. Then, the modality features are aggregated through Cross Modal Feature Fusion Module (CMFFM) based on cross linear attention to capture global information from modality features. Finally, Adaptive Multi-Scale Cross-positional Fusion Module (AMCFM) utilizes the fused features to integrate consistent semantic information in the Decoder stage. Our framework can improve the interaction of cross modal features. Extensive experiments on urban scene datasets demonstrate that our proposed framework outperforms other RGB-Thermal semantic segmentation methods in terms of objective metrics and subjective visual assessments.