U-Net Framework Research Articles

An optimized siamese neural network with deep linear graph attention model for gynaecological abdominal pelvic masses classification.

An adnexal mass, also known as a pelvic mass, is a growth that develops in or near the uterus, ovaries, fallopian tubes, and supporting tissues. For women suspected of having ovarian cancer, timely and accurate detection of a malignant pelvic mass is crucial for effective triage, referral, and follow-up therapy. While various deep learning techniques have been proposed for identifying pelvic masses, current methods are often not accurate enough and can be computationally intensive. To address these issues, this manuscript introduces an optimized Siamese circle-inspired neural network with deep linear graph attention (SCINN-DLGN) model designed for pelvic mass classification. The SCINN-DLGN model is intended to classify pelvic masses into three categories: benign, malignant, and healthy. Initially, real-time MRI pelvic mass images undergo pre-processing using semantic-aware structure-preserving median morpho-filtering to enhance image quality. Following this, the region of interest (ROI) within the pelvic mass images is segmented using an EfficientNet-based U-Net framework, which reduces noise and improves the accuracy of segmentation. The segmented images are then analysed using the SCINN-DLGN model, which extracts geometric features from the ROI. These features are classified into benign, malignant, or healthy categories using a deep clustering algorithm integrated into the linear graph attention model. The proposed system is implemented on a Python platform, and its performance is evaluated using real-time MRI pelvic mass datasets. The SCINN-DLGN model achieves an impressive 99.9% accuracy and 99.8% recall, demonstrating superior efficiency compared to existing methods and highlighting its potential for further advancement in the field.

Enhanced Hybrid U-Net Framework for Sophisticated Building Automation Extraction Utilizing Decay Matrix

Automatically extracting buildings from remote sensing imagery using deep learning techniques has become essential for various real-world applications. However, mainstream methods often encounter difficulties in accurately extracting and reconstructing fine-grained features due to the heterogeneity and scale variations in building appearances. To address these challenges, we propose LDFormer, an advanced building segmentation model based on linear decay. LDFormer introduces a multi-scale detail fusion bridge (MDFB), which dynamically integrates shallow features to enhance the representation of local details and capture fine-grained local features effectively. To improve global feature extraction, the model incorporates linear decay self-attention (LDSA) and depthwise large separable kernel multi-layer perceptron (DWLSK-MLP) optimizations in the decoder. Specifically, LDSA employs a linear decay matrix within the self-attention mechanism to address long-distance dependency issues, while DWLSK-MLP utilizes step-wise convolutions to achieve a large receptive field. The proposed method has been evaluated on the Massachusetts, Inria, and WHU building datasets, achieving IoU scores of 76.10%, 82.87%, and 91.86%, respectively. LDFormer demonstrates superior performance compared to existing state-of-the-art methods in building segmentation tasks, showcasing its significant potential for building automation extraction.

Efficient Biomedical Waste Management: Optimizing Segmentation and Classification with EnU-Net-DNN-BMWC

<p style="margin-bottom:8px;text-align:justify;"><span style="font-family:"Times New Roman",serif;font-size:12.0pt;line-height:200%;tab-stops:0in 21.3pt;">In India, biomedical waste (BMW) management is governed by strict regulations to mitigate health and environmental risks. It includes materials such as sharps, infectious wastes, pharmaceuticals, and non-hazardous items contaminated with potentially infectious substances. Proper management and disposal of BMW are critical to prevent environmental contamination and health risks.  This article introduces an Enhanced U-Network (EnU-Net) integrated with Deep Neural Network BMW Classification (EnU-Net-DNN-BMWC) to enhance accuracy in this critical task. Leveraging the U-Net framework with an Encoder-Decoder Network (EDN) and pixel-wise classification layer initially optimizes image segmentation. Bayesian functions mitigate segmentation uncertainties, while Content-Sensitive Sampling (CSS) refines pixel sampling to prioritize data-sparse regions. Data collected from G. Kuppuswamy Naidu Memorial Hospital and Kovai Medical Center and Hospital (KMCH) in Coimbatore over six months, categorizing sharps, infectious materials, pharmaceuticals, and non-hazardous waste, informs waste management strategies. Experimental validation using 100 biomedical waste images demonstrates EnU-Net-DNN-BMWC achieving good accuracy, surpassing standalone DNN-BMWC using Matlab 2022. The comparative analysis across different metrics such as accuracy, precision, F-measure, recall, error rate, and RMSE between DNN-BMWC and the proposed EnU-Net-DNN-BMWC framework were evaluated in this study, highlighting EnU-Net-DNN-BMWC's superior performance. Finally, this study underscores EnU-Net-DNN-BMWC’s efficacy in enhancing biomedical waste classification, crucial for sustainable waste management practices and regulatory compliance in healthcare settings.</span></p>

Computer vision-based cascade detection of peeling and cracking in masonry house walls

Self-built masonry structures are commonly used as houses in rural areas of the central plains of China. These structures pose significant safety risks due to the low level of self-building and non-standard designs. The primary type of damage to the walls of masonry houses is cracks, while the decorative layers of the walls may also experience peeling or cracking. The shapes, sizes, and surface characteristics of these damages are complex and unique to each type. To achieve fast identification of surface damage on masonry house walls, this paper proposes a cascade detection model that combines a multilevel cascade classifier with a parallel sub-segmentation network. The VGG16 neural network is used as the backbone for a multi-level series classifier. The model is trained by using a damage image set of the wall decorative layers in rural masonry houses. The classification of background and damage, as well as the classification of cracks and peels, are sequentially completed. Then, the parallel sub-segmentation model uses EfficientNet-B7 as the encoder and combines it with the U-Net framework skeleton to perform pixel-level segmentation of peeling and cracks. Finally, the output of parallel sub-segmentation networks is superimposed and fused to generate a complete image containing segmentation information of peeling and cracks. The cascade form network structure adopted in this model significantly reduces the training difficulty and enhances the model accuracy. Compared with Segformer and Deeplabv3+ network, the average IoU of the proposed method for on-site masonry wall images can be increased by 0.29 and 0.08, respectively.

EfficientUNetViT: Efficient Breast Tumor Segmentation Utilizing UNet Architecture and Pretrained Vision Transformer.

This study introduces a sophisticated neural network structure for segmenting breast tumors. It achieves this by combining a pretrained Vision Transformer (ViT) model with a UNet framework. The UNet architecture, commonly employed for biomedical image segmentation, is further enhanced with depthwise separable convolutional blocks to decrease computational complexity and parameter count, resulting in better efficiency and less overfitting. The ViT, renowned for its robust feature extraction capabilities utilizing self-attention processes, efficiently captures the overall context within images, surpassing the performance of conventional convolutional networks. By using a pretrained ViT as the encoder in our UNet model, we take advantage of its extensive feature representations acquired from extensive datasets, resulting in a major enhancement in the model's ability to generalize and train efficiently. The suggested model has exceptional performance in segmenting breast cancers from medical images, highlighting the advantages of integrating transformer-based encoders with efficient UNet topologies. This hybrid methodology emphasizes the capabilities of transformers in the field of medical image processing and establishes a new standard for accuracy and efficiency in activities related to tumor segmentation.

Dental Caries Segmentation using Deformable Dense Residual Half U-Net for Teledentistry System

Clinical practitioners’ workload and challenges are significantly reduced by classifying, predicting, and localizing lesions or dental caries. In recent research, a high-reliability diagnostic system within deep learning models has been implemented in a clinical teledentistry system. In order to construct an efficient, precise, and lightweight deep learning architecture, it is dynamically structured. In this paper, we present an efficient, accurate, and lightweight deep learning architecture for augmenting spatial locations and improving the transformation modeling abilities of fixed-structure CNNs. Deformable Dense Residual (DDR) enhances the efficacy of the residual convolution block by optimizing its structure, thereby mitigating model redundancy and ameliorating the challenge of vanishing gradients encountered during the training stages. DDR Half U-Net presents notable advancements to the simplified U-Net framework across three pivotal domains: the encoder, decoder, and loss function. Specifically, the encoder integrates deformable convolutions, thereby enhancing the model's capacity to discern features of diverse scales and configurations. In the decoder, a sophisticated arrangement of dense residual connections facilitates the fusion of low-level and high-level features, contributing to comprehensive feature extraction. Moreover, the utilization of a weight-adaptive loss function ensures equitable consideration of both caries and non-caries samples, thereby promoting balanced optimization during training.

Retinal Vessel Segmentation by a Transformer-U-Net Hybrid Model With Dual-Path Decoder.

This paper introduces an effective and efficient framework for retinal vessel segmentation. First, we design a Transformer-CNN hybrid model in which a Transformer module is inserted inside the U-Net to capture long-range interactions. Second, we design a dual-path decoder in the U-Net framework, which contains two decoding paths for multi-task outputs. Specifically, we train the extra decoder to predict vessel skeletons as an auxiliary task which helps the model learn balanced features. The proposed framework, named as TSNet, not only achieves good performances in a fully supervised learning manner but also enables a rough skeleton annotation process. The annotators only need to roughly delineate vessel skeletons instead of giving precise pixel-wise vessel annotations. To learn with rough skeleton annotations plus a few precise vessel annotations, we propose a skeleton semi-supervised learning scheme. We adopt a mean teacher model to produce pseudo vessel annotations and conduct annotation correction for roughly labeled skeletons annotations. This learning scheme can achieve promising performance with fewer annotation efforts. We have evaluated TSNet through extensive experiments on five benchmarking datasets. Experimental results show that TSNet yields state-of-the-art performances on retinal vessel segmentation and provides an efficient training scheme in practice.

Spatial–angular–epipolar transformer for light field spatial and angular super-resolution

Transformer-based light field (LF) super-resolution (SR) methods have recently achieved significant performance improvements due to global feature modeling by self-attention mechanisms. However, as a method designed for natural language processing, 4D LFs are reshaped into 1D sequences with an immense set of tokens, which results in a quadratic computational complexity cost. In this paper, a spatial–angular–epipolar swin transformer (SAEST) is proposed for spatial and angular SR (SASR), which sufficiently extracts SR information in the spatial, angular, and epipolar domains using local self-attention with shifted windows. Specifically, in SAEST, a spatial swin transformer and an angular standard transformer are firstly cascaded to extract spatial and angular SR features, separately. Then, the extracted SR feature is reshaped into the epipolar plane image pattern and fed into an epipolar swin transformer to extract the spatial–angular correlation information. Finally, several SAEST blocks are cascaded in a Unet framework to extract multi-scale SR features for SASR. Experiment results indicate that SAEST is a fast transformer-based SASR method with less running time and GPU consumption and has outstanding performance on simulated and real-world public datasets.

AsymUNet: An Efficient Multi-Layer Perceptron Model Based on Asymmetric U-Net for Medical Image Noise Removal

With the continuous advancement of deep learning technology, U-Net–based algorithms for image denoising play a crucial role in medical image processing. However, most U-Net-based medical image denoising algorithms typically have large parameter sizes, which poses significant limitations in practical applications where computational resources are limited or large-scale patient data processing are required. In this paper, we propose a medical image denoising algorithm called AsymUNet, developed using an asymmetric U-Net framework and a spatially rearranged multilayer perceptron (MLP). AsymUNet utilizes an asymmetric U-Net to reduce the computational burden, while a multiscale feature fusion module enhances the feature interaction between the encoder and decoder. To better preserve the image details, spatially rearranged MLP blocks serve as the core building blocks of AsymUNet. These blocks effectively extract both the local and global features of the image, reducing the model’s reliance on prior knowledge of the image and further accelerating the training and inference processes. Experimental results demonstrate that AsymUNet achieves superior performance metrics and visual results compared with other state-of-the-art methods.

Defog YOLO for road object detection in foggy weather

Abstract Object detection research predominantly focuses on clear weather conditions, often overlooking the challenges posed by foggy weather. Fog impairs the vision of onboard cameras, creating significant obstacles for autonomous vehicles. To tackle these issues, we present the Defog YOLO algorithm, specifically designed for road object detection in foggy conditions. Our approach integrates an enhanced U-Net framework for visual defogging, where the encoder leverages super-resolution back projection to combine multi-layer features. The decoder employs a back projection feedback mechanism to improve image restoration. Additionally, we augment the Feature Pyramid Network with a noise-aware attention mechanism, allowing the network to emphasize critical channel and spatial information while mitigating noise. Given the scarcity of labeled foggy images, we introduce a fog addition module to generate a more diverse training dataset. We validate our method using a synthesized FOG-TRAINVAL dataset, derived from the VOC dataset, demonstrating its robustness in foggy scenarios. Experimental results show that our proposed method achieves an mAP score of 60% on the Real-world Task-driven Testing Set foggy weather test set, with a precision of 86.7% and a recall of 54.2%. These findings underscore the effectiveness and improved generalizability of our approach for object detection in adverse weather conditions.

Synthesizing High b-Value Diffusion-Weighted Imaging of Gastric Cancer Using an Improved Vision Transformer CycleGAN.

Diffusion-weighted imaging (DWI), a pivotal component of multiparametric magnetic resonance imaging (mpMRI), plays a pivotal role in the detection, diagnosis, and evaluation of gastric cancer. Despite its potential, DWI is often marred by substantial anatomical distortions and sensitivity artifacts, which can hinder its practical utility. Presently, enhancing DWI's image quality necessitates reliance on cutting-edge hardware and extended scanning durations. The development of a rapid technique that optimally balances shortened acquisition time with improved image quality would have substantial clinical relevance. This study aims to construct and evaluate the unsupervised learning framework called attention dual contrast vision transformer cyclegan (ADCVCGAN) for enhancing image quality and reducing scanning time in gastric DWI. The ADCVCGAN framework, proposed in this study, employs high b-value DWI (b = 1200 s/mm2) as a reference for generating synthetic b-value DWI (s-DWI) from acquired lower b-value DWI (a-DWI, b = 800 s/mm2). Specifically, ADCVCGAN incorporates an attention mechanism CBAM module into the CycleGAN generator to enhance feature extraction from the input a-DWI in both the channel and spatial dimensions. Subsequently, a vision transformer module, based on the U-net framework, is introduced to refine detailed features, aiming to produce s-DWI with image quality comparable to that of b-DWI. Finally, images from the source domain are added as negative samples to the discriminator, encouraging the discriminator to steer the generator towards synthesizing images distant from the source domain in the latent space, with the goal of generating more realistic s-DWI. The image quality of the s-DWI is quantitatively assessed using metrics such as the peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), feature similarity index (FSIM), mean squared error (MSE), weighted peak signal-to-noise ratio (WPSNR), and weighted mean squared error (WMSE). Subjective evaluations of different DWI images were conducted using the Wilcoxon signed-rank test. The reproducibility and consistency of b-ADC and s-ADC, calculated from b-DWI and s-DWI, respectively, were assessed using the intraclass correlation coefficient (ICC). A statistical significance level of p < 0.05 was considered. The s-DWI generated by the unsupervised learning framework ADCVCGAN scored significantly higher than a-DWI in quantitative metrics such as PSNR, SSIM, FSIM, MSE, WPSNR, and WMSE, with statistical significance (p < 0.001). This performance is comparable to the optimal level achieved by the latest synthetic algorithms. Subjective scores for lesion visibility, image anatomical details, image distortion, and overall image quality were significantly higher for s-DWI and b-DWI compared to a-DWI (p < 0.001). At the same time, there was no significant difference between the scores of s-DWI and b-DWI (p > 0.05). The consistency of b-ADC and s-ADC readings was comparable among different readers (ICC: b-ADC 0.87-0.90; s-ADC 0.88-0.89, respectively). The repeatability of b-ADC and s-ADC readings by the same reader was also comparable (Reader1 ICC: b-ADC 0.85-0.86, s-ADC 0.85-0.93; Reader2 ICC: b-ADC 0.86-0.87, s-ADC 0.89-0.92, respectively). ADCVCGAN shows excellent promise in generating gastric cancer DWI images. It effectively reduces scanning time, improves image quality, and ensures the authenticity of s-DWI images and their s-ADC values, thus providing a basis for assisting clinical decision making.

Deep learning-based segmentation of left ventricular myocardium on dynamic contrast-enhanced MRI: a comprehensive evaluation across temporal frames.

Cardiac perfusion MRI is vital for disease diagnosis, treatment planning, and risk stratification, with anomalies serving as markers of underlying ischemic pathologies. AI-assisted methods and tools enable accurate and efficient left ventricular (LV) myocardium segmentation on all DCE-MRI timeframes, offering a solution to the challenges posed by the multidimensional nature of the data. This study aims to develop and assess an automated method for LV myocardial segmentation on DCE-MRI data of a local hospital. The study consists of retrospective DCE-MRI data from 55 subjects acquired at the local hospital using a 1.5T MRI scanner. The dataset included subjects with and without cardiac abnormalities. The timepoint for the reference frame (post-contrast LV myocardium) was identified using standard deviation across the temporal sequences. Iterative image registration of other temporal images with respect to this reference image was performed using Maxwell's demons algorithm. The registered stack was fed to the model built using the U-Net framework for predicting the LV myocardium at all timeframes of DCE-MRI. The mean and standard deviation of the dice similarity coefficient (DSC) for myocardial segmentation using pre-trained network Net_cine is 0.78 ± 0.04, and for the fine-tuned network Net_dyn which predicts mask on all timeframes individually, it is 0.78 ± 0.03. The DSC for Net_dyn ranged from 0.71 to 0.93. The average DSC achieved for the reference frame is 0.82 ± 0.06. The study proposed a fast and fully automated AI-assisted method to segment LV myocardium on all timeframes of DCE-MRI data. The method is robust, and its performance is independent of the intra-temporal sequence registration and can easily accommodate timeframes with potential registration errors.

Segmentation model of soft tissue sarcoma based on self-supervised learning.

Soft tissue sarcomas, similar in incidence to cervical and esophageal cancers, arise from various soft tissues like smooth muscle, fat, and fibrous tissue. Effective segmentation of sarcomas in imaging is crucial for accurate diagnosis. This study collected multi-modal MRI images from 45 patients with thigh soft tissue sarcoma, totaling 8,640 images. These images were annotated by clinicians to delineate the sarcoma regions, creating a comprehensive dataset. We developed a novel segmentation model based on the UNet framework, enhanced with residual networks and attention mechanisms for improved modality-specific information extraction. Additionally, self-supervised learning strategies were employed to optimize feature extraction capabilities of the encoders. The new model demonstrated superior segmentation performance when using multi-modal MRI images compared to single-modal inputs. The effectiveness of the model in utilizing the created dataset was validated through various experimental setups, confirming the enhanced ability to characterize tumor regions across different modalities. The integration of multi-modal MRI images and advanced machine learning techniques in our model significantly improves the segmentation of soft tissue sarcomas in thigh imaging. This advancement aids clinicians in better diagnosing and understanding the patient's condition, leveraging the strengths of different imaging modalities. Further studies could explore the application of these techniques to other types of soft tissue sarcomas and additional anatomical sites.

Marine oil spill detection and segmentation in SAR data with two steps Deep Learning framework

Marine oil spills pose significant ecological and economic threats worldwide, requiring effective decision-making tools. In this study, the optimal parameters, and configurations for Deep Learning models in oil spill classification and segmentation using Sentinel-1 SAR imagery were identified. First, a new Sentinel-1 image dataset was created. Ninety CNN configurations were explored for classification by varying the number of convolutional layers, filters, hidden layers, and neurons in each layer. For segmentation tasks, MLP and U-Net models were evaluated with variations in convolutional layers, filters, and incorporation of IoU and Focal Loss. The results indicated that a CNN model with six layers, 32 filters, and two hidden layers achieved 99 % classification accuracy. For segmentation, the U-Net model with more layers and filters using Focal Loss achieved 99 % accuracy and 96 % IoU. Therefore, a CNN and U-Net framework was proposed that achieves an overall accuracy of 95 % and an IoU of 90 %.

Synthetic CT generation based on CBCT using improved vision transformer CycleGAN

Cone-beam computed tomography (CBCT) is a crucial component of adaptive radiation therapy; however, it frequently encounters challenges such as artifacts and noise, significantly constraining its clinical utility. While CycleGAN is a widely employed method for CT image synthesis, it has notable limitations regarding the inadequate capture of global features. To tackle these challenges, we introduce a refined unsupervised learning model called improved vision transformer CycleGAN (IViT-CycleGAN). Firstly, we integrate a U-net framework that builds upon ViT. Next, we augment the feed-forward neural network by incorporating deep convolutional networks. Lastly, we enhance the stability of the model training process by introducing gradient penalty and integrating an additional loss term into the generator loss. The experiment demonstrates from multiple perspectives that our model-generated synthesizing CT(sCT) has significant advantages compared to other unsupervised learning models, thereby validating the clinical applicability and robustness of our model. In future clinical practice, our model has the potential to assist clinical practitioners in formulating precise radiotherapy plans.

Attention-Based Two-Branch Hybrid Fusion Network for Medical Image Segmentation

Accurate segmentation of medical images is vital for disease detection and treatment. Convolutional Neural Networks (CNN) and Transformer models are widely used in medical image segmentation due to their exceptional capabilities in image recognition and segmentation. However, CNNs often lack an understanding of the global context and may lose spatial details of the target, while Transformers struggle with local information processing, leading to reduced geometric detail of the target. To address these issues, this research presents a Global-Local Fusion network model (GLFUnet) based on the U-Net framework and attention mechanisms. The model employs a dual-branch network that utilizes ConvNeXt and Swin Transformer to simultaneously extract multi-level features from pathological images. It enhances ConvNeXt’s local feature extraction with spatial and global attention up-sampling modules, while improving Swin Transformer’s global context dependency with channel attention. The Attention Feature Fusion module and skip connections efficiently merge local detailed and global coarse features from CNN and Transformer branches at various scales. The fused features are then progressively restored to the original image resolution for pixel-level prediction. Comprehensive experiments on datasets of stomach and liver cancer demonstrate GLFUnet’s superior performance and adaptability in medical image segmentation, holding promise for clinical analysis and disease diagnosis.

A Novel Multi-Scale Feature Map Fusion for Oil Spill Detection of SAR Remote Sensing

The efficient and timely identification of oil spill areas is crucial for ocean environmental protection. Synthetic aperture radar (SAR) is widely used in oil spill detection due to its all-weather monitoring capability. Meanwhile, existing deep learning-based oil spill detection methods mainly rely on the classical U-Net framework and have achieved impressive results. However, SAR images exhibit high noise, blurry boundaries, and irregular shapes of target areas, as well as speckles and shadows, which lead to the loss of performance in existing algorithms. In this paper, we propose a novel network architecture to achieve more precise segmentation of oil spill areas by reintroducing rich semantic contextual information before obtaining the final segmentation mask. Specifically, the proposed architecture can re-fuse feature maps from different levels at the decoder end. We design a multi-convolutional layer (MCL) module to extract basic feature information from SAR images, and a feature extraction module (FEM) module further extracts and fuses feature maps generated by the U-Net decoder at different levels. Through these operations, the network can learn rich global and local contextual information, enable sufficient interaction of feature information at different stages, enhance the model’s contextual awareness, and improve its ability to recognize complex textures and blurry boundaries, thereby enhancing the segmentation accuracy of SAR images. Compared to many U-Net based segmentation networks, our method shows promising results and achieves state-of-the-art performance on multiple evaluation metrics.

Semantic Segmentation of Aerial Imagery Using U-Net with Self-Attention and Separable Convolutions

This research addresses the crucial task of improving accuracy in the semantic segmentation of aerial imagery, essential for applications such as urban planning and environmental monitoring. This study emphasizes the significance of maintaining the Intersection over Union (IOU) score as a metric and employs data augmentation with the Patchify library, using a patch size of 256, to effectively augment the dataset, which is subsequently split into training and testing sets. The core of this investigation lies in a novel architecture that combines a U-Net framework with self-attention mechanisms and separable convolutions. The introduction of self-attention mechanisms enhances the model’s understanding of image context, while separable convolutions expedite the training process, contributing to overall efficiency. The proposed model demonstrates a substantial accuracy improvement, surpassing the previous state-of-the-art Dense Plus U-Net, achieving an accuracy of 91% compared to the former’s 86%. Visual representations, including original patch images, original masked patches, and predicted patch masks, showcase the model’s proficiency in semantic segmentation, marking a significant advancement in aerial image analysis and underscoring the importance of innovative architectural elements for enhanced accuracy and efficiency in such tasks.

Breast cancer lesion segmentation based on co-learning feature fusion and Transformer

The PET/CT imaging technology combining positron emission tomography (PET) and computed tomography (CT) is the most advanced imaging examination method currently, and is mainly used for tumor screening, differential diagnosis of benign and malignant tumors, staging and grading. This paper proposes a method for breast cancer lesion segmentation based on PET/CT bimodal images, and designs a dual-path U-Net framework, which mainly includes three modules: encoder module, feature fusion module and decoder module. Among them, the encoder module uses traditional convolution for feature extraction of single mode image; The feature fusion module adopts collaborative learning feature fusion technology and uses Transformer to extract the global features of the fusion image; The decoder module mainly uses multi-layer perceptron to achieve lesion segmentation. This experiment uses actual clinical PET/CT data to evaluate the effectiveness of the algorithm. The experimental results show that the accuracy, recall and accuracy of breast cancer lesion segmentation are 95.67%, 97.58% and 96.16%, respectively, which are better than the baseline algorithm. Therefore, it proves the rationality of the single and bimodal feature extraction method combining convolution and Transformer in the experimental design of this article, and provides reference for feature extraction methods for tasks such as multimodal medical image segmentation or classification.

Pure Vision Transformer (CT-ViT) with Noise2Neighbors Interpolation for Low-Dose CT Image Denoising.

Convolutional neural networks (CNN) have been used for a wide variety of deep learning applications, especially in computer vision. For medical image processing, researchers have identified certain challenges associated with CNNs. These challenges encompass the generation of less informative features, limitations in capturing both high and low-frequency information within feature maps, and the computational cost incurred when enhancing receptive fields by deepening the network. Transformers have emerged as an approach aiming to address and overcome these specific limitations of CNNs in the context of medical image analysis. Preservation of all spatial details of medical images is necessary to ensure accurate patient diagnosis. Hence, this research introduced the use of a pure Vision Transformer (ViT) for a denoising artificial neural network for medical image processing specifically for low-dose computed tomography (LDCT) image denoising. The proposed model follows a U-Net framework that contains ViT modules with the integration of Noise2Neighbor (N2N) interpolation operation. Five different datasets containing LDCT and normal-dose CT (NDCT) image pairs were used to carry out this experiment. To test the efficacy of the proposed model, this experiment includes comparisons between the quantitative and visual results among CNN-based (BM3D, RED-CNN, DRL-E-MP), hybrid CNN-ViT-based (TED-Net), and the proposed pure ViT-based denoising model. The findings of this study showed that there is about 15-20% increase in SSIM and PSNR when using self-attention transformers than using the typical pure CNN. Visual results also showed improvements especially when it comes to showing fine structural details of CT images.