Architecture For Segmentation Research Articles

DMA‐Net: A dual branch encoder and multi‐scale cross attention fusion network for skin lesion segmentation

AbstractAutomatic segmentation of skin lesion is an important step in computer‐aided diagnosis. However, due to the significant variations in the size and shape of the lesion areas, as well as the low contrast with normal skin tissue, the boundaries are not clearly distinguishable, leading to a high possibility of incorrect segmentation. Therefore, this task is highly challenging. To overcome these difficulties, this paper proposes a medical image segmentation architecture named dual branch encoder and multi‐scale cross attention fusion network, which includes a dual‐branch encoder based on convolutional neural network and an improved channel‐enhanced Mamba to comprehensively extract local and global information from dermoscopy images. Additionally, to enhance the feature interaction and fusion of local and global information, a multi‐scale cross attention fusion module is adopted to cross‐merge features in different directions and at different scales, maximizing the advantages of the dual‐branch encoder and achieving precise segmentation of skin lesions. Extensive experiments are conducted on three public skin lesion datasets: ISIC‐2018, ISIC‐2017, and ISIC‐2016, to verify the effectiveness and superiority of the proposed method. The dice similarity coefficient scores on the three datasets reached 81.77%, 81.68% and 85.60%, respectively, surpassing most state‐of‐the‐art methods.

Identifying rice lodging based on semantic segmentation architecture optimization with UAV remote sensing imaging

Lodging, a prevalent issue during rice growth, detrimentally impacts both yield and quality. It also complicates the harvesting process, reducing the efficiency of mechanized collection. Existing monitoring methods, predominantly based on manual observation and satellite remote sensing, fall short in addressing the requirements of contemporary, efficient, and real-time agriculture. This research integrates image analysis techniques with advanced optimization algorithms to develop a semantic segmentation model specifically designed for detecting rice lodging in remote sensing images. The model, named MI-UConvNeXt, employs a ConvNeXt-based feature extraction network utilizing UNet architecture (UConvNeXt) and incorporates an improved multi-objective salp swarm algorithm with Latin hypercube sampling and an elite opposition-based learning strategy (ISSA-LE) to dynamically adjusting the number of UConvNeXt channels. MI-UConvNeXt achieves a balance between accuracy and complexity. Compared to seven other semantic segmentation models from the literature, MI-UConvNeXt exhibits enhanced performance, with a Pixel Accuracy (PA) of 95.59%, mean Pixel Accuracy (mPA) of 95.62%, and mean Intersection over Union (mIoU) of 91.91% on the validation set. This demonstrates the model’s superior accuracy, lower computational resource demands, and enhanced efficiency. By integrating deep learning with intelligent optimization algorithms, this study offers a novel and effective approach for monitoring crop lodging in agricultural production, providing robust technical support for the accurate extraction of crop phenotypic information.

Stroke‐Seg: A deep learning‐based framework for chinese stroke segmentation

AbstractChinese stroke segmentation is a crucial and challenging task for various downstream applications such as font generation, aesthetic evaluation etc. Conventional semantic segmentation techniques typically face difficulties in accurately segmenting strokes, as intersection regions in Chinese characters can belong to multiple strokes simultaneously, and these approaches often lack a holistic understanding of character composition. This paper proposes a character stroke segmentation framework named Stroke‐Seg that integrates with various semantic segmentation architectures, demonstrating adaptability to different backbone networks to tackle the above tasks. A multi‐label output strategy is proposed to effectively classify strokes in intersection areas, overcoming the limitations of traditional semantic segmentation approaches. Additionally, a prior knowledge vector is incorporated into the input layer to provide character‐specific information on stroke composition, enhancing the ability of the framework to precisely identify and segment strokes. The effectiveness of the proposed framework is demonstrated through evaluations of a comprehensive dataset (brush calligraphy stroke segmentation dataset). Evaluations show that the proposed framework significantly improves the capability of semantic segmentation networks, achieving a remarkable improvement of up to 19.9% in the true stroke rate, compared to traditional stroke segmentation techniques. The Stroke‐Seg framework integrated with TransUNet demonstrates its high performance with an impressive 98.2% true stroke rate. Furthermore, the proposed framework combined with FCN still achieves good performance while consuming the least amount of computational resources and memory, demonstrating its potential for lightweight design.

Application of a Modified SegFormer Architecture for Gleason Pattern Segmentation

A key element of morphological assessment in prostate cancer (PCa) diagnostics is the Gleason score, which is based on the architecture of tumor growth in biopsy or prostatectomy samples. During this process, biopsies from the prostate's pathological areas are evaluated to identify the most common and severe architectural patterns (referred to as Gleason patterns), each of which is assigned a score in the Gleason grading system. The resulting total score serves as a critical predictor for disease progression and metastasis. This research focuses on enhancing the efficiency of morphological examination in PCa by utilizing automatic segmentation of Gleason patterns in whole slide images (WSI) in *.svs format, with 40x magnification and an average spatial resolution of 0.258 microns per pixel. Segmentation was performed using a neural network with a SegFormer architecture, supplemented by an EdgeNeXt (small variant) subnet. The proposed solution successfully identified Gleason patterns of grades 3, 4, and 5. The best model performance for detecting combined Gleason patterns 3, 4, 5, and 4, 5 was achieved with input images sized 512x512 (tile size) and a batch size of 8, resulting in an F-score of 0.835 and 0.769, respectively.

Enhanced WGAN Model for Diagnosing Laryngeal Carcinoma

This study modifies the U-Net architecture for pixel-based segmentation to automatically classify lesions in laryngeal endoscopic images. The advanced U-Net incorporates five-level encoders and decoders, with an autoencoder layer to derive latent vectors representing the image characteristics. To enhance performance, a WGAN was implemented to address common issues such as mode collapse and gradient explosion found in traditional GANs. The dataset consisted of 8171 images labeled with polygons in seven colors. Evaluation metrics, including the F1 score and intersection over union, revealed that benign tumors were detected with lower accuracy compared to other lesions, while cancers achieved notably high accuracy. The model demonstrated an overall accuracy rate of 99%. This enhanced U-Net model shows strong potential in improving cancer detection, reducing diagnostic errors, and enhancing early diagnosis in medical applications.

Multi-Modal Pose Representations for 6-DOF Object Tracking

Pose estimation methods for robotics should return a distribution of poses rather than just a single pose estimate. Motivated by this, in this work we investigate multi-modal pose representations for reliable 6-DoF object tracking. A neural network architecture for simultaneous object segmentation and estimation of fiducial points of the object on RGB images is proposed. Given a priori probability distribution of object poses a particle filter is employed to estimate the posterior probability distribution of object poses. An advanced observation model relying on matching the projected 3D model with the segmented object and a distance transform-based object representation is used to weight samples representing the probability distribution. Afterwards, the object pose determined by the PnP algorithm is included in the probability distribution via replacing a particle with the smallest weight. Next, a k-means++ algorithm is executed to determine modes in a multi-modal probability distribution. A multi-swarm particle swarm optimization is then executed to determine the finest modes in the probability distribution. A subset of particles for final pose optimization is found in a multi-criteria analysis using the TOPSIS algorithm. They are verified using conflicting criteria that are determined on the basis of object keypoints, segmented object, and the distance transform. On the challenging YCB-Video dataset it outperforms recent algorithms for both object pose estimation and object pose tracking.

Real-time robust liver and gallbladder segmentation during laparoscopic cholecystectomy using convolutional neural networks: an analysis

Aim: Images in different laparoscopic cholecystectomy datasets are acquired using various camera models, parameters, and settings, with the annotation methods varying by institution. These factors result in inconsistent inference performance of the network model. This study aims to identify the optimal network model architecture for liver and gallbladder segmentation from several options. Then, the performance and robustness of the optimal network model are evaluated using an independent dataset that is not included in the training. Methods: The public dataset, CholecSeg8k, was utilized as the input for the network model training, validation, and testing. A local private dataset from KPJ Damansara Hospital, Selangor, Malaysia, was used for testing purposes only. For the implementation of liver and gallbladder segmentation, segmentation models, a public Python library was employed. Results: Among the experiments, highly accurate liver and gallbladder segmentation results were achieved using the feature pyramid network (FPN) architecture as the network model, with the Inception-ResNet-v2 architecture as the network backbone. The best-trained network model resulted in a loss of 0.070955, a mean intersection over union (IoU) score of 0.95896, and a mean F1-score of 0.9773 on the test set. However, visualized results for the private dataset contained considerable false-negative areas. Conclusion: The proposed automated technique has the potential to serve as an alternative to the conventional indocyanine green injection along with near-infrared fluorescence imaging (ICG-NIRF)-based method for liver and gallbladder segmentation during laparoscopic cholecystectomy. Future work will focus on enhancing the results of the private dataset. Additionally, a surgeon-assistant robotic arm that will use the liver and gallbladder segmentation results for camera steering will be analyzed.

HyFormer: a hybrid transformer-CNN architecture for retinal OCT image segmentation

Optical coherence tomography (OCT) has become the leading imaging technique in diagnosing and treatment planning for retinal diseases. Retinal OCT image segmentation involves extracting lesions and/or tissue structures to aid in the decisions of ophthalmologists, and multi-class segmentation is commonly needed. As the target regions often spread widely inside the retina, and the intensities and locations of different categories can be close, good segmentation networks must possess both global modeling capabilities and the ability to capture fine details. To address the challenge in capturing both global and local features simultaneously, we propose HyFormer, an efficient, lightweight, and robust hybrid network architecture. The proposed architecture features parallel Transformer and convolutional encoders for independent feature capture. A multi-scale gated attention block and a group positional embedding block are introduced within the Transformer encoder to enhance feature extraction. Feature integration is achieved in the decoder composed of the proposed three-path fusion modules. A class activation map-based cross-entropy loss function is also proposed to improve segmentation results. Evaluations are performed on a private dataset with myopic traction maculopathy lesions and the public AROI dataset for retinal layer and lesion segmentation with age-related degeneration. The results demonstrate HyFormer's superior segmentation performance and robustness compared to existing methods, showing promise for accurate and efficient OCT image segmentation. .

U-Net architecture for high-resolution thermal image segmentation: Enhancing CFD models in smart HVAC design

This paper presents an enhanced U-Net architecture for accurately segmenting high-resolution thermal images to facilitate the design of intelligent Heating, Ventilation, and Air Conditioning (HVAC) systems using Computational Fluid Dynamics (CFD) models. Indoor thermal comfort for occupants will improve under the same energy load by imposing comfort demands. Segmented data is fed into CFD simulations using a U-Net variant for thermal imagery. Integrating the segmented thermal data into the CFD simulations increases the model’s accuracy by 15 % for segmentation and 20% for prediction when compared to a traditional U-Net architecture. Experimental findings demonstrate the potential to achieve energy savings of up to 25 % in the simulated scenario by using the trained and validated model on a dataset of 1,500 high-resolution thermal images of various building styles. Compared to existing methods, the proposed method saves 30% in total simulation time for complex building models. Deep learning enables building the next generation of HVAC systems by pushing the boundaries of energy-efficient and environmentally friendly buildings. This study shows that advanced computer vision can be leveraged to create more innovative and sustainable built environments, leading to drastic improvements in energy efficiency.

Transformers-based architectures for stroke segmentation: a review

Stroke remains a significant global health concern, necessitating precise and efficient diagnostic tools for timely intervention and improved patient outcomes. The emergence of deep learning methodologies has transformed the landscape of medical image analysis. Recently, Transformers, initially designed for natural language processing, have exhibited remarkable capabilities in various computer vision applications, including medical image analysis. This comprehensive review aims to provide an in-depth exploration of the cutting-edge Transformer-based architectures applied in the context of stroke segmentation. It commences with an exploration of stroke pathology, imaging modalities, and the challenges associated with accurate diagnosis and segmentation. Subsequently, the review delves into the fundamental ideas of Transformers, offering detailed insights into their architectural intricacies and the underlying mechanisms that empower them to effectively capture complex spatial information within medical images. The existing literature is systematically categorized and analyzed, discussing various approaches that leverage Transformers for stroke segmentation. A critical assessment is provided, highlighting the strengths and limitations of these methods, including considerations of performance and computational efficiency. Additionally, this review explores potential avenues for future research and development.

Neural Memory State Space Models for Medical Image Segmentation.

With the rapid advancement of deep learning, computer-aided diagnosis and treatment have become crucial in medicine. UNet is a widely used architecture for medical image segmentation, and various methods for improving UNet have been extensively explored. One popular approach is incorporating transformers, though their quadratic computational complexity poses challenges. Recently, State-Space Models (SSMs), exemplified by Mamba, have gained significant attention as a promising alternative due to their linear computational complexity. Another approach, neural memory Ordinary Differential Equations (nmODEs), exhibits similar principles and achieves good results. In this paper, we explore the respective strengths and weaknesses of nmODEs and SSMs and propose a novel architecture, the nmSSM decoder, which combines the advantages of both approaches. This architecture possesses powerful nonlinear representation capabilities while retaining the ability to preserve input and process global information. We construct nmSSM-UNet using the nmSSM decoder and conduct comprehensive experiments on the PH2, ISIC2018, and BU-COCO datasets to validate its effectiveness in medical image segmentation. The results demonstrate the promising application value of nmSSM-UNet. Additionally, we conducted ablation experiments to verify the effectiveness of our proposed improvements on SSMs and nmODEs.

NA-segformer: A multi-level transformer model based on neighborhood attention for colonoscopic polyp segmentation

In various countries worldwide, the incidence of colon cancer-related deaths has been on the rise in recent years. Early detection of symptoms and identification of intestinal polyps are crucial for improving the cure rate of colon cancer patients. Automated computer-aided diagnosis (CAD) has emerged as a solution to the low efficiency of traditional methods relying on manual diagnosis by physicians. Deep learning is the latest direction of CAD development and has shown promise for colonoscopic polyp segmentation. In this paper, we present a multi-level encoder-decoder architecture for polyp segmentation based on the Transformer architecture, termed NA-SegFormer. To improve the performance of existing Transformer-based segmentation algorithms for edge segmentation on colon polyps, we propose a patch merging module with a neighbor attention mechanism based on overlap patch merging. Since colon tract polyps vary greatly in size and different datasets have different sample sizes, we used a unified focal loss to solve the problem of category imbalance in colon tract polyp data. To assess the effectiveness of our proposed method, we utilized video capsule endoscopy and typical colonoscopy polyp datasets, as well as a dataset containing surgical equipment. On the datasets Kvasir-SEG, Kvasir-Instrument and KvasirCapsule-SEG, the Dice score of our proposed model reached 94.30%, 94.59% and 82.73%, with an accuracy of 98.26%, 99.02% and 81.84% respectively. The proposed method achieved inference speed with an Frame-per-second (FPS) of 125.01. The results demonstrated that our suggested model effectively segmented polyps better than several well-known and latest models. In addition, the proposed method has advantages in trade-off between inference speed and accuracy, and it will be of great significance to real-time colonoscopic polyp segmentation. The code is available at https://github.com/promisedong/NAFormer.

Inception UNet architecture for breast tumor segmentation and detection using hybrid deep learning approach

Inception UNet architecture for breast tumor segmentation and detection using hybrid deep learning approach

HEDN: multi-oriented hierarchical extraction and dual-frequency decoupling network for 3D medical image segmentation.

Previous 3D encoder-decoder segmentation architectures struggled with fine-grained feature decomposition, resulting in unclear feature hierarchies when fused across layers. Furthermore, the blurred nature of contour boundaries in medical imaging limits the focus on high-frequency contour features. To address these challenges, we propose a Multi-oriented Hierarchical Extraction and Dual-frequency Decoupling Network (HEDN), which consists of three modules: Encoder-Decoder Module (E-DM), Multi-oriented Hierarchical Extraction Module (Multi-HEM), and Dual-frequency Decoupling Module (Dual-DM). The E-DM performs the basic encoding and decoding tasks, while Multi-HEM decomposes and fuses spatial and slice-level features in 3D, enriching the feature hierarchy by weighting them through 3D fusion. Dual-DM separates high-frequency features from the reconstructed network using self-supervision. Finally, the self-supervised high-frequency features separated by Dual-DM are inserted into the process following Multi-HEM, enhancing interactions and complementarities between contour features and hierarchical features, thereby mutually reinforcing both aspects. On the Synapse dataset, HEDN outperforms existing methods, boosting Dice Similarity Score (DSC) by 1.38% and decreasing 95% Hausdorff Distance (HD95) by 1.03 mm. Likewise, on the Automatic Cardiac Diagnosis Challenge (ACDC) dataset, HEDN achieves 0.5% performance gains across all categories.

Automated Deep Learning-Based Finger Joint Segmentation in 3-D Ultrasound Images With Limited Dataset.

Ultrasound imaging has shown promise in assessing synovium inflammation associated early stages of rheumatoid arthritis (RA). The precise identification of the synovium and the quantification of inflammation-specific imaging biomarkers is a crucial aspect of accurately quantifying and grading RA. In this study, a deep learning-based approach is presented that automates the segmentation of the synovium in ultrasound images of finger joints affected by RA. Two convolutional neural network architectures for image segmentation were trained and validated in a limited number of 2-D images, extracted from N = 18 3-D ultrasound volumes acquired from N = 9 RA patients, with sparse ground truth annotations of the synovium. Various augmentation strategies were employed to enhance the diversity and size of the training dataset. The utilization of geometric and noise augmentation transforms resulted in the highest dice score (0.768 0.040, N = 6), as determined via six-fold cross-validation. In addition, the segmentation model is used to generate dense 3-D segmentation maps in the ultrasound volumes, based on the available sparse annotations. The developed technique shows promise in facilitating more efficient and standardized workflow for RA screening using ultrasound imaging.

TBConvL-Net: A hybrid deep learning architecture for robust medical image segmentation

Deep learning has shown great potential for automated medical image segmentation to improve the precision and speed of disease diagnostics. However, the task presents significant difficulties due to variations in the scale, shape, texture, and contrast of the pathologies. Traditional convolutional neural network (CNN) models have certain limitations when it comes to effectively modelling multiscale context information and facilitating information interaction between skip connections across levels. To overcome these limitations, a novel deep learning architecture is introduced for medical image segmentation, which takes advantage of CNNs and vision transformers. Our proposed model, named TBConvL-Net, involves a hybrid network that combines the local features of a CNN encoder–decoder architecture with long-range and temporal dependencies using biconvolutional long-short-term memory (LSTM) networks and vision transformers (ViT). This enables the model to capture contextual channel relationships in the data and account for the uncertainty of the segmentation over time. Additionally, we introduce a novel composite loss function that considers both segmentation robustness and boundary agreement of the predicted output with the gold standard. Our proposed model shows consistent improvement over the state of the art on ten publicly available datasets of seven different medical imaging modalities.

Analysis and comparison of retinal vascular parameters under different glucose metabolic status based on deep learning.

To develop a deep learning-based model for automatic retinal vascular segmentation, analyzing and comparing parameters under diverse glucose metabolic status (normal, prediabetes, diabetes) and to assess the potential of artificial intelligence (AI) in image segmentation and retinal vascular parameters for predicting prediabetes and diabetes. Retinal fundus photos from 200 normal individuals, 200 prediabetic patients, and 200 diabetic patients (600 eyes in total) were used. The U-Net network served as the foundational architecture for retinal artery-vein segmentation. An automatic segmentation and evaluation system for retinal vascular parameters was trained, encompassing 26 parameters. Significant differences were found in retinal vascular parameters across normal, prediabetes, and diabetes groups, including artery diameter (P=0.008), fractal dimension (P=0.000), vein curvature (P=0.003), C-zone artery branching vessel count (P=0.049), C-zone vein branching vessel count (P=0.041), artery branching angle (P=0.005), vein branching angle (P=0.001), artery angle asymmetry degree (P=0.003), vessel length density (P=0.000), and vessel area density (P=0.000), totaling 10 parameters. The deep learning-based model facilitates retinal vascular parameter identification and quantification, revealing significant differences. These parameters exhibit potential as biomarkers for prediabetes and diabetes.

Dmg2Former-AR: Vision Transformers with Adaptive Rescaling for High-Resolution Structural Visual Inspection.

Developments in drones and imaging hardware technology have opened up countless possibilities for enhancing structural condition assessments and visual inspections. However, processing the inspection images requires considerable work hours, leading to delays in the assessment process. This study presents a semantic segmentation architecture that integrates vision transformers with Laplacian pyramid scaling networks, enabling rapid and accurate pixel-level damage detection. Unlike conventional methods that often lose critical details through resampling or cropping high-resolution images, our approach preserves essential inspection-related information such as microcracks and edges using non-uniform image rescaling networks. This innovation allows for detailed damage identification of high-resolution images while significantly reducing the computational demands. Our main contributions in this study are: (1) proposing two rescaling networks that together allow for processing high-resolution images while significantly reducing the computational demands; and (2) proposing Dmg2Former, a low-resolution segmentation network with a Swin Transformer backbone that leverages the saved computational resources to produce detailed visual inspection masks. We validate our method through a series of experiments on publicly available visual inspection datasets, addressing various tasks such as crack detection and material identification. Finally, we examine the computational efficiency of the adaptive rescalers in terms of multiply-accumulate operations and GPU-memory requirements.

Enhancing pixel-level crack segmentation with visual mamba and convolutional networks

Computer vision-based semantic segmentation methods are currently the most widely used for automated detection of structural cracks in buildings and pavements. However, these methods face persistent challenges in detecting fine cracks with small widths and in distinguishing cracks from background stains. This paper addresses these issues by introducing MambaCrackNet, a new network architecture for pixel-level crack segmentation. MambaCrackNet incorporates residual visual Mamba blocks and integrates visual Mamba and convolutional neural network-based segmentation techniques. This approach effectively enhances the detection of fine cracks, reduces misdetections of background stains, and remains robust to variations in patch size and training sample sizes, making it highly practical for engineering applications. On two open access crack datasets, MambaCrackNet outperformed mainstream crack segmentation models, achieving MIoU scores of 0.8939 and 0.8560 and F1-scores of 0.8817 and 0.8412.

Enhanced damage segmentation in RC components using pyramid Haar wavelet downsampling and attention U-net

Damage identification in post-earthquake reinforced concrete (RC) structures based on semantic segmentation has been recognized as a promising approach for rapid and non-contact damage localization and quantification. In damage segmentation tasks, damage regions are often set against complex backgrounds, featuring irregular geometric boundaries and intricate textures, posing significant challenges to model segmentation performance. Additionally, the absence of public datasets exacerbates these challenges, hindering advancements in this field. In this paper, a pyramid Haar wavelet downsampling attention UNet (PHA-UNet) semantic segmentation network is proposed, and a database containing 1400 images of damaged RC components (PEDRC-Dataset) with pixel-level annotations is established. In the proposed PHA-UNet, attention mechanisms, multiscale feature fusion, Haar wavelet downsampling, and transfer learning are introduced to address above challenges. Finally, the proposed PHA-UNet is compared with four existing image segmentation architectures on both the Cityspace and the PEDRC-Dataset.