Architecture For Segmentation Research Articles

Analysis of the effectiveness of using U-net architecture for classification and segmentation of glioma in MRI images

The purpose of the research is to analyze the efficiency of the U-net neural network architecture in decision support systems for glioma diagnostics and segmentation of brain areas affected by it on MRI images.Methods. To conduct experimental studies, a training dataset was generated and the data was normalized. A software implementation of the U-Net neural network architecture was performed using the Keras framework in the Python programming language. The neural network model was trained.Results. A series of experiments were conducted, during which error and classification matrices were obtained, the efficiency of classification of the trained neural network model for the "Tumor" and "No tumor" classes was assessed using metrics such as Recall, Precision and F1-measure, and the quality of segmentation of glioma-affected areas on the test data set was assessed. The quality of segmentation was assessed using the IoU metric, which reflects the ratio of the areas of the bounding boxes and is used to assess the accuracy of the spatial correspondence of the predicted segmented areas highlighted on the masks. Based on the results of testing the neural network model in solving the problem of segmenting brain areas affected by glioma, the average value of the IoU metric was 0.812, which is an acceptable result.Conclusion. The testing results showed that the neural network model based on the U-net architecture is able to effectively diagnose the presence of glioma with acceptable values of the classification and segmentation quality metrics, which indicates the possibility of using this neural network model in medical decision support systems for glioma diagnostics, as well as its segmentation on MRI images. However, it is advisable to refine this neural network model to reduce the number of false negative classification results, which is critically important in medical diagnostics.

ScSEETV‐Net: Spatial and Channel Squeeze‐Excitation and Edge Attention Guidance V‐Shaped Network for Skin Lesion Segmentation

Early detection of skin cancer ensures the survival of many cases. There are still challenges in segmenting dermoscopic skin lesion images. Artifacts in the lesion images, such as various dirt, hairs, low contrast, and unclear boundaries, are challenges that affect segmentation accuracy. Convolutional neural networks have brought success in skin lesion segmentation. U‐shaped and V‐shaped deep learning‐based segmentation architectures learn boundary information in the first layers. However, this information becomes weaker in the following layers. Herein, the Edge‐aTtention module is added to the V‐Net architecture to move edge information to the last layer, and the spatial and channel squeeze‐excitation module is added to emphasize high‐level features by recalibrating the channel information to learn lesion boundaries better. The scSEETV‐Net is supported by fusing the binary cross‐entropy, which calculates the loss on a pixel‐based, and the focal Twersky loss function, which has significant success in class imbalances. The proposed architecture achieves 0.9212 Jaccard and 0.9552 Dice in the ISIC2016 dataset, 0.8273 Jaccard and 0.8949 Dice in the ISIC2017 dataset, and 0.8070 Jaccard and 0.8831 Dice in the ISIC2018 dataset. Comparative analyses show that the proposed methodology outperforms the state‐of‐the‐art techniques in the literature.

A novel adjunctive diagnostic method for bone cancer: Osteosarcoma cell segmentation based on Twin Swin Transformer with multi-scale feature fusion

BackgroundOsteosarcoma, the most common primary bone tumor originating from osteoblasts, poses a significant challenge in medical practice, particularly among adolescents. Conventional diagnostic methods heavily rely on manual analysis of magnetic resonance imaging (MRI) scans, which often fall short in providing accurate and timely diagnosis. This underscores the critical need for advancements in medical imaging technologies to improve the detection and characterization of osteosarcoma. MethodsIn this study, we sought to address the limitations of current diagnostic approaches by leveraging Hoechst-stained images of osteosarcoma cells obtained via fluorescence microscopy. Our primary objective was to enhance the segmentation of osteosarcoma cells, a crucial step in precise diagnosis and treatment planning. Recognizing the shortcomings of existing feature extraction networks in capturing detailed cellular structures, we propose a novel approach utilizing a twin swin transformer architecture for osteosarcoma cell segmentation, with a focus on multi-scale feature fusion. ResultsThe experimental findings demonstrate the effectiveness of the proposed Twin Swin Transformer with multi-scale feature fusion in significantly improving osteosarcoma cell segmentation. Compared to conventional techniques, our method achieves superior segmentation performance, highlighting its potential utility in clinical settings. ConclusionThe development of our Twin Swin Transformer with multi-scale feature fusion method represents a significant advancement in medical imaging technology, particularly in the field of osteosarcoma diagnosis. By harnessing advanced computational techniques and leveraging high-resolution imaging data, our approach offers enhanced accuracy and efficiency in osteosarcoma cell segmentation, ultimately facilitating better patient care and clinical decision-making.

Point cloud segmentation neural network with same-type point cloud assistance

This paper proposes neural network architectures for point cloud segmentation, which leverage prior knowledge derived from same-type point clouds. The approach involves concurrent processing of two point clouds: a target point cloud necessitating segmentation and a labeled same-type point cloud. The labeled point cloud provides preliminary labeling information, assisting in segmenting the target point cloud. A feature combination module is proposed to identify and combine corresponding features across the point clouds. The module augments the feature representation of the target cloud and improves its capacity for object discrimination. Experiments on the ShapeNetPart and S3DIS datasets demonstrate that when integrated into classical network architectures, the proposed approach can achieve improved segmentation performance over the corresponding networks, significantly in some of them.

Real-time segmentation and classification of whole-slide images for tumor biomarker scoring

Histopathology image segmentation and classification are essential for diagnosing and treating breast cancer. This study introduced a highly accurate segmentation and classification for histopathology images using a single architecture. We utilized the famous segmentation architectures, SegNet and U-Net, and modified the decoder to attach ResNet, VGG and DenseNet to perform classification tasks. These hybrid models are integrated with Stardist as the backbone, and implemented in a real-time pathologist workflow with a graphical user interface. These models were trained and tested offline using the ER-IHC-stained private and H&E-stained public datasets (MoNuSeg). For real-time evaluation, the proposed model was evaluated using PR-IHC-stained glass slides. It achieved the highest segmentation pixel-based F1-score of 0.902 and 0.903 for private and public datasets respectively, and a classification-based F1-score of 0.833 for private dataset. The experiment shows the robustness of our method where a model trained on ER-IHC dataset able to perform well on real-time microscopy of PR-IHC slides on both 20x and 40x magnification. This will help the pathologists with a quick decision-making process.

Demystifying the effect of receptive field size in U-Net models for medical image segmentation.

Medical image segmentation is a critical task in healthcare applications, and U-Nets have demonstrated promising results in this domain. We delve into the understudied aspect of receptive field (RF) size and its impact on the U-Net and attention U-Net architectures used for medical imaging segmentation. We explore several critical elements including the relationship among RF size, characteristics of the region of interest, and model performance, as well as the balance between RF size and computational costs for U-Net and attention U-Net methods for different datasets. We also propose a mathematical notation for representing the theoretical receptive field (TRF) of a given layer in a network and propose two new metrics, namely, the effective receptive field (ERF) rate and the object rate, to quantify the fraction of significantly contributing pixels within the ERF against the TRF area and assessing the relative size of the segmentation object compared with the TRF size, respectively. The results demonstrate that there exists an optimal TRF size that successfully strikes a balance between capturing a wider global context and maintaining computational efficiency, thereby optimizing model performance. Interestingly, a distinct correlation is observed between the data complexity and the required TRF size; segmentation based solely on contrast achieved peak performance even with smaller TRF sizes, whereas more complex segmentation tasks necessitated larger TRFs. Attention U-Net models consistently outperformed their U-Net counterparts, highlighting the value of attention mechanisms regardless of TRF size. These insights present an invaluable resource for developing more efficient U-Net-based architectures for medical imaging and pave the way for future exploration of other segmentation architectures. A tool is also developed, which calculates the TRF for a U-Net (and attention U-Net) model and also suggests an appropriate TRF size for a given model and dataset.

DMA‐Net: A dual branch encoder and multi‐scale cross attention fusion network for skin lesion segmentation

AbstractAutomatic segmentation of skin lesion is an important step in computer‐aided diagnosis. However, due to the significant variations in the size and shape of the lesion areas, as well as the low contrast with normal skin tissue, the boundaries are not clearly distinguishable, leading to a high possibility of incorrect segmentation. Therefore, this task is highly challenging. To overcome these difficulties, this paper proposes a medical image segmentation architecture named dual branch encoder and multi‐scale cross attention fusion network, which includes a dual‐branch encoder based on convolutional neural network and an improved channel‐enhanced Mamba to comprehensively extract local and global information from dermoscopy images. Additionally, to enhance the feature interaction and fusion of local and global information, a multi‐scale cross attention fusion module is adopted to cross‐merge features in different directions and at different scales, maximizing the advantages of the dual‐branch encoder and achieving precise segmentation of skin lesions. Extensive experiments are conducted on three public skin lesion datasets: ISIC‐2018, ISIC‐2017, and ISIC‐2016, to verify the effectiveness and superiority of the proposed method. The dice similarity coefficient scores on the three datasets reached 81.77%, 81.68% and 85.60%, respectively, surpassing most state‐of‐the‐art methods.

Identifying rice lodging based on semantic segmentation architecture optimization with UAV remote sensing imaging

Lodging, a prevalent issue during rice growth, detrimentally impacts both yield and quality. It also complicates the harvesting process, reducing the efficiency of mechanized collection. Existing monitoring methods, predominantly based on manual observation and satellite remote sensing, fall short in addressing the requirements of contemporary, efficient, and real-time agriculture. This research integrates image analysis techniques with advanced optimization algorithms to develop a semantic segmentation model specifically designed for detecting rice lodging in remote sensing images. The model, named MI-UConvNeXt, employs a ConvNeXt-based feature extraction network utilizing UNet architecture (UConvNeXt) and incorporates an improved multi-objective salp swarm algorithm with Latin hypercube sampling and an elite opposition-based learning strategy (ISSA-LE) to dynamically adjusting the number of UConvNeXt channels. MI-UConvNeXt achieves a balance between accuracy and complexity. Compared to seven other semantic segmentation models from the literature, MI-UConvNeXt exhibits enhanced performance, with a Pixel Accuracy (PA) of 95.59%, mean Pixel Accuracy (mPA) of 95.62%, and mean Intersection over Union (mIoU) of 91.91% on the validation set. This demonstrates the model’s superior accuracy, lower computational resource demands, and enhanced efficiency. By integrating deep learning with intelligent optimization algorithms, this study offers a novel and effective approach for monitoring crop lodging in agricultural production, providing robust technical support for the accurate extraction of crop phenotypic information.

Stroke‐Seg: A deep learning‐based framework for chinese stroke segmentation

AbstractChinese stroke segmentation is a crucial and challenging task for various downstream applications such as font generation, aesthetic evaluation etc. Conventional semantic segmentation techniques typically face difficulties in accurately segmenting strokes, as intersection regions in Chinese characters can belong to multiple strokes simultaneously, and these approaches often lack a holistic understanding of character composition. This paper proposes a character stroke segmentation framework named Stroke‐Seg that integrates with various semantic segmentation architectures, demonstrating adaptability to different backbone networks to tackle the above tasks. A multi‐label output strategy is proposed to effectively classify strokes in intersection areas, overcoming the limitations of traditional semantic segmentation approaches. Additionally, a prior knowledge vector is incorporated into the input layer to provide character‐specific information on stroke composition, enhancing the ability of the framework to precisely identify and segment strokes. The effectiveness of the proposed framework is demonstrated through evaluations of a comprehensive dataset (brush calligraphy stroke segmentation dataset). Evaluations show that the proposed framework significantly improves the capability of semantic segmentation networks, achieving a remarkable improvement of up to 19.9% in the true stroke rate, compared to traditional stroke segmentation techniques. The Stroke‐Seg framework integrated with TransUNet demonstrates its high performance with an impressive 98.2% true stroke rate. Furthermore, the proposed framework combined with FCN still achieves good performance while consuming the least amount of computational resources and memory, demonstrating its potential for lightweight design.

Application of a Modified SegFormer Architecture for Gleason Pattern Segmentation

A key element of morphological assessment in prostate cancer (PCa) diagnostics is the Gleason score, which is based on the architecture of tumor growth in biopsy or prostatectomy samples. During this process, biopsies from the prostate's pathological areas are evaluated to identify the most common and severe architectural patterns (referred to as Gleason patterns), each of which is assigned a score in the Gleason grading system. The resulting total score serves as a critical predictor for disease progression and metastasis. This research focuses on enhancing the efficiency of morphological examination in PCa by utilizing automatic segmentation of Gleason patterns in whole slide images (WSI) in *.svs format, with 40x magnification and an average spatial resolution of 0.258 microns per pixel. Segmentation was performed using a neural network with a SegFormer architecture, supplemented by an EdgeNeXt (small variant) subnet. The proposed solution successfully identified Gleason patterns of grades 3, 4, and 5. The best model performance for detecting combined Gleason patterns 3, 4, 5, and 4, 5 was achieved with input images sized 512x512 (tile size) and a batch size of 8, resulting in an F-score of 0.835 and 0.769, respectively.

Enhanced WGAN Model for Diagnosing Laryngeal Carcinoma

This study modifies the U-Net architecture for pixel-based segmentation to automatically classify lesions in laryngeal endoscopic images. The advanced U-Net incorporates five-level encoders and decoders, with an autoencoder layer to derive latent vectors representing the image characteristics. To enhance performance, a WGAN was implemented to address common issues such as mode collapse and gradient explosion found in traditional GANs. The dataset consisted of 8171 images labeled with polygons in seven colors. Evaluation metrics, including the F1 score and intersection over union, revealed that benign tumors were detected with lower accuracy compared to other lesions, while cancers achieved notably high accuracy. The model demonstrated an overall accuracy rate of 99%. This enhanced U-Net model shows strong potential in improving cancer detection, reducing diagnostic errors, and enhancing early diagnosis in medical applications.

Multi-Modal Pose Representations for 6-DOF Object Tracking

Pose estimation methods for robotics should return a distribution of poses rather than just a single pose estimate. Motivated by this, in this work we investigate multi-modal pose representations for reliable 6-DoF object tracking. A neural network architecture for simultaneous object segmentation and estimation of fiducial points of the object on RGB images is proposed. Given a priori probability distribution of object poses a particle filter is employed to estimate the posterior probability distribution of object poses. An advanced observation model relying on matching the projected 3D model with the segmented object and a distance transform-based object representation is used to weight samples representing the probability distribution. Afterwards, the object pose determined by the PnP algorithm is included in the probability distribution via replacing a particle with the smallest weight. Next, a k-means++ algorithm is executed to determine modes in a multi-modal probability distribution. A multi-swarm particle swarm optimization is then executed to determine the finest modes in the probability distribution. A subset of particles for final pose optimization is found in a multi-criteria analysis using the TOPSIS algorithm. They are verified using conflicting criteria that are determined on the basis of object keypoints, segmented object, and the distance transform. On the challenging YCB-Video dataset it outperforms recent algorithms for both object pose estimation and object pose tracking.

Real-time robust liver and gallbladder segmentation during laparoscopic cholecystectomy using convolutional neural networks: an analysis

Aim: Images in different laparoscopic cholecystectomy datasets are acquired using various camera models, parameters, and settings, with the annotation methods varying by institution. These factors result in inconsistent inference performance of the network model. This study aims to identify the optimal network model architecture for liver and gallbladder segmentation from several options. Then, the performance and robustness of the optimal network model are evaluated using an independent dataset that is not included in the training. Methods: The public dataset, CholecSeg8k, was utilized as the input for the network model training, validation, and testing. A local private dataset from KPJ Damansara Hospital, Selangor, Malaysia, was used for testing purposes only. For the implementation of liver and gallbladder segmentation, segmentation models, a public Python library was employed. Results: Among the experiments, highly accurate liver and gallbladder segmentation results were achieved using the feature pyramid network (FPN) architecture as the network model, with the Inception-ResNet-v2 architecture as the network backbone. The best-trained network model resulted in a loss of 0.070955, a mean intersection over union (IoU) score of 0.95896, and a mean F1-score of 0.9773 on the test set. However, visualized results for the private dataset contained considerable false-negative areas. Conclusion: The proposed automated technique has the potential to serve as an alternative to the conventional indocyanine green injection along with near-infrared fluorescence imaging (ICG-NIRF)-based method for liver and gallbladder segmentation during laparoscopic cholecystectomy. Future work will focus on enhancing the results of the private dataset. Additionally, a surgeon-assistant robotic arm that will use the liver and gallbladder segmentation results for camera steering will be analyzed.

HyFormer: a hybrid transformer-CNN architecture for retinal OCT image segmentation

Optical coherence tomography (OCT) has become the leading imaging technique in diagnosing and treatment planning for retinal diseases. Retinal OCT image segmentation involves extracting lesions and/or tissue structures to aid in the decisions of ophthalmologists, and multi-class segmentation is commonly needed. As the target regions often spread widely inside the retina, and the intensities and locations of different categories can be close, good segmentation networks must possess both global modeling capabilities and the ability to capture fine details. To address the challenge in capturing both global and local features simultaneously, we propose HyFormer, an efficient, lightweight, and robust hybrid network architecture. The proposed architecture features parallel Transformer and convolutional encoders for independent feature capture. A multi-scale gated attention block and a group positional embedding block are introduced within the Transformer encoder to enhance feature extraction. Feature integration is achieved in the decoder composed of the proposed three-path fusion modules. A class activation map-based cross-entropy loss function is also proposed to improve segmentation results. Evaluations are performed on a private dataset with myopic traction maculopathy lesions and the public AROI dataset for retinal layer and lesion segmentation with age-related degeneration. The results demonstrate HyFormer's superior segmentation performance and robustness compared to existing methods, showing promise for accurate and efficient OCT image segmentation. .

U-Net architecture for high-resolution thermal image segmentation: Enhancing CFD models in smart HVAC design

This paper presents an enhanced U-Net architecture for accurately segmenting high-resolution thermal images to facilitate the design of intelligent Heating, Ventilation, and Air Conditioning (HVAC) systems using Computational Fluid Dynamics (CFD) models. Indoor thermal comfort for occupants will improve under the same energy load by imposing comfort demands. Segmented data is fed into CFD simulations using a U-Net variant for thermal imagery. Integrating the segmented thermal data into the CFD simulations increases the model’s accuracy by 15 % for segmentation and 20% for prediction when compared to a traditional U-Net architecture. Experimental findings demonstrate the potential to achieve energy savings of up to 25 % in the simulated scenario by using the trained and validated model on a dataset of 1,500 high-resolution thermal images of various building styles. Compared to existing methods, the proposed method saves 30% in total simulation time for complex building models. Deep learning enables building the next generation of HVAC systems by pushing the boundaries of energy-efficient and environmentally friendly buildings. This study shows that advanced computer vision can be leveraged to create more innovative and sustainable built environments, leading to drastic improvements in energy efficiency.

Transformers-based architectures for stroke segmentation: a review

Stroke remains a significant global health concern, necessitating precise and efficient diagnostic tools for timely intervention and improved patient outcomes. The emergence of deep learning methodologies has transformed the landscape of medical image analysis. Recently, Transformers, initially designed for natural language processing, have exhibited remarkable capabilities in various computer vision applications, including medical image analysis. This comprehensive review aims to provide an in-depth exploration of the cutting-edge Transformer-based architectures applied in the context of stroke segmentation. It commences with an exploration of stroke pathology, imaging modalities, and the challenges associated with accurate diagnosis and segmentation. Subsequently, the review delves into the fundamental ideas of Transformers, offering detailed insights into their architectural intricacies and the underlying mechanisms that empower them to effectively capture complex spatial information within medical images. The existing literature is systematically categorized and analyzed, discussing various approaches that leverage Transformers for stroke segmentation. A critical assessment is provided, highlighting the strengths and limitations of these methods, including considerations of performance and computational efficiency. Additionally, this review explores potential avenues for future research and development.

Neural Memory State Space Models for Medical Image Segmentation.

With the rapid advancement of deep learning, computer-aided diagnosis and treatment have become crucial in medicine. UNet is a widely used architecture for medical image segmentation, and various methods for improving UNet have been extensively explored. One popular approach is incorporating transformers, though their quadratic computational complexity poses challenges. Recently, State-Space Models (SSMs), exemplified by Mamba, have gained significant attention as a promising alternative due to their linear computational complexity. Another approach, neural memory Ordinary Differential Equations (nmODEs), exhibits similar principles and achieves good results. In this paper, we explore the respective strengths and weaknesses of nmODEs and SSMs and propose a novel architecture, the nmSSM decoder, which combines the advantages of both approaches. This architecture possesses powerful nonlinear representation capabilities while retaining the ability to preserve input and process global information. We construct nmSSM-UNet using the nmSSM decoder and conduct comprehensive experiments on the PH2, ISIC2018, and BU-COCO datasets to validate its effectiveness in medical image segmentation. The results demonstrate the promising application value of nmSSM-UNet. Additionally, we conducted ablation experiments to verify the effectiveness of our proposed improvements on SSMs and nmODEs.

NA-segformer: A multi-level transformer model based on neighborhood attention for colonoscopic polyp segmentation

In various countries worldwide, the incidence of colon cancer-related deaths has been on the rise in recent years. Early detection of symptoms and identification of intestinal polyps are crucial for improving the cure rate of colon cancer patients. Automated computer-aided diagnosis (CAD) has emerged as a solution to the low efficiency of traditional methods relying on manual diagnosis by physicians. Deep learning is the latest direction of CAD development and has shown promise for colonoscopic polyp segmentation. In this paper, we present a multi-level encoder-decoder architecture for polyp segmentation based on the Transformer architecture, termed NA-SegFormer. To improve the performance of existing Transformer-based segmentation algorithms for edge segmentation on colon polyps, we propose a patch merging module with a neighbor attention mechanism based on overlap patch merging. Since colon tract polyps vary greatly in size and different datasets have different sample sizes, we used a unified focal loss to solve the problem of category imbalance in colon tract polyp data. To assess the effectiveness of our proposed method, we utilized video capsule endoscopy and typical colonoscopy polyp datasets, as well as a dataset containing surgical equipment. On the datasets Kvasir-SEG, Kvasir-Instrument and KvasirCapsule-SEG, the Dice score of our proposed model reached 94.30%, 94.59% and 82.73%, with an accuracy of 98.26%, 99.02% and 81.84% respectively. The proposed method achieved inference speed with an Frame-per-second (FPS) of 125.01. The results demonstrated that our suggested model effectively segmented polyps better than several well-known and latest models. In addition, the proposed method has advantages in trade-off between inference speed and accuracy, and it will be of great significance to real-time colonoscopic polyp segmentation. The code is available at https://github.com/promisedong/NAFormer.

Inception UNet architecture for breast tumor segmentation and detection using hybrid deep learning approach

Inception UNet architecture for breast tumor segmentation and detection using hybrid deep learning approach

HEDN: multi-oriented hierarchical extraction and dual-frequency decoupling network for 3D medical image segmentation.

Previous 3D encoder-decoder segmentation architectures struggled with fine-grained feature decomposition, resulting in unclear feature hierarchies when fused across layers. Furthermore, the blurred nature of contour boundaries in medical imaging limits the focus on high-frequency contour features. To address these challenges, we propose a Multi-oriented Hierarchical Extraction and Dual-frequency Decoupling Network (HEDN), which consists of three modules: Encoder-Decoder Module (E-DM), Multi-oriented Hierarchical Extraction Module (Multi-HEM), and Dual-frequency Decoupling Module (Dual-DM). The E-DM performs the basic encoding and decoding tasks, while Multi-HEM decomposes and fuses spatial and slice-level features in 3D, enriching the feature hierarchy by weighting them through 3D fusion. Dual-DM separates high-frequency features from the reconstructed network using self-supervision. Finally, the self-supervised high-frequency features separated by Dual-DM are inserted into the process following Multi-HEM, enhancing interactions and complementarities between contour features and hierarchical features, thereby mutually reinforcing both aspects. On the Synapse dataset, HEDN outperforms existing methods, boosting Dice Similarity Score (DSC) by 1.38% and decreasing 95% Hausdorff Distance (HD95) by 1.03 mm. Likewise, on the Automatic Cardiac Diagnosis Challenge (ACDC) dataset, HEDN achieves 0.5% performance gains across all categories.