Network For Image Segmentation Research Articles

MFHARFNet: multi-branch feature hybrid and adaptive receptive field network for image segmentation

Abstract Accurate segmentation of medical images is crucial for disease diagnosis and understanding disease changes. Deep learning methods, utilizing encoder-decoder structures, have demonstrated cutting-edge performance in various medical image segmentation tasks. However, the pooling operation in the encoding stage results in feature loss, which makes the network lack the ability to fuse multi-scale information at different levels, hinders its effective perception of multi-scale information, and leads to poor segmentation performance. Drawing inspiration from the U-shaped network, this study introduces a multi-branch feature hybrid attention and adaptive receptive field network (MFHARFNet) for medical image segmentation. Building upon the encoder-decoder framework, we initially devise a multi-branch feature hybrid attention module (MFHAM) to seamlessly integrate feature maps of varying scales, capturing both fine-grained features and coarse-grained semantics across the entire scale. Furthermore, we redesign the skip connection to amalgamate feature information from different branches in the encoder stage and efficiently transmit it to the decoder, providing the decoder with global context feature maps at different levels. Finally, the adaptive receptive field (ARF) module is introduced in the decoder feature reconstruction stage to adapt and focus on related fields, ensuring the model’s adaptation to different segmentation target features, and achieving different weights for the output of different convolution kernels to improve segmentation performance. We comprehensively evaluate our method on medical image segmentation tasks, by using four public datasets across CT and MRI. Remarkably, MFHARFNet method consistently outperforms other state-of-the-art methods, exceeding UNet by 2.1%, 0.9%, 6.6% and 1.0% on Dice on ATLAS, LiTs, BraTs2019 and Spine and intervertebral disc datasets, respectively. In addition, MFHARFNet minimizes network parameters and computational complexity as much as possible. The source codes are in https://github.com/OneHundred99/MFHARFNet.

MCI Net: Mamba- Convolutional lightweight self-attention medical image segmentation network.

With the development of deep learning in the field of medical image segmentation, various network segmentation models have been developed. Currently, the most common network models in medical image segmentation can be roughly categorized into pure convolutional networks, Transformer-based networks, and networks combining convolution and Transformer architectures. However, when dealing with complex variations and irregular shapes in medical images, existing networks face issues such as incomplete information extraction, large model parameter sizes, high computational complexity, and long processing times. In contrast, models with lower parameter counts and complexity can efficiently, quickly, and accurately identify lesion areas, significantly reducing diagnosis time and providing valuable time for subsequent treatments. Therefore, this paper proposes a lightweight network named MCI-Net, with only 5.48 M parameters, a computational complexity of 4.41, and a time complexity of just 0.263. By performing linear modeling on sequences, MCI-Net permanently marks effective features and filters out irrelevant information. It efficiently captures local-global information with a small number of channels, reduces the number of parameters, and utilizes attention calculations with exchange value mapping. This achieves model lightweighting and enables thorough interaction of local-global information within the computation, establishing an overall semantic relationship of local-global information. To verify the effectiveness of the MCI-Net network, we conducted comparative experiments with other advanced representative networks on five public datasets: X-ray, Lung, ISIC-2016, ISIC-2018, and capsule endoscopy and gastrointestinal segmentation. We also performed ablation experiments on the first four datasets. The experimental results outperformed the other compared networks, confirming the effectiveness of MCI-Net. This research provides a valuable reference for achieving lightweight, accurate, and high-performance medical image segmentation network models.

Revisiting Representation Learning of Color Information: Color Medical Image Segmentation Incorporating Quaternion

Currently, color medical image segmentation methods commonly extract color and texture features mixed together by default, however, the distribution of color information and texture information is different: color information is represented differently in different color channels of a color image, while the distribution of texture information remains the same. Such a simple and brute-force feature extraction pattern will inevitably result in a partial bias in the model's semantics understanding. In this paper, we decouple the representation learning for color-texture information, and propose a novel network for color medical image segmentation, named CTNet. Specifically, CTNet introduces the Quaternion CNN (QCNN) module to capture the correlation among different color channels of color medical images to generate color features, and uses a designed local-global texture feature integrator (LoG) to mine the textural features from local to global. Moreover, a multi-stage features interaction strategy is proposed to minimize the semantic understanding gap of color and texture features in CTNet, so that they can be subsequently fused to generate a unified and robust feature representation. Comparative experiments on four different color medical image segmentation benchmark datasets show that CTNet strikes an optimal trade-off between segmentation accuracy and computational overhead when compared to current state-of-the-art methods. We also conduct extensive ablation experiments to verify the effectiveness of the proposed components. Our code will be available at https://github.com/Notmezhan/CTNet.

Medical image segmentation network based on feature filtering with low number of parameters

In recent years, the medical image segmentation method based on hybrid convolutional neural network (CNN) and Vision Transformer (ViT) has made great progress, but it still faces the challenge of unbalanced global and local modeling, and excessive parameters. In addition, ViT repeatedly uses the whole feature map to model the global information, thus generating irrelevant and weakly related information, which will weaken the performance of the model when facing small datasets and segmentation targets. Therefore, this paper proposes a feature screening network based on similarity, named Screening Feature (SF)-MixedNet. Specifically, this paper first proposes a new feature extractor, namely Correlation based Similarity Transformer (CSimFormer). On the basis of parameter pruning, it uses the Screening Feature Multi-head Self Attention (SF-MSA) to establish the remote dependency, and calculates the similarity between local elements through the Location-Sensitive Mechanism (LsM) to obtain the weight matrix. Then, the correlation between regional elements is mined by Region Matching and Selection (RMS) mechanism, and the obtained information is filtered according to the corresponding rules to reduce the side effects of redundant information. Extensive experiments on Synapse dataset, ACDC dataset and SegPC-2021 dataset show that the segmentation accuracy reaches 83.51%, 92.20% and 81.27% respectively. Especially in the Synapse dataset, our method is 6.31% higher than the baseline. The method proposed in this paper effectively improves the segmentation accuracy, provides more detailed information for medical diagnosis and promotes the development of medical artificial intelligence technology.

Dual-attention transformer-based hybrid network for multi-modal medical image segmentation

Accurate medical image segmentation plays a vital role in clinical practice. Convolutional Neural Network and Transformer are mainstream architectures for this task. However, convolutional neural network lacks the ability of modeling global dependency while Transformer cannot extract local details. In this paper, we propose DATTNet, DualATTentionNetwork, an encoder-decoder deep learning model for medical image segmentation. DATTNet is exploited in hierarchical fashion with two novel components: (1) Dual Attention module is designed to model global dependency in spatial and channel dimensions. (2) Context Fusion Bridge is presented to remix the feature maps with multiple scales and construct their correlations. The experiments on ACDC, Synapse and Kvasir-SEG datasets are conducted to evaluate the performance of DATTNet. Our proposed model shows superior performance, effectiveness and robustness compared to SOTA methods, with mean Dice Similarity Coefficient scores of 92.2%, 84.5% and 89.1% on cardiac, abdominal organs and gastrointestinal poly segmentation tasks. The quantitative and qualitative results demonstrate that our proposed DATTNet attains favorable capability across different modalities (MRI, CT, and endoscopy) and can be generalized to various tasks. Therefore, it is envisaged as being potential for practicable clinical applications. The code has been released on https://github.com/MhZhang123/DATTNet/tree/main.

MW‐SAM:Mangrove wetland remote sensing image segmentation network based on segment anything model

AbstractMangrove wetlands are important ecosystems in tropical and subtropical coastal areas, providing wind and wave attenuation and embankment protection functions. However, mangrove wetlands worldwide are facing severe loss and degradation. Accurate identification of mangrove wetland extent is crucial for their protection, but traditional methods struggle to meet the requirements of large‐scale, high‐precision identification. Furthermore, field data collection and annotation in wetlands in complex intertidal zones pose challenges, and the lack of high‐quality training samples limits the application of deep learning methods in this field. This paper proposes a novel semantic segmentation framework for mangrove wetlands called mangrove wetland remote sensing image segmentation network based on segment anything model (MW‐SAM) to address these issues. MW‐SAM is based on the pre‐trained SAM and achieves cross‐domain adaptation through parameter‐efficient fine‐tuning techniques. It introduces wetland‐specific prompts, auxiliary branches, semi‐supervised training, and iterative optimization strategies to improve accuracy and tackle the problem of sample scarcity. Moreover, on the mangrove wetland dataset constructed in this paper, MW‐SAM significantly outperforms SAM and other traditional methods. MW‐SAM provides a new technology for monitoring and protecting mangrove wetlands, and the constructed dataset will be made publicly available, which is expected to promote the development of mangrove wetland conservation efforts.

Deep Learning Approaches for Wildfire Severity Prediction: A Comparative Study of Image Segmentation Networks and Visual Transformers on the EO4WildFires Dataset

This paper investigates the applicability of deep learning models for predicting the severity of forest wildfires, utilizing an innovative benchmark dataset called EO4WildFires. EO4WildFires integrates multispectral imagery from Sentinel-2, SAR data from Sentinel-1, and meteorological data from NASA Power annotated with EFFIS data for forest fire detection and size estimation. These data cover 45 countries with a total of 31,730 wildfire events from 2018 to 2022. All of these various sources of data are archived into data cubes, with the intention of assessing wildfire severity by considering both current and historical forest conditions, utilizing a broad range of data including temperature, precipitation, and soil moisture. The experimental setup has been arranged to test the effectiveness of different deep learning architectures in predicting the size and shape of wildfire-burned areas. This study incorporates both image segmentation networks and visual transformers, employing a consistent experimental design across various models to ensure the comparability of the results. Adjustments were made to the training data, such as the exclusion of empty labels and very small events, to refine the focus on more significant wildfire events and potentially improve prediction accuracy. The models’ performance was evaluated using metrics like F1 score, IoU score, and Average Percentage Difference (aPD). These metrics offer a multi-faceted view of model performance, assessing aspects such as precision, sensitivity, and the accuracy of the burned area estimation. Through extensive testing the final model utilizing LinkNet and ResNet-34 as backbones, we obtained the following metric results on the test set: 0.86 F1 score, 0.75 IoU, and 70% aPD. These results were obtained when all of the available samples were used. When the empty labels were absent during the training and testing, the model increased its performance significantly: 0.87 F1 score, 0.77 IoU, and 44.8% aPD. This indicates that the number of samples, as well as their respectively size (area), tend to have an impact on the model’s robustness. This restriction is well known in the remote sensing domain, as accessible, accurately labeled data may be limited. Visual transformers like TeleViT showed potential but underperformed compared to segmentation networks in terms of F1 and IoU scores.

Alignment of color discrimination in humans and image segmentation networks.

The experiments allowed by current machine learning models imply a revival of the debate on the causes of specific trends of human visual psychophysics. Machine learning facilitates the exploration of the effect of specific visual goals (such as image segmentation) by different neural architectures in different statistical environments in an unprecedented manner. In this way, (1) the principles behind psychophysical facts such as the non-Euclidean nature of human color discrimination and (2) the emergence of human-like behaviour in artificial systems can be explored under a new light. In this work, we show for the first time that the tolerance or invariance of image segmentation networks for natural images under changes of illuminant in the color space (a sort of insensitivity region around the white) is an ellipsoid oriented similarly to a (human) MacAdam ellipse. This striking similarity between an artificial system and human vision motivates a set of experiments checking the relevance of the statistical environment on the emergence of such insensitivity regions. Results suggest, that in this case, the statistics of the environment may be more relevant than the architecture selected to perform the image segmentation.

A solution for the automatic detection of expansion joints in dam stilling pools using underwater robots

This research addresses the critical challenges of detecting and measuring underwater concrete structure expansion joints (CSEJ), with a particular focus on dam stilling basin environments. A full set of algorithms for detecting expansion joints with underwater robots is presented by the research, enabling real-time and offline analysis along with visualization. These algorithms introduce the use of Convolutional Neural Networks (CNNs) for image feature extraction and segmentation in CSEJ detection, significantly improving accuracy and adaptability over traditional methods. An automated approach known as the Underwater Dam Crack Detection Network (UDCD Network) was introduced by this research. And advanced attention gates and hybrid dilated convolutions to enhance its detection capabilities are incorporated in this system. Additionally, a three-phase deep transfer learning strategy during training was utilized in this research, which significantly improves the model's performance. The research suggests an underwater image restoration algorithm that uses a dark channel prior and dynamic white balance correction to improve image clarity. Furthermore, unique underwater environmental challenges were tackled by this research, such as non-contact measurement of expansion joint widths using a view-geometry-based algorithm and comprehensive detection of dam wall surfaces. Experimental validation demonstrates that the advanced imaging technologies of the proposed underwater robots hold significant potential for autonomous detection of expansion joints in dam stilling basins.

Semi‐supervised contextual cognitive augmentation‐based cross‐teaching network for multiclass medical image segmentation

AbstractThe application of medical image segmentation technology enables accurate localization of human tissues, providing doctors with a reliable foundation for diagnosis. While deep learning methods have proven effective in this task, most current approaches rely on a single prediction framework, which overlooks Edge semantic features and results in flawed texture features. Moreover, existing supervised methods face challenges due to limited availability of high‐quality annotations in the field of medical imaging. In this article, a Semi‐supervised Contextual Cognitive Augmentation‐based Cross‐teaching Network is proposed. A Contextual Cognitive Enhancement Module is introduced consisting of two components: data augmentation and information extraction. The data augmentation component provides multi‐level data distribution by incorporating diverse perturbation strategies such as Discrete Cosine Transform and Gaussian noise. The information extraction component employs the Comprehensive Information Extraction module, which consists of Global Perception Information Extraction module and Multi‐channel Information Extraction module to extract perceptual information from images and enhance interaction between image channels, respectively. Additionally, a cross‐teaching strategy is adopted and a hybrid loss function is utilized to encourage knowledge sharing among the networks, leveraging the advantages of dual networks for improved performance. Experimental results demonstrate significant enhancements in multiclass medical image segmentation compared to several state‐of‐the‐art single‐framework networks.

Sub-pixel multi-scale fusion network for medical image segmentation

Sub-pixel multi-scale fusion network for medical image segmentation

SFINet: A semantic feature interactive learning network for full-time infrared and visible image fusion

Infrared and visible image fusion aims to combine data from various source images to generate a high-quality image. Nevertheless, numerous fusion methods often prioritize visual quality above semantic information. To address this problem, we present a Semantic Feature Interactive Learning Network (SFINet) for full-time infrared and visible images. The SFINet encompasses an image fusion network and an image segmentation network through a Semantic Feature Interaction (SFI) module. The image fusion network employs Multi-scale Feature Extraction (MFE) modules to capture global and local information at multiple scales. Meanwhile, it performs an adaptive fusion of complementary information using a Dual Attention Feature Fusion (DAFF) module. The image segmentation network guides the image fusion network using the SFI module for semantic feature interaction. Comparative results prove that the proposed method is superior to state-of-the-art (SOTA) models in image fusion and semantic segmentation tasks. The code is available at https://github.com/songwenhao123/SFINet.

Lightweight medical image segmentation network with multi-scale feature-guided fusion

In the field of computer-aided medical diagnosis, it is crucial to adapt medical image segmentation to limited computing resources. There is tremendous value in developing accurate, real-time vision processing models that require minimal computational resources. When building lightweight models, there is always a trade-off between computational cost and segmentation performance. Performance often suffers when applying models to meet resource-constrained scenarios characterized by computation, memory, or storage constraints. This remains an ongoing challenge. This paper proposes a lightweight network for medical image segmentation. It introduces a lightweight transformer, proposes a simplified core feature extraction network to capture more semantic information, and builds a multi-scale feature interaction guidance framework. The fusion module embedded in this framework is designed to address spatial and channel complexities. Through the multi-scale feature interaction guidance framework and fusion module, the proposed network achieves robust semantic information extraction from low-resolution feature maps and rich spatial information retrieval from high-resolution feature maps while ensuring segmentation performance. This significantly reduces the parameter requirements for maintaining deep features within the network, resulting in faster inference and reduced floating-point operations (FLOPs) and parameter counts. Experimental results on ISIC2017 and ISIC2018 datasets confirm the effectiveness of the proposed network in medical image segmentation tasks. For instance, on the ISIC2017 dataset, the proposed network achieved a segmentation accuracy of 82.33 % mIoU, and a speed of 71.26 FPS on 256 × 256 images using a GeForce GTX 3090 GPU. Furthermore, the proposed network is tremendously lightweight, containing only 0.524M parameters. The corresponding source codes are available at https://github.com/CurbUni/LMIS-lightweight-network.

A fine-segmentation algorithm for XCT images of multiphase composite building materials based on deep learning

In response to the challenges presented by the variable internal structures and complex components of multiphase composite building materials, this study introduces a novel image segmentation network named UT-FusionNet. Building on the U-Net model, UT-FusionNet integrates Transformer module and tensor concatenation and fusion mechanism to overcome the limitations of convolutional networks in relation to the receptive field. UT-FusionNet is employed to segment CT images of conventional concrete, grout consolidation bodies, and fiber-reinforced concrete. The results demonstrate that UT-FusionNet achieves superior segmentation accuracy and robustness, with Accuracy (ACC), Intersection over Union (IoU) and Dice scores exceeding 90 % across all subtasks. The mean accuracy metrics are 99.44 %, 96.70 %, and 98.31 %, respectively. This innovative end-to-end network offers robust support for detailed structural analysis, damage detection, and digital modeling of multiphase composite building materials through deep learning.

Link Aggregation for Skip Connection–Mamba: Remote Sensing Image Segmentation Network Based on Link Aggregation Mamba

The semantic segmentation of satellite and UAV remote sensing imagery is pivotal for address exploration, change detection, quantitative analysis and urban planning. Recent advancements have seen an influx of segmentation networks utilizing convolutional neural networks and transformers. However, the intricate geographical features and varied land cover boundary interferences in remote sensing imagery still challenge conventional segmentation networks’ spatial representation and long-range dependency capabilities. This paper introduces a novel U-Net-like network for UAV image segmentation. We developed a link aggregation Mamba at the critical skip connection stage of UNetFormer. This approach maps and aggregates multi-scale features from different stages into a unified linear dimension through four Mamba branches containing state-space models (SSMs), ultimately decoupling and fusing these features to restore the contextual relationships in the mask. Moreover, the Mix-Mamba module is incorporated, leveraging a parallel self-attention mechanism with SSMs to merge the advantages of a global receptive field and reduce modeling complexity. This module facilitates nonlinear modeling across different channels and spaces through multipath activation, catering to international and local long-range dependencies. Evaluations on public remote sensing datasets like LovaDA, UAVid and Vaihingen underscore the state-of-the-art performance of our approach.

Key Technologies for Autonomous Fruit- and Vegetable-Picking Robots: A Review

With the rapid pace of urbanization, a significant number of rural laborers are migrating to cities, leading to a severe shortage of agricultural labor. Consequently, the modernization of agriculture has become a priority. Autonomous picking robots represent a crucial component of agricultural technological innovation, and their development drives progress across the entire agricultural sector. This paper reviews the current state of research on fruit- and vegetable-picking robots, focusing on key aspects such as the vision system sensors, target detection, localization, and the design of end-effectors. Commonly used target recognition algorithms, including image segmentation and deep learning-based neural networks, are introduced. The challenges of target recognition and localization in complex environments, such as those caused by branch and leaf obstruction, fruit overlap, and oscillation in natural settings, are analyzed. Additionally, the characteristics of the three main types of end-effectors—clamping, suction, and cutting—are discussed, along with an analysis of the advantages and disadvantages of each design. The limitations of current agricultural picking robots are summarized, taking into account the complexity of operation, research and development costs, as well as the efficiency and speed of picking. Finally, the paper offers a perspective on the future of picking robots, addressing aspects such as environmental adaptability, functional diversity, innovation and technological convergence, as well as policy and farm management.

Label refinement network from synthetic error augmentation for medical image segmentation

Deep convolutional neural networks for image segmentation do not learn the label structure explicitly and may produce segmentations with an incorrect structure, e.g., with disconnected cylindrical structures in the segmentation of tree-like structures such as airways or blood vessels. In this paper, we propose a novel label refinement method to correct such errors from an initial segmentation, implicitly incorporating information about label structure. This method features two novel parts: (1) a model that generates synthetic structural errors, and (2) a label appearance simulation network that produces segmentations with synthetic errors that are similar in appearance to the real initial segmentations. Using these segmentations with synthetic errors and the original images, the label refinement network is trained to correct errors and improve the initial segmentations. The proposed method is validated on two segmentation tasks: airway segmentation from chest computed tomography (CT) scans and brain vessel segmentation from 3D CT angiography (CTA) images of the brain. In both applications, our method significantly outperformed a standard 3D U-Net, four previous label refinement methods, and a U-Net trained with a loss tailored for tubular structures. Improvements are even larger when additional unlabeled data is used for model training. In an ablation study, we demonstrate the value of the different components of the proposed method.

HEDN: multi-oriented hierarchical extraction and dual-frequency decoupling network for 3D medical image segmentation.

Previous 3D encoder-decoder segmentation architectures struggled with fine-grained feature decomposition, resulting in unclear feature hierarchies when fused across layers. Furthermore, the blurred nature of contour boundaries in medical imaging limits the focus on high-frequency contour features. To address these challenges, we propose a Multi-oriented Hierarchical Extraction and Dual-frequency Decoupling Network (HEDN), which consists of three modules: Encoder-Decoder Module (E-DM), Multi-oriented Hierarchical Extraction Module (Multi-HEM), and Dual-frequency Decoupling Module (Dual-DM). The E-DM performs the basic encoding and decoding tasks, while Multi-HEM decomposes and fuses spatial and slice-level features in 3D, enriching the feature hierarchy by weighting them through 3D fusion. Dual-DM separates high-frequency features from the reconstructed network using self-supervision. Finally, the self-supervised high-frequency features separated by Dual-DM are inserted into the process following Multi-HEM, enhancing interactions and complementarities between contour features and hierarchical features, thereby mutually reinforcing both aspects. On the Synapse dataset, HEDN outperforms existing methods, boosting Dice Similarity Score (DSC) by 1.38% and decreasing 95% Hausdorff Distance (HD95) by 1.03 mm. Likewise, on the Automatic Cardiac Diagnosis Challenge (ACDC) dataset, HEDN achieves 0.5% performance gains across all categories.

Diagnosis of ophthalmologic diseases in canines based on images using neural networks for image segmentation

The primary challenge in diagnosing ocular diseases in canines based on images lies in developing an accurate and reliable machine learning method capable of effectively segmenting and diagnosing these conditions through image analysis. Addressing this challenge, the study focuses on developing and rigorously evaluating a machine learning model for diagnosing ocular diseases in canines, employing the U-Net neural network architecture as a foundational element of this investigation.Through this extensive evaluation, the authors identified a model that exhibited good reliability, achieving prediction scores with an Intersection over Union (IoU) exceeding 80 %, as measured by the Jaccard index. The research methodology encompassed a systematic exploration of various neural network backbones (VGG, ResNet, Inception, EfficientNet) and the U-Net model, combined with an extensive model selection process and an in-depth analysis of a custom training dataset consisting of historical images of different medical symptoms and diseases in dog eyes.The results indicate a fairly high degree of accuracy in the segmentation and diagnosis of ocular diseases in canines, demonstrating the model's effectiveness in real-world applications. In conclusion, this potentially makes a significant contribution to the field by utilizing advanced machine-learning techniques to develop image-based diagnostic routines in veterinary ophthalmology. This model's successful development and validation offer a promising new tool for veterinarians and pet owners, enhancing early disease detection and improving health outcomes for canine patients.

Dual Channel‐Spatial Self‐Attention Transformer and CNN synergy network for 3D medical image segmentation

Even though the Vision Transformer leverages the self-attention mechanism to capture long-range dependencies, showing significant potential in medical image segmentation, the limited annotations in the image dataset make it difficult for the Transformer model to extract different global features, resulting in attention collapse and generating similar or identical attention maps. Previous studies have attempted to solve the problem by integrating convolutional neural layers into Transformer-based architectures. However, improper integration may lead to the inability of the model to effectively capture local and global information in both spatial and channel dimensions. To address the above issue, we propose a hybrid architecture using the Dual Channel-Spatial Self-Attention Transformer and CNN Synergy Network (DTC-SUNETR) for medical image segmentation. Specifically, we redesigned the self-attention mechanism. A novel Channel-Spatial Self-Attention (CSSA) block is introduced to integrate the enhanced channel and spatial self-attention mechanism to capture the global relationship and local structure among image features. This helps the model to more comprehensively understand the inter-dependencies between different channels and capture the relationships between different pixels, thus enhancing the feature representation of the corresponding dimensions. Simultaneously, it also improves the overall computational efficiency of the network. Extensive experiments on four different medical image segmentation datasets, including Synapse, ACDC, Brain Tumor, and Lung Tumor, demonstrate the superiority of the proposed DTC-SUNETR over state-of-the-art methods.