Pixel-wise Segmentation Research Articles

Crack analysis of tall concrete wind towers using an ad-hoc deep multiscale encoder–decoder with depth separable convolutions under severely imbalanced data

An accurate and timely cracking assessment, including the presence, location and crack geometric feature measurement, is crucial for evaluating concrete wind towers. Therefore, the early identification of cracks is a critical procedure in promptly evaluating structural integrity. This study proposed an ad-hoc encoder–decoder network based on DeepLabv3+ with depth separable convolutions to automatically segment cracks from real-world images captured from various concrete wind towers. The combined advantages of the improved DeepLabv3+ and the lightweight MobileNet v2 are suitable as a benchmark due to their high performance and universality. Four experiments were conducted to determine the model design choice and crack feature measurement capability: (1) six parametric tests using various pre-trained base networks and algorithm optimisers, (2) the influence of complex background noise (i.e., handwriting script) on crack segmentation performance, (3) comparative studies with cutting-edge pixel-wise segmentation models and (4) crack feature measurement (i.e., length and width). The research outcome demonstrated that DeepLabv3+ with MobileNet v2 can potentially be applied for efficient and accurate crack segmentation in concrete wind towers with complex backgrounds.

BreasTDLUSeg: A coarse-to-fine framework for segmentation of breast terminal duct lobular units on histopathological whole-slide images

Automatic segmentation of breast terminal duct lobular units (TDLUs) on histopathological whole-slide images (WSIs) is crucial for the quantitative evaluation of TDLUs in the diagnostic and prognostic analysis of breast cancer. However, TDLU segmentation remains a great challenge due to its highly heterogeneous sizes, structures, and morphologies as well as the small areas on WSIs. In this study, we propose BreasTDLUSeg, an efficient coarse-to-fine two-stage framework based on multi-scale attention to achieve localization and precise segmentation of TDLUs on hematoxylin and eosin (H&E)-stained WSIs. BreasTDLUSeg consists of two networks: a superpatch-based patch-level classification network (SPPC-Net) and a patch-based pixel-level segmentation network (PPS-Net). SPPC-Net takes a superpatch as input and adopts a sub-region classification head to classify each patch within the superpatch as TDLU positive or negative. PPS-Net takes the TDLU positive patches derived from SPPC-Net as input. PPS-Net deploys a multi-scale CNN-Transformer as an encoder to learn enhanced multi-scale morphological representations and an upsampler to generate pixel-wise segmentation masks for the TDLU positive patches. We also constructed two breast cancer TDLU datasets containing a total of 530 superpatch images with patch-level annotations and 2,322 patch images with pixel-level annotations to enable the development of TDLU segmentation methods. Experiments on the two datasets demonstrate that BreasTDLUSeg outperforms other state-of-the-art methods with the highest Dice similarity coefficients of 79.97% and 92.93%, respectively. The proposed method shows great potential to assist pathologists in the pathological analysis of breast cancer. An open-source implementation of our approach can be found at https://github.com/Dian-kai/BreasTDLUSeg.

Spectrally distinct pixel extraction and kernel filtering for brain tumour diagnosis

Before surgery, medical segmentation is carried out to ascertain the limits of regions of interest (ROI). Critical information may be gathered during the planning phase of the operation if it is permitted that the growth, structure, and behaviour of the ROI be studied. This will increase the probability that the operation will be successful. Most of the time, segmentations are carried out either manually or via machine learning algorithms based on hand annotations. The difference between pure background information and an approximation of such information would affect the performance of brain tumour detection algorithms. This paper offers a technique for the identification of tumour regions that is based on kernel background filtration. The proposed approach presented has a more precise estimation of background information as its primary objective. It is composed of four stages. The first stage includes a probabilistic method based on image denoising. The second stage includes extracting the pure background pixels using a kernel-based technique. The third stage estimates the background covariance matrix using the pixels taken from the pure background pixels. In the fourth phase, we perform segmentation using DCNN. The segmentation procedure begins with training a deep convolutional neural network (DCNN) to recreate damaged brain regions. Window-based methods identify pixels with the largest reconstruction loss. This study isolates tumorous regions using pixel-wise segmentation. The proposed method segments tumours of various sizes with an average Dice score of 0.82.

MAPLES-DR: MESSIDOR Anatomical and Pathological Labels for Explainable Screening of Diabetic Retinopathy

Reliable automatic diagnosis of Diabetic Retinopathy (DR) and Macular Edema (ME) is an invaluable asset in improving the rate of monitored patients among at-risk populations and in enabling earlier treatments before the pathology progresses and threatens vision. However, the explainability of screening models is still an open question, and specifically designed datasets are required to support the research. We present MAPLES-DR (MESSIDOR Anatomical and Pathological Labels for Explainable Screening of Diabetic Retinopathy), which contains, for 198 images of the MESSIDOR public fundus dataset, new diagnoses for DR and ME as well as new pixel-wise segmentation maps for 10 anatomical and pathological biomarkers related to DR. This paper documents the design choices and the annotation procedure that produced MAPLES-DR, discusses the interobserver variability and the overall quality of the annotations, and provides guidelines on using the dataset in a machine learning context.

NetraDeep: An Integrated Deep Learning and Image Processing System for Precise Detection of Hard Exudates

Hard Exudate (HE) is a common manifestation of various eye diseases, such as diabetic retinopathy (DR), and a prominent cause of vision loss and blindness. Researchers aim to visualize and quantify these exudates using deep learning (DL) and image processing (IP) models from retinal images. However, the requirement for a large number of labelled image datasets for DL models to work on diverse and poor-quality images makes this task challenging. To address this challenge, we introduce NetraDeep, a system that integrates data-driven DL and rule-based IP techniques for exudate segmentation. Our system uses IP models to detect and extract some features and assists DL models in detecting more advanced features and vice versa. The IP models are rule-based and use predefined rules to process images, while the DL models are data-driven and learn from the input data. NetraDeep provides visual and quantitative assessments while mitigating noise and other confounding factors such as artifacts and noise. The training of DL models of this system requires only a limited number of labelled fundus images from publicly available datasets. It provides accurate pixel-wise segmentation results on the public and private image datasets collected from local eye hospitals. Through extensive evaluation, our system achieved remarkable performance, with a dice coefficient of \(0.84\) for the public dataset and a rating of \(9.78\) and \(9.43\) out of 10, as corroborated by two medical experts with experience of more than 20 and 5 years, respectively, for the private image dataset.

Enhancing Prostate Cancer Diagnosis: Artificial Intelligence–Driven Virtual Biopsy for Optimal Magnetic Resonance Imaging-Targeted Biopsy Approach and Gleason Grading Strategy

An optimal approach to magnetic resonance imaging fusion targeted prostate biopsy (PBx) remains unclear (number of cores, intercore distance, Gleason grading [GG] principle). The aim of this study was to develop a precise pixel-wise segmentation diagnostic artificial intelligence (AI) algorithm for tumor detection and GG as well as an algorithm for virtual prostate biopsy that are used together to systematically investigate and find an optimal approach to targeted PBx. Pixel-wise AI algorithms for tumor detection and GG were developed using a high-quality, manually annotated data set (slides n = 442) after fast-track annotation transfer into segmentation style. To this end, a virtual biopsy algorithm was developed that can perform random biopsies from tumor regions in whole-mount whole-slide images with predefined parameters. A cohort of 115 radical prostatectomy (RP) patient cases with clinically significant, magnetic resonance imaging-visible tumors (n = 121) was used for systematic studies of the optimal biopsy approach. Three expert genitourinary (GU) pathologists (Y.T., A.P., A.Q.) participated in the validation. The tumor detection algorithm (aware version sensitivity/specificity 0.99/0.90, balanced version 0.97/0.97) and GG algorithm (quadratic kappa range vs pathologists 0.77-0.78) perform on par with expert GU pathologists. In total, 65,340 virtual biopsies were performed to study different biopsy approaches with the following results: (1) 4 biopsy cores is the optimal number for a targeted PBx, (2) cumulative GG strategy is superior to using maximal Gleason score for single cores, (3) controlling for minimal intercore distance does not improve the predictive accuracy for the RP Gleason score, (4) using tertiary Gleason pattern principle (for AI tool) in cumulative GG strategy might allow better predictions of final RP Gleason score. The AI algorithm (based on cumulative GG strategy) predicted the RP Gleason score of the tumor better than 2 of the 3 expert GU pathologists. In this study, using an original approach of virtual prostate biopsy on the real cohort of patient cases, we find the optimal approach to the biopsy procedure and the subsequent GG of a targeted PBx. We publicly release 2 large data sets with associated expert pathologists’ GG and our virtual biopsy algorithm.

Multi-dimensional dense attention network for pixel-wise segmentation of optic disc in colour fundus images.

Segmentation of retinal fragments like blood vessels, Optic Disc (OD), and Optic Cup (OC) enables the early detection of different retinal pathologies like Diabetic Retinopathy (DR), Glaucoma, etc. Accurate segmentation of OD remains challenging due to blurred boundaries, vessel occlusion, and other distractions and limitations. These days, deep learning is rapidly progressing in the segmentation of image pixels, and a number of network models have been proposed for end-to-end image segmentation. However, there are still certain limitations, such as limited ability to represent context, inadequate feature processing, limited receptive field, etc., which lead to the loss of local details and blurred boundaries. A multi-dimensional dense attention network, or MDDA-Net, is proposed for pixel-wise segmentation of OD in retinal images in order to address the aforementioned issues and produce more thorough and accurate segmentation results. In order to acquire powerful contexts when faced with limited context representation capabilities, a dense attention block is recommended. A triple-attention (TA) block is introduced in order to better extract the relationship between pixels and obtain more comprehensive information, with the goal of addressing the insufficient feature processing. In the meantime, a multi-scale context fusion (MCF) is suggested for acquiring the multi-scale contexts through context improvement. Specifically, we provide a thorough assessment of the suggested approach on three difficult datasets. In the MESSIDOR and ORIGA data sets, the suggested MDDA-NET approach obtains accuracy levels of 99.28% and 98.95%, respectively. The experimental results show that the MDDA-Net can obtain better performance than state-of-the-art deep learning models under the same environmental conditions.

Pixel-wise segmentation of cells in digitized Pap smear images

A simple and cheap way to recognize cervical cancer is using light microscopic analysis of Pap smear images. Training artificial intelligence-based systems becomes possible in this domain, e.g., to follow the European recommendation to screen negative smears to reduce false negative cases. The first step for such a process is segmenting the cells. A large and manually segmented dataset is required for this task, which can be used to train deep learning-based solutions. We describe a corresponding dataset with accurate manual segmentations for the enclosed cells. Altogether, the APACS23 (Annotated PAp smear images for Cell Segmentation 2023) dataset contains about 37 000 manually segmented cells and is separated into dedicated training and test parts, which could be used for an official benchmark of scientific investigations or a grand challenge.

U-Net Ensemble for Enhanced Semantic Segmentation in Remote Sensing Imagery

Semantic segmentation of remote sensing imagery stands as a fundamental task within the domains of both remote sensing and computer vision. Its objective is to generate a comprehensive pixel-wise segmentation map of an image, assigning a specific label to each pixel. This facilitates in-depth analysis and comprehension of the Earth’s surface. In this paper, we propose an approach for enhancing semantic segmentation performance by employing an ensemble of U-Net models with three different backbone networks: Multi-Axis Vision Transformer, ConvFormer, and EfficientNet. The final segmentation maps are generated through a geometric mean ensemble method, leveraging the diverse representations learned by each backbone network. The effectiveness of the base U-Net models and the proposed ensemble is evaluated on multiple datasets commonly used for semantic segmentation tasks in remote sensing imagery, including LandCover.ai, LoveDA, INRIA, UAVid, and ISPRS Potsdam datasets. Our experimental results demonstrate that the proposed approach achieves state-of-the-art performance, showcasing its effectiveness and robustness in accurately capturing the semantic information embedded within remote sensing images.

An Adaptive Image Segmentation Network for Surface Defect Detection.

Surface defect detection plays an essential role in industry, and it is challenging due to the following problems: 1) the similarity between defect and nondefect texture is very high, which eventually leads to recognition or classification errors and 2) the size of defects is tiny, which are much more difficult to be detected than larger ones. To address such problems, this article proposes an adaptive image segmentation network (AIS-Net) for pixelwise segmentation of surface defects. It consists of three main parts: multishuffle-block dilated convolution (MSDC), dual attention context guidance (DACG), and adaptive category prediction (ACP) modules, where MSDC is designed to merge the multiscale defect features for avoiding the loss of tiny defect feature caused by model depth, DACG is designed to capture more contextual information from the defect feature map for locating defect regions and obtaining clear segmentation boundaries, and ACP is used to make classification and regression for predicting defect categories. Experimental results show that the proposed AIS-Net is superior to the state-of-the-art approaches on four actual surface defect datasets (NEU-DET: 98.38% ± 0.03%, DAGM: 99.25% ± 0.02%, Magnetic-tile: 98.73% ± 0.13%, and MVTec: 99.72% ± 0.02%).

A framework for general-purpose microscopic image analysis via self-supervised learning

Combining materials science, artificial intelligence (AI) offers great potential for the extensive quantitative analysis and processing of material characterization associated with high-throughput experiments. However, due to the complex and diverse morphology of structural components, the development of AI models requires substantial annotation and models are usually task-specific with limited generalizability to different clinical applications. Here, we present a universal self-supervised learning framework for microscopic images. Our framework learns generalizable representations from unlabelled images and provides a pixel-wise segmentation for quantitative microstructure analysis in a variety of materials science applications. Specifically, the framework learns feature from a single image by means of self-supervised learning, and adapts it to a series of related tasks. We show that our method consistently outperforms several comparisons supervised or weakly supervised learning models in the context of various applications. Our approach provides a generalizable solution to improve model performance and alleviate the annotation workload of experts to enable practical AI applications from microscopic imaging.

Ultra Fast Deep Lane Detection With Hybrid Anchor Driven Ordinal Classification.

Modern methods mainly regard lane detection as a problem of pixel-wise segmentation, which is struggling to address the problems of efficiency and challenging scenarios like severe occlusions and extreme lighting conditions. Inspired by human perception, the recognition of lanes under severe occlusions and extreme lighting conditions is mainly based on contextual and global information. Motivated by this observation, we propose a novel, simple, yet effective formulation aiming at ultra fast speed and the problem of challenging scenarios. Specifically, we treat the process of lane detection as an anchor-driven ordinal classification problem using global features. First, we represent lanes with sparse coordinates on a series of hybrid (row and column) anchors. With the help of the anchor-driven representation, we then reformulate the lane detection task as an ordinal classification problem to get the coordinates of lanes. Our method could significantly reduce the computational cost with the anchor-driven representation. Using the large receptive field property of the ordinal classification formulation, we could also handle challenging scenarios. Extensive experiments on four lane detection datasets show that our method could achieve state-of-the-art performance in terms of both speed and accuracy. A lightweight version could even achieve 300+ frames per second(FPS). Our code is at https://github.com/cfzd/Ultra-Fast-Lane-Detection-v2.

A Novel 3D Multi-Layer Convolutional Neural Networks for Lung Cancer Segmentation in CT Images

Background/Objectives: A novel three-dimensional efficient Multi-Layer Convolutional Neural Network (3D-MLCNN) is proposed for detecting lung tumors accurately using Computerized Tomography (CT) lung tumor images. The proposed K-means segmentation algorithm for labeling the tumor region automatically. This proposed K-means segmentation algorithm automatically labels the tumor regions to process with the 3D MLCNN model to predict tiny tumors and extract tumor regions accurately. Methods: The proposed 3DMLCNN network goal is to extract the tumor region in CT lung images to classify the lung tumor volume by pixel-wise segmentation model. Findings: The proposed 3D MLCNN segmentation model for detecting the segmenting tumors produces outperforming results for predicting even tiny tumors in the lung images. Experimental results demonstrated with lung cancer CT images in TCIA datasets show that the proposed model 3D-MLCNN achieved a dice coefficient (9.6%), Intersection over Union (IoU) (80%), F1-Score (9.33%), Sensitivity (17.11%), and Accuracy (98%) respectively. However, the proposed model 3D MLCNN was evaluated and compared with the existing state-of-the-art segmentation methods, which shows a 10% improvement in the segmentation process. Novelty: A novel 3D MLCNN model enhances the tumor region and predicts the tumor accurately by labeling the tumor using K-means labeling techniques. Keywords: 3D Convolutional Neural Networks, Lung Cancer CT Images, K - Means Labeling, Feature Visualization, Deep Learning

MRL-Seg: Overcoming Imbalance in Medical Image Segmentation With Multi-Step Reinforcement Learning.

Medical image segmentation is a critical task for clinical diagnosis and research. However, dealing with highly imbalanced data remains a significant challenge in this domain, where the region of interest (ROI) may exhibit substantial variations across different slices. This presents a significant hurdle to medical image segmentation, as conventional segmentation methods may either overlook the minority class or overly emphasize the majority class, ultimately leading to a decrease in the overall generalization ability of the segmentation results. To overcome this, we propose a novel approach based on multi-step reinforcement learning, which integrates prior knowledge of medical images and pixel-wise segmentation difficulty into the reward function. Our method treats each pixel as an individual agent, utilizing diverse actions to evaluate its relevance for segmentation. To validate the effectiveness of our approach, we conduct experiments on four imbalanced medical datasets, and the results show that our approach surpasses other state-of-the-art methods in highly imbalanced scenarios. These findings hold substantial implications for clinical diagnosis and research.

Sustainable Inference of Remote Sensing Data by Recursive Semantic Segmentation – a Flood Extent Mapping Study

In times of climate change and large machine learning models with petabytes of training data, the demand for responsible AI methodologies is more pressing than ever. This is in-particular true for the vast amount of remote sensing data. Its value to explore and inform about earth processes is of paramount importance, especially in times of global warming. Thus, it is nearly ironic that such applications can be the cause of substantial green-house-gas emissions, through energy demands from compute and communication systems. Thus, this study aims at reducing the data transfer between data centers while maintaining near-real time insights from remote sensing data. A recursive inference approach is introduced consisting of three steps: i) Data pyramid preparation in the host data center (sequence of upscaled raster data). ii) The transfer of low-resolution images to the service data center, where a deep-learning model performs a semantic segmentation task, including an uncertainty estimation. Images of higher resolution are then requested and segmented in a recursive fashion, in areas of high uncertainty only. iii) Finally, the merging of the predictions at different resolutions is performed to result in the final pixel-wise segmentation at scale. The method is demonstrated on synthetic and real-world data for a flood mapping task. A U-Net encoder-decoder model is used for the semantic segmentation task, using Monte-Carlo dropout to result in the uncertainty map. The proof-of-concept demonstrated a 35-38% performance gain per transferred pixel compared to high-resolution image segmentation only. Further, we perform a scaling study to estimate the true potential of the recursive inference approach, indicating the potential to reduce a data transfer by up to 98%, considering four hierarchy levels in the data pyramid. With this study, we hope to have contributed a small but important step towards sustainable machine learning.

Concrete crack segmentation based on multi-dimensional structure information fusion-based network

The state-of-the-art (SOTA) crack segmentation methods still face the challenges in low-contrast shallow crack recognition and multi-scene compatibility. Therefore, this paper proposes a solution based on a multi-dimensional structure information fusion-based network (MDSIFNet). This network consists of two branches, one for extracting two-dimensional (2D) spatial feature information and another for acquiring three-dimensional (3D) geometric one. In the former, the designed 2D curve structure constraint module based on prior knowledge combined with a pretrained 2D feature extraction module can reduce learning samples and realize crack segmentation in unseen scenes. In the latter, a one-hot transformation and a 3D geometric structure information extraction module are designed to make the network pay attention to the geometric features of shallow cracks and improve their segmentation accuracy. Finally, a fusion module couples the multi-dimensional structure feature information of these two branches and outputs pixel-wise segmentation results that can be used for crack measurement and quantitative analysis. The experimental results show that the designed model network MDSIFNet performs better than the SOTA methods in terms of performance and visualized results.

Korean Pansori Vocal Note Transcription Using Attention-Based Segmentation and Viterbi Decoding

In this paper, first, we delved into the experiment by comparing various attention mechanisms in the semantic pixel-wise segmentation framework to perform frame-level transcription tasks. Second, the Viterbi algorithm was utilized by transferring the knowledge of the frame-level transcription model to obtain the vocal notes of Korean Pansori. We considered a semantic pixel-wise segmentation framework for frame-level transcription as the source task and a Viterbi algorithm-based Korean Pansori note-level transcription as the target task. The primary goal of this paper was to transcribe the vocal notes of Pansori music, a traditional Korean art form. To achieve this goal, the initial step involved conducting the experiments with the source task, where a trained model was employed for vocal melody extraction. To achieve the desired vocal note transcription for the target task, the Viterbi algorithm was utilized with the frame-level transcription model. By leveraging this approach, we sought to accurately transcribe the vocal notes present in Pansori performances. The effectiveness of our attention-based segmentation methods for frame-level transcription in the source task has been compared with various algorithms using the vocal melody task of the MedleyDB dataset, enabling us to measure the voicing recall, voicing false alarm, raw pitch accuracy, raw chroma accuracy, and overall accuracy. The results of our experiments highlight the significance of attention mechanisms for enhancing the performance of frame-level music transcription models. We also conducted a visual and subjective comparison to evaluate the results of the target task for vocal note transcription. Since there was no ground truth vocal note for Pansori, this analysis provides valuable insights into the preservation and appreciation of this culturally rich art form.

Deep learning framework for bovine iris segmentation.

Iris segmentation is an initial step for identifying the biometrics of animals when establishing a traceability system for livestock. In this study, we propose a deep learning framework for pixel-wise segmentation of bovine iris with a minimized use of annotation labels utilizing the BovineAAEyes80 public dataset. The proposed image segmentation framework encompasses data collection, data preparation, data augmentation selection, training of 15 deep neural network (DNN) models with varying encoder backbones and segmentation decoder DNNs, and evaluation of the models using multiple metrics and graphical segmentation results. This framework aims to provide comprehensive and in-depth information on each model's training and testing outcomes to optimize bovine iris segmentation performance. In the experiment, U-Net with a VGG16 backbone was identified as the optimal combination of encoder and decoder models for the dataset, achieving an accuracy and dice coefficient score of 99.50% and 98.35%, respectively. Notably, the selected model accurately segmented even corrupted images without proper annotation data. This study contributes to the advancement of iris segmentation and the establishment of a reliable DNN training framework.

A pixel-wise segmentation method for automatic X-ray image detection of chip packaging defects

A pixel-wise segmentation method for automatic X-ray image detection of chip packaging defects

Rethinking automatic segmentation of gross target volume from a decoupling perspective

Accurate and reliable segmentation of Gross Target Volume (GTV) is critical in cancer Radiation Therapy (RT) planning, but manual delineation is time-consuming and subject to inter-observer variations. Recently, deep learning methods have achieved remarkable success in medical image segmentation. However, due to the low image contrast and extreme pixel imbalance between GTV and adjacent tissues, most existing methods usually obtained limited performance on automatic GTV segmentation. In this paper, we propose a Heterogeneous Cascade Framework (HCF) from a decoupling perspective, which decomposes the GTV segmentation into independent recognition and segmentation subtasks. The former aims to screen out the abnormal slices containing GTV, while the latter performs pixel-wise segmentation of these slices. With the decoupled two-stage framework, we can efficiently filter normal slices to reduce false positives. To further improve the segmentation performance, we design a multi-level Spatial Alignment Network (SANet) based on the feature pyramid structure, which introduces a spatial alignment module into the decoder to compensate for the information loss caused by downsampling. Moreover, we propose a Combined Regularization (CR) loss and Balance-Sampling Strategy (BSS) to alleviate the pixel imbalance problem and improve network convergence. Extensive experiments on two public datasets of StructSeg2019 challenge demonstrate that our method outperforms state-of-the-art methods, especially with significant advantages in reducing false positives and accurately segmenting small objects. The code is available at https://github.com/shijun18/GTV_AutoSeg.