Neural Network For Image Segmentation Research Articles

Scalable semantic 3D mapping of coral reefs with deep learning

Abstract Coral reefs are among the most diverse ecosystems on our planet, and essential to the livelihood of hundreds of millions of people who depend on them for food security, income from tourism and coastal protection. Unfortunately, most coral reefs are existentially threatened by global climate change and local anthropogenic pressures. To better understand the dynamics underlying deterioration of reefs, monitoring at high spatial and temporal resolution is key. However, conventional monitoring methods for quantifying coral cover and species abundance are limited in scale due to the extensive manual labor required. Although computer vision tools have been employed to aid in this process, in particular structure‐from‐motion (SfM) photogrammetry for 3D mapping and deep neural networks for image segmentation, analysis of the data products creates a bottleneck, effectively limiting their scalability. This paper presents a new paradigm for mapping underwater environments from ego‐motion video, unifying 3D mapping systems that use machine learning to adapt to challenging conditions under water, combined with a modern approach for semantic segmentation of images. The method is exemplified on coral reefs in the northern Gulf of Aqaba, Red Sea, demonstrating high‐precision 3D semantic mapping at unprecedented scale with significantly reduced required labor costs: given a trained model, a 100 m video transect acquired within 5 min of diving with a cheap consumer‐grade camera can be fully automatically transformed into a semantic point cloud within 5 min. We demonstrate the spatial accuracy of our method and the semantic segmentation performance (of at least 80% total accuracy), and publish a large dataset of ego‐motion videos from the northern Gulf of Aqaba, along with a dataset of video frames annotated for dense semantic segmentation of benthic classes. Our approach significantly scales up coral reef monitoring by taking a leap towards fully automatic analysis of video transects. The method advances coral reef transects by reducing the labor, equipment, logistics, and computing cost. This can help to inform conservation policies more efficiently. The underlying computational method of learning‐based Structure‐from‐Motion has broad implications for fast low‐cost mapping of underwater environments other than coral reefs.

Semi-supervised segmentation for construction and demolition waste recognition in-the-wild: Adversarial dual-view networks

Precise, and automated segmentation of construction and demolition waste (CDW) is crucial for recognizing the composition of mixed waste streams and facilitating automatic waste sorting. Training a neural network for image segmentation is challenging due to the time and resource-intensive nature of annotating large-scale datasets, particularly for domain-specific waste recognition in cluttered environments. In this paper, we propose a semi-supervised multi-class segmentation approach to recognize CDW in real-world settings, utilizing an adversarial dual-view framework. In doing so, we utilize a critic network to enable mutual learning between views using high-confidence predictions. We collected and annotated images of CDW in-the-wild and experimented with various portions of unlabelled data. By minimizing a multi-task loss function, inclusive of supervised, unsupervised, and adversarial losses, our method achieves a frequency-weighted intersection over union of 0.62 and mean pixel accuracy of 0.76 across eight classes, with equal splits of labelled and unlabelled data. The findings realize the proposed method achieves competitive performance compared to fully supervised methods even with limited labelled data. This is useful in waste recognition practices by reducing the time and resources needed for data annotations. Furthermore, it paves the way for accurate waste sorting, facilitating efficient CDW recycling and resource recovery.

Supine Positioning for Graft Attachment After Descemet Membrane Endothelial Keratoplasty: A Randomized Controlled Trial

The Supine Positioning for Descemet Membrane Endothelial Keratoplasty Attachment (SUPER-DMEK) trial assessed the efficacy of prolonged supine head positioning on graft attachment. Randomized controlled trial. Participants with Fuchs' dystrophy were randomized to 5 days of supine head positioning (intervention) or to 1 day (control). Participants, surgeons, and investigators were masked until the day after surgery. Adherence to the allocated intervention was monitored using a head sensor. Main outcome measures were area and volume of graft detachment (coprimary end points) 2 weeks after surgery quantified using a validated neural network for image segmentation on anterior segment optical coherence tomography images, and repeat air injection (rebubbling), subjective visual function, and adverse events (secondary end points). A total of 86 participants received the allocated intervention (35 eyes intervention and 51 eyes control). In the intention-to-treat analysis, the mean area of graft detachment was 28.6% in the intervention arm and 27.5% in the control arm (adjusted between-arm difference, 1.3; 95% CI, -8.7 to 11.4; P = .80). Results for volume of detachment and as-treated analyses based on head position sensor data indicated no potentially clinically relevant effect of prolonged supine positioning on graft attachment. Results were not compatible with a relevant treatment effect on rebubbling or subjective visual function. Adverse events, most commonly back pain, were more common and more severe with the intervention. In this randomized controlled trial, graft attachment was not improved with prolonged supine head positioning. Prolonged supine positioning frequently caused back pain. Prolonged supine positioning after Descemet membrane endothelial keratoplasty for Fuchs' dystrophy may not be needed in routine practice.

Vision-enhanced Peg-in-Hole for automotive body parts using semantic image segmentation and object detection

Artificial Intelligence (AI) is an enabling technology in the context of Industry 4.0. In particular, the automotive sector is among those who can benefit most of the use of AI in conjunction with advanced vision techniques. The scope of this work is to integrate deep learning algorithms in an industrial scenario involving a robotic Peg-in-Hole task. More in detail, we focus on a scenario where a human operator manually positions a carbon fiber automotive part in the workspace of a 7 Degrees of Freedom (DOF) manipulator. To cope with the uncertainty on the relative position between the robot and the workpiece, we adopt a three stage strategy. The first stage concerns the Three-Dimensional (3D) reconstruction of the workpiece using a registration algorithm based on the Iterative Closest Point (ICP) paradigm. Such a procedure is integrated with a semantic image segmentation neural network, which is in charge of removing the background of the scene to improve the registration. The adoption of such network allows to reduce the registration time of about 28.8%. In the second stage, the reconstructed surface is compared with a Computer Aided Design (CAD) model of the workpiece to locate the holes and their axes. In this stage, the adoption of a Convolutional Neural Network (CNN) allows to improve the holes’ position estimation of about 57.3%. The third stage concerns the insertion of the peg by implementing a search phase to handle the remaining estimation errors. Also in this case, the use of the CNN reduces the search phase duration of about 71.3%. Quantitative experiments, including a comparison with a previous approach without both the segmentation network and the CNN, have been conducted in a realistic scenario. The results show the effectiveness of the proposed approach and how the integration of AI techniques improves the success rate from 84.5% to 99.0%.

Mathematics of biomedical imaging today—a perspective

Biomedical imaging is a fascinating, rich and dynamic research area, which has huge importance in biomedical research and clinical practice alike. The key technology behind the processing, and automated analysis and quantification of imaging data is mathematics. Starting with the optimisation of the image acquisition and the reconstruction of an image from indirect tomographic measurement data, all the way to the automated segmentation of tumours in medical images and the design of optimal treatment plans based on image biomarkers, mathematics appears in all of these in different flavours. Non-smooth optimisation in the context of sparsity-promoting image priors, partial differential equations for image registration and motion estimation, and deep neural networks for image segmentation, to name just a few. In this article, we present and review mathematical topics that arise within the whole biomedical imaging pipeline, from tomographic measurements to clinical support tools, and highlight some modern topics and open problems. The article is addressed to both biomedical researchers who want to get a taste of where mathematics arises in biomedical imaging as well as mathematicians who are interested in what mathematical challenges biomedical imaging research entails.

Automated image segmentation of scanning electron microscopy images of graphene using U-Net Neural Network

Scanning electron microscopy (SEM) is one of the most common approaches to the characterization of synthesized 2D materials such as graphene. Images from SEM contain detailed information about crystalline properties, domain size, and nucleation density, but typically are analyzed through a laborious, serial process that relies on the trained eye of synthesis experts. In this work, we demonstrate an image segmentation neural network that automatically distinguishes between pixels in SEM images that correspond to regions where graphene is and is not present. We utilize the U-Net architecture to learn on a training data set of more than 90 pre-labeled images coupled with moderate image augmentation. Comparing the performance of models trained on smaller high fidelity data set to those trained on larger low fidelity data sets, we find that higher quality is more valuable than higher quantity for achieving good performance. When neural network hyperparameters such as batch size and learning rate are properly tuned, the learned model shows an accuracy for classification of over 90% and an F1 score over 80%. The neural network trained on SEM images of graphene shows reasonable performance when directly applied to other 2D materials, suggesting the possibility of use in transfer learning. Detailed analysis of the inner workings of the model reveals that the domain edges are most critical for making classifications when segmenting the image. We also show the use of a post processing technique to estimate the graphene domain size using segmented masks. This demonstration shows the potential for SEM image segmentation at scale using deep learning approaches and gives insights into best practices for improving model performance.

A Deep Neural Network for Image Segmentation

A Deep Neural Network for Image Segmentation

Graft Detachment after Descemet Membrane Endothelial Keratoplasty with and without Cataract Surgery.

To evaluate graft detachment after Descemet membrane endothelial keratoplasty (DMEK) in pseudophakic eyes and DMEK combined with cataract surgery (triple DMEK). Analysis of 3 single-center prospective cohort studies and 1 randomized controlled trial. Participants with Fuchs' endothelial corneal dystrophy. A validated neural network for image segmentation quantified graft detachment on anterior segment OCT (AS-OCT) images 3 days after DMEK and at the 2-week postoperative visit. Area and volume of graft detachment were compared between DMEK only and triple DMEK using generalized estimating equation models and adjusting for participant age and the size of the air bubble. Area and volume of DMEK graft detachment. Among 207 participants with 270 eyes included, 75 pseudophakic eyes had DMEK only and 195 eyes had triple DMEK. A total of 147 eyes had less than one third of detachment at day 3. In 139 of these eyes (95%), detachment was still less than one third at the 2-week scan, indicating that postoperative graft detachment at 2 weeks occurred mainly in eyes with early detachment. When superimposing all 3-dimensional maps from 2 weeks after surgery, the central graft was mainly attached and detachment was located at the graft margin. The mean area of graft detachment decreased from 28% in DMEK only and 38% in triple DMEK to 16% in DMEK only and 25% in triple DMEK at the 2-week postoperative visit. At 2 weeks, the mean area of detachment was 1.85-fold higher (95% confidence interval [CI], 1.34-2.56) and the mean volume was 2.41-fold higher (95% CI, 1.51-3.86) in triple DMEK compared with DMEK. A total of 46 eyes received rebubbling procedures, with 7 eyes (9%) in the DMEK group and 39 eyes (20%) in the triple DMEK group (adjusted risk ratio, 3.1; 95% CI, 1.3-7.1), indicating that rebubbling was more common in eyes undergoing triple DMEK. Automated segmentation of AS-OCT images allowed precise quantification of graft detachment over time and identified DMEK combined with cataract surgery as a risk factor. Frequency of operative follow-up might be guided by extent of detachment in the first postoperative days after DMEK.

A deep learning approach for automatic echocardiographic right ventricular strain measurements using a limited dataset

Abstract Background Speckle tracking echocardiography provides quantification of myocardial deformation and is useful in the assessment of myocardial function. Right ventricular (RV) strain has been suggested as a sensitive tool for diagnosing cardiomyopathies and assessing long term patient outcomes for patients with pulmonary hypertension, severe tricuspid regurgitation and COVID-19 infection. Recent advances in deep learning (DL) have made promising advances in automating the labour-intensive delineation of regions of interest (ROIs). However, compared to echocardiograms with left ventricular (LV) strain, RV strain data is scarce, making DL models difficult to train. Purpose To investigate whether annotated LV strain data could be beneficial in training a DL model for automatic RV strain when using a limited RV dataset. Methods The dataset consisted of anonymized still frames from 141 echocardiograms of the RV in the RV-focused 4 chamber view with corresponding cardiologist-defined ROI. Exams included healthy subjects and patients with heart failure, valvular disease, and conduction abnormalities. ROIs and still images were extracted at the mid-systole, and then quality assessed by an experienced cardiologist as high, medium, or low. The dataset was randomly split into 68%/17%/15% sets for training, validation, and testing. A convolutional neural network for image segmentation (U-Net) with a residual neural network (ResNet50) encoder was used, with a combination of binary cross entropy and Dice loss functions. Augmentation, predefined ImageNet weights and pre-training were also employed. For pre-training, 715 still images in the apical 4 chamber view with LV defined ROIs were used, both in their original and horizontally flipped view. Predicted ROIs were reintroduced into commercially available echocardiogram analysis software to automatically calculate longitudinal strain (LS) values. Results The model pre-trained with the flipped LV images achieved the highest performance with a mean absolute difference of 1.26 percentage points (95% confidence interval (CI): 0.62–1.89 percentage points) between manually measured and DL-assisted LS. Median absolute LS difference was 0.85 (95% CI: 0.28–1.57) percentage points. A Bland-Altman plot revealed two outliers and no obvious trends. In comparison, the mean and median absolute LS differences for the model without pre-training were 1.87 (95% CI: 0.73–3.00) and 1.09 (95% CI: 0.56–1.63) percentage points, respectively. Conclusions The current study demonstrates that DL-assisted, automated RV strain measurement is feasible even with a small dataset, and that performance can be increased by using images annotated for LV strain. While the majority of the predicted RV strain results were within the typical range of intra- and interobserver variability, a few outliers were observed. These outliers could possibly be avoided with the use of larger datasets. Funding Acknowledgement Type of funding sources: Public grant(s) – National budget only. Main funding source(s): Research council of Norway

Mapping artisanal and small-scale mines at large scale from space with deep learning.

Artisanal and small-scale mines (asm) are on the rise. They represent a crucial source of wealth for numerous communities but are rarely monitored or regulated. The main reason being the unavailability of reliable information on the precise location of the asm which are mostly operated informally or illegally. We address this issue by developing a strategy to map the asm locations using a convolutional neural network for image segmentation, aiming to detect surface mining with satellite data. Our novel dataset is the first comprehensive measure of asm activity over a vast area: we cover 1.75 million km2 across 13 countries in Sub-Tropical West Africa. The detected asm activities range from 0.1 ha to around 2, 000 ha and present a great diversity, yet we succeed in hitting acceptable compromises of performance, as achieving 70% precision while maintaining simultaneously 42% recall. Ultimately, the remarkable robustness of our procedure makes us confident that our method can be applied to other parts of Africa or the world, thus facilitating research and policy opportunities in this sector.

PERFORMANCE EVALUATION OF FUSION TECHNIQUES FOR CROSS-DOMAIN BUILDING ROOFTOP SEGMENTATION

Abstract. Convolutional Neural Networks have been widely introduced to building rooftop segmentation using satellite and aerial imagery. Preparing efficient training data is still among the critical issues on this topic. Therefore, adopting available annotated cross-domain multisource dataset is needed. This paper evaluates the performance of fusing the state-of-art deep learning neural network architectures for cross-domain building rooftop segmentation. We have selected three semantic image segmentation neural networks, including Swin transformer, OCRNet and HRNet. The predictions from these three neural networks are combined with majority voting, max value and union fusion techniques, a refined building rooftop segmentation mask is therefore delivered. The experiments on two benchmark datasets show that the proposed fusion techniques outperform single models and other state-of-art cross-domain segmentation approaches.

Multi-landmark environment analysis with reinforcement learning for pelvic abnormality detection and quantification.

Morphological abnormalities of the femoroacetabular (hip) joint are among the most common human musculoskeletal disorders and often develop asymptomatically at early easily treatable stages. In this paper, we propose an automated framework for landmark-based detection and quantification of hip abnormalities from magnetic resonance (MR) images. The framework relies on a novel idea of multi-landmark environment analysis with reinforcement learning. In particular, we merge the concepts of the graphical lasso and Morris sensitivity analysis with deep neural networks to quantitatively estimate the contribution of individual landmark and landmark subgroup locations to the other landmark locations. Convolutional neural networks for image segmentation are utilized to propose the initial landmark locations, and landmark detection is then formulated as a reinforcement learning (RL) problem, where each landmark-agent can adjust its position by observing the local MR image neighborhood and the locations of the most-contributive landmarks. The framework was validated on T1-, T2- and proton density-weighted MR images of 260 patients with the aim to measure the lateral center-edge angle (LCEA), femoral neck-shaft angle (NSA), and the anterior and posterior acetabular sector angles (AASA and PASA) of the hip, and derive the quantitative abnormality metrics from these angles. The framework was successfully tested using the UNet and feature pyramid network (FPN) segmentation architectures for landmark proposal generation, and the deep Q-network (DeepQN), deep deterministic policy gradient (DDPG), twin delayed deep deterministic policy gradient (TD3), and actor-critic policy gradient (A2C) RL networks for landmark position optimization. The resulting overall landmark detection error of 1.5 mm and angle measurement error of 1.4° indicates a superior performance in comparison to existing methods. Moreover, the automatically estimated abnormality labels were in 95% agreement with those generated by an expert radiologist.

Real-time space object tracklet extraction from telescope survey images with machine learning

In this study, a novel approach based on the U-Net deep neural network for image segmentation is leveraged for real-time extraction of tracklets from optical acquisitions. As in all machine learning (ML) applications, a series of steps is required for a working pipeline: dataset creation, preprocessing, training, testing, and post-processing to refine the trained network output. Online websites usually lack ready-to-use datasets; thus, an in-house application artificially generates 360 labeled images. Particularly, this software tool produces synthetic night-sky shots of transiting objects over a specified location and the corresponding labels: dual-tone pictures with black backgrounds and white tracklets. Second, both images and labels are downscaled in resolution and normalized to accelerate the training phase. To assess the network performance, a set of both synthetic and real images was inputted. After the preprocessing phase, real images were fine-tuned for vignette reduction and background brightness uniformity. Additionally, they are down-converted to eight bits. Once the network outputs labels, post-processing identifies the centroid right ascension and declination of the object. The average processing time per real image is less than 1.2 s; bright tracklets are easily detected with a mean centroid angular error of 0.25 deg in 75% of test cases with a 2 deg field-of-view telescope. These results prove that an ML-based method can be considered a valid choice when dealing with trail reconstruction, leading to acceptable accuracy for a fast image processing pipeline.

Identifying persons from iris images using neural networks for image segmentation and feature extraction

The problem of personal identification plays an important role in information security. In recent years, biometric methods of personal identification have become most relevant and promising. The article presents a study of a method for identifying a person from iris images using a neural network approach at the stages of segmentation and a feature representation from the data. A description of a dataset used to implement the segmentation stage using convolutional neural networks is presented and access to the segmentation masks of the entire dataset is provided. A method is proposed for extracting a feature representation of the data using pretrained convolutional neural networks to solve a problem of iris classification. A comparative analysis of methods for extracting iris features, including classical approaches and a neural network approach, has been carried out. A comparative analysis of classification methods is carried out, including classical machine learning algorithms, namely, support vector machines, random forest, and a k-nearest neighbors method. The results of experimental studies have shown the high quality of the classification based on the proposed approach.

An Expanded MLP Neural Network for Fast Angiodysplasia Lesions Segmentation in Capsule Endoscopy Images

In this paper, an expanded multilayer perceptron (EMLP) neural network is proposed to automatically segment angiodysplasia regions in wireless capsule endoscopy (WCE) images The main idea is to minimize the distance between the input image and the corresponding binary ground truth, i.e., the mask image. After the training phase, when a test image is given to the network, the lesion pixels will be close to “one” and the other pixels will be close to “zero” and finally, the lesion area can be segmented using thresholding. Since angiodysplasia lesions appear in images with different spectrums of red color, the classical MLP neural network cannot be trained with a wide range of red color, hence leads to undesirable network accuracy. To solve this problem, we proposed an EMLP neural network for image segmentation. In the EMLP neural network, neurons are divided into several groups, each of which is for learning a spectrum of the lesion. The EMLP is able to learn a wider range of red colors. The proposed method is able to segment WCE images containing angiodysplasia faster than the existing methods. Our investigation shows that our method also outperforms existing methods in terms of segmentation scores.

CT Segmentation of Dinosaur Fossils by Deep Learning

Recently, deep learning has reached significant advancements in various image-related tasks, particularly in medical sciences. Deep neural networks have been used to facilitate diagnosing medical images generated from various observation techniques including CT (computed tomography) scans. As a non-destructive 3D imaging technique, CT scan has also been widely used in paleontological research, which provides the solid foundation for taxon identification, comparative anatomy, functional morphology, etc. However, the labeling and segmentation of CT images are often laborious, prone to error, and subject to researchers own judgements. It is essential to set a benchmark in CT imaging processing of fossils and reduce the time cost from manual processing. Since fossils from the same localities usually share similar sedimentary environments, we constructed a dataset comprising CT slices of protoceratopsian dinosaurs from the Gobi Desert, Mongolia. Here we tested the fossil segmentation performances of U-net, a classic deep neural network for image segmentation, and constructed a modified DeepLab v3+ network, which included MobileNet v1 as feature extractor and practiced an atrous convolutional method that can capture features from various scales. The results show that deep neural network can efficiently segment protoceratopsian dinosaur fossils, which can save significant time from current manual segmentation. But further test on a dataset generated by other vertebrate fossils, even from similar localities, is largely limited.

PROCEDURE FOR USING NEURAL NETWORKS FOR SEGMENTATION OF RASTER IMAGES

Currently, means of semantic segmentation of images, based on the use of neural networks, are increasingly used in computer systems for various purposes. Despite significant successes in this field, one of the most important unsolved problems is the task of determining the type and parameters of convolutional neural networks, which are the basis of the encoder and decoder. As a result of the research, an appropriate procedure was developed that allows the neural network encoder and decoder to be adapted to the following conditions of the segmentation problem: image size, number of color channels, permissible minimum accuracy of segmentation, permissible maximum computational complexity of segmentation, the need to label segments, the need to select several segments, the need to select deformed, displaced and rotated objects, the maximum computational complexity of learning a neural network model is permissible; admissible training period of the neural network model. The implementation of the procedure of applying neural networks for image segmentation consists in the formation of the basic mathematical support, the construction of the main blocks and the general scheme of the procedure. The developed procedure was verified experimentally on examples of semantic segmentation of images containing objects such as a car. The obtained experimental results show that the application of the proposed procedure allows, avoiding complex long-term experiments, to build a neural network model that, with a sufficiently short training period, ensures the achievement of image segmentation accuracy of about 0.8, which corresponds to the best systems of a similar purpose. It is shown that the ways of further research in the direction of improving the methodological support of neural network segmentation of raster images should be correlated with the justified use of modern modules and mechanisms in the encoder and decoder, adapted to the significant conditions of the given task. For example, the use of the ResNet module allows you to increase the depth of the neural network due to the leveling of the gradient drop effect, and the Inception module provides a reduction in the number of weighting factors and the processing of objects of different sizes.

Automatic segmentation of satellite images based on a convolutional neural network

The article deals with neural network methods for automatic segmentation of satellite images. The most popular Python libraries with neural networks for image segmentation based on Keras and TensorFlow are compared. In addition, a modification of the U-Net architecture is proposed, which optimizes the operation of the algorithm for the set task. To use the algorithm in satellite navigation systems, the definition of the algorithm is to be accurate without computational resources. The result is a change in the values of the loss function and metric during training.

Detecting Small Anatomical Structures in 3D Knee MRI Segmentation by Fully Convolutional Networks

Accurately identifying the pixels of small organs or lesions from magnetic resonance imaging (MRI) has a critical impact on clinical diagnosis. U-net is the most well-known and commonly used neural network for image segmentation. However, the small anatomical structures in medical images cannot be well recognised by U-net. This paper explores the performance of the U-net architectures in knee MRI segmentation to find a relative structure that can obtain high accuracies for both small and large anatomical structures. To maximise the utilities of U-net architecture, we apply three types of components, residual blocks, squeeze-and-excitation (SE) blocks, and dense blocks, to construct four variants of U-net, namely U-net variants. Among these variants, our experiments show that SE blocks can improve the segmentation accuracies of small labels. We adopt DeepLabv3plus architecture for 3D medical image segmentation by equipping SE blocks based on this discovery. The experimental results show that U-net with SE block achieves higher accuracy in parts of small anatomical structures. In contrast, DeepLabv3plus with SE block performs better on the average dice coefficient of small and large labels.

Active Contours Driven by Cellular Neural Networks for Image Segmentation in Biomedical Application

Active Contours Driven by Cellular Neural Networks for Image Segmentation in Biomedical Application