Proposed Segmentation Method Research Articles

Threshold estimation based on local minima for nucleus and cytoplasm segmentation

BackgroundImage segmentation is the process of partitioning an image into separate objects or regions. It is an essential step in image processing to segment the regions of interest for further processing. We propose a method for segmenting the nuclei and cytoplasms from white blood cells (WBCs).MethodsInitially, the method computes an initial value based on the minimum and maximum values of the input image. Then, a histogram of the input image is computed and approximated to obtain function values. The method searches for the first local maximum and local minimum from the approximated function values in the order of increasing of knots sequence. We approximate the required threshold from the first local minimum and the computed initial value based on defined conditions. The threshold is applied to the input image to binarize it, and then post-processing is performed to obtain the final segmented nucleus. We segment the whole WBC before segmenting the cytoplasm depending on the complexity of the objects in the image. For WBCs that are well separated from red blood cells (RBCs), n thresholds are generated and then produce n thresholded images. Then, a standard Otsu method is used to binarize the average of the produced images. Morphological operations are applied on the binarized image, and then a single-pixel point from the segmented nucleus is used to segment the WBC. For images in which RBCs touch the WBCs, we segment the whole WBC using SLIC and watershed methods. The cytoplasm is obtained by subtracting the segmented nucleus from the segmented WBC.ResultsThe method is tested on two different public data sets and the results are compared to the state of art methods. The performance analysis shows that the proposed method segments the nucleus and cytoplasm well.ConclusionWe propose a method for nucleus and cytoplasm segmentation based on the local minima of the approximated function values from the image’s histogram. The method has demonstrated its utility in segmenting nuclei, WBCs, and cytoplasm, and the results are satisfactory.

Hybrid Deep Feature Fusion of 2D CNN and 3D CNN for Vestibule Segmentation from CT Images.

The accurate vestibule segmentation from CT images is essential to the quantitative analysis of the anatomical structure of the ear. However, it is a challenging task due to the tiny size, blur boundary, and drastic variations in shape and size. In this paper, according to the specific characteristics and segmentation requirements of the vestibule, a vestibule segmentation network with a hybrid deep feature fusion of 2D CNN and 3D CNN is proposed. First, a 2D CNN is designed to extract the intraslice features through multiple deep feature fusion strategies, including a convolutional feature fusion strategy for different receptive fields, a feature channel fusion strategy based on channel attention mechanism, and an encoder-decoder feature fusion strategy. Next, a 3D DenseUNet is designed to extract the interslice features. Finally, a hybrid feature fusion module is proposed to fuse the intraslice and interslice features to effectively exploit the context information, thus achieving the accurate segmentation of the vestibule structure. At present, there is no publicly available dataset for vestibule segmentation. Therefore, the proposed segmentation method is validated on two self-established datasets, namely, VestibuleDataSet and IEBL-DataSet. It has been compared with several state-of-the-art methods on the datasets, including the general DeeplabV3+ method and specific 3D DSD vestibule segmentation method. The experimental results show that our proposed method can achieve superior segmentation accuracy.

An FA-SegNet Image Segmentation Model Based on Fuzzy Attention and Its Application in Cardiac MRI Segmentation

Aiming at the medical images segmentation with low-recognition and high background noise, a deep convolution neural network image segmentation model based on fuzzy attention mechanism is proposed, which is called FA-SegNet. It takes SegNet as the basic framework. In the down-sampling module for image feature extraction, a fuzzy channel-attention module is added to strengthen the discrimination of different target regions. In the up-sampling module for image size restoration and multi-scale feature fusion, a fuzzy spatial-attention module is added to reduce the loss of image details and expand the receptive field. In this paper, fuzzy cognition is introduced into the feature fusion of CNNs. Based on the attention mechanism, fuzzy membership is used to re-calibrate the importance of the pixel value in local regions. It can strengthen the distinguishing ability of image features, and the fusion ability of the contextual information, which improves the segmentation accuracy of the target regions. Taking MRI segmentation as an experimental example, multiple targets such as the left ventricles, right ventricles, and left ventricular myocardium are selected as the segmentation targets. The pixels accuracy is 92.47%, the mean intersection to union is 86.18%, and the Dice coefficient is 92.44%, which are improved compared with other methods. It verifies the accuracy and applicability of the proposed method for the medical images segmentation, especially the targets with low-recognition and serious occlusion.

Deep Learning Segmentation in 2D X-ray Images and Non-Rigid Registration in Multi-Modality Images of Coronary Arteries.

X-ray angiography is commonly used in the diagnosis and treatment of coronary artery disease with the advantage of visualization of the inside of blood vessels in real-time. However, it has several disadvantages that occur in the acquisition process, which causes inconvenience and difficulty. Here, we propose a novel segmentation and nonrigid registration method to provide useful real-time assistive images and information. A convolutional neural network is used for the segmentation of coronary arteries in 2D X-ray angiography acquired from various angles in real-time. To compensate for errors that occur during the 2D X-ray angiography acquisition process, 3D CT angiography is used to analyze the topological structure. A novel energy function-based 3D deformation and optimization is utilized to implement real-time registration. We evaluated the proposed method for 50 series from 38 patients by comparing the ground truth. The proposed segmentation method showed that Precision, Recall, and F1 score were 0.7563, 0.6922, and 0.7176 for all vessels, 0.8542, 0.6003, and 0.7035 for markers, and 0.8897, 0.6389, and 0.7386 for bifurcation points, respectively. In the nonrigid registration method, the average distance of 0.8705, 1.06, and 1. 5706 mm for all vessels, markers, and bifurcation points was achieved. The overall process execution time was 0.179 s.

OCT Retinal and Choroidal Layer Instance Segmentation Using Mask R-CNN.

Optical coherence tomography (OCT) of the posterior segment of the eye provides high-resolution cross-sectional images that allow visualization of individual layers of the posterior eye tissue (the retina and choroid), facilitating the diagnosis and monitoring of ocular diseases and abnormalities. The manual analysis of retinal OCT images is a time-consuming task; therefore, the development of automatic image analysis methods is important for both research and clinical applications. In recent years, deep learning methods have emerged as an alternative method to perform this segmentation task. A large number of the proposed segmentation methods in the literature focus on the use of encoder–decoder architectures, such as U-Net, while other architectural modalities have not received as much attention. In this study, the application of an instance segmentation method based on region proposal architecture, called the Mask R-CNN, is explored in depth in the context of retinal OCT image segmentation. The importance of adequate hyper-parameter selection is examined, and the performance is compared with commonly used techniques. The Mask R-CNN provides a suitable method for the segmentation of OCT images with low segmentation boundary errors and high Dice coefficients, with segmentation performance comparable with the commonly used U-Net method. The Mask R-CNN has the advantage of a simpler extraction of the boundary positions, especially avoiding the need for a time-consuming graph search method to extract boundaries, which reduces the inference time by 2.5 times compared to U-Net, while segmenting seven retinal layers.

Scribble-Supervised Meibomian Glands Segmentation in Infrared Images

Infrared imaging is currently the most effective clinical method to evaluate the morphology of the meibomian glands (MGs) in patients. As an important indicator for monitoring the development of MG dysfunction, it is necessary to accurately measure gland-drop and gland morphology. Although there are existing methods for automatic segmentation of MGs using deep learning frameworks, they require fully annotated ground-truth labels for training, which is time-consuming and laborious. In this article, we proposed a new scribble-supervised deep learning framework for segmenting the MGs, which only requires easily attainable scribble annotations for training. To cope with the shortage of supervision and regularize the network, a transformation consistent strategy is incorporated, which requires the prediction to follow the same transformation if a transform is performed on an input image of the network. The proposed segmentation method consists of two stages. In the first stage, a U-Net network is used to obtain the meibomian region segmentation map. In the second stage, we concentrate on segmenting glands in the meibomian region. We utilize the gradient prior information of the original image at the decoder part of the segmentation network, which can coarsely locate the target contour. We automatically generate reliable labels using the exponential moving average of the predictions during training and filter out the unreliable pseudo-label by uncertainty threshold. Experimental results on a local MG dataset and two other public medical image datasets demonstrate the effectiveness of the proposed segmentation framework.

Evaluation of Diagnostic Value of Mediastinum for Differentiation of Drug Sensitive, Multi and Extensively Drug Resistant Tuberculosis Using Chest X-Rays

Background and ObjectiveThe rise of Drug Resistant Tuberculosis (DR TB), particularly Multi DR (MDR), and Extensively DR (XDR) has reduced the rate of control of the disease. Computer aided diagnosis using Chest X-rays (CXRs) can help in mass screening and timely diagnosis of DR TB, which is essential to administer proper treatment regimens. In CXRs, lungs and mediastinum are two significant regions which contain the information about the likelihood of DR TB. The objective of this work is to analyze the shape characteristics of lungs and mediastinum to improve the diagnostics accuracy for differentiation of Drug Sensitive (DS), MDR and XDR TB using computer aided diagnostics system. MethodsThe CXR images of DS and DR TB patients are obtained from a public database. The lung fields are segmented from the CXRs using Reaction Diffusion Level Set Evolution. Mediastinum is segmented from the delineated lung masks using Chan Vese model. The shape features from each lung and mediastinum masks are extracted and analysed. The discriminative power of individual and combination of both lung and mediastinum features are evaluated using machine learning techniques for classification of DS vs MDR, MDR vs XDR and DS vs XDR TB images. The performances of classifiers are compared using standard metrics. ResultsThe proposed segmentation methods are able to delineate lungs and mediastinum from the CXR images. The extracted lung and mediastinum features are found to be statistically significant (p < 0.05) for differentiation of DS and DR TB conditions. Using the combination of both lung and mediastinum features, Multi-Layer Perceptron classifier achieves maximum F-measure of 82.4%, 81.0% and 87.0% for differentiation of DS vs MDR, MDR vs XDR and DS vs XDR, respectively. ConclusionAnalysis of mediastinum along with the lungs in chest X-rays could improve the diagnostic performance for differentiation of drug sensitive and resistant TB conditions. The proposed methodology is able to differentiate DS, MDR and XDR TB, and found to be clinically relevant. Hence, this work is useful for computer-based early detection of DS and DR TB conditions.

Boosting Marine Predators Algorithm by Salp Swarm Algorithm for Multilevel Thresholding Image Segmentation.

Pixel rating is considered one of the commonly used critical factors in digital image processing that depends on intensity. It is used to determine the optimal image segmentation threshold. In recent years, the optimum threshold has been selected with great interest due to its many applications. Several methods have been used to find the optimum threshold, including the Otsu and Kapur methods. These methods are appropriate and easy to implement to define a single or bi-level threshold. However, when they are extended to multiple levels, they will cause some problems, such as long time-consuming, the high computational cost, and the needed improvement in their accuracy. To avoid these problems and determine the optimal multilevel image segmentation threshold, we proposed a hybrid Marine Predators Algorithm (MPA) with Salp Swarm Algorithm (SSA) to determine the optimal multilevel threshold image segmentation MPASSA. The obtained solutions of the proposed method are represented using the image histogram. Several standard evaluation measures, such as (the fitness function, time consumer, Peak Signal-to-Noise Ratio, Structural Similarity Index, etc.…) are employed to evaluate the proposed segmentation method’s effectiveness. Several benchmark images are used to validate the proposed algorithm’s performance (MPASSA). The results showed that the proposed MPASSA got better results than other well-known optimization algorithms published in the literature.

Defect detection in rotating machine elements by using an improved image segmentation technique

In industrial applications rotating machinery is widely used. Therefore it becomes necessary to diagnose the health of rotating machinery and detect any fault during their working. The pre-determination of defects can alert the operators and overall system for any type of incident and loss. Generally, fault in the rotating machine causes a rise in temperature of the rotating elements of the machine during their working. In this work, the image segmentation technique is used on thermal images of rotating elements of a machine to identify and detect the defects in them. The main focus of this investigation is to apply linear transformation and improved hyper smoothing based local binary pattern(LTIHLBP) image segmentation technique for defect detection and compare its performance with other conventional defect detection techniques namely improved hyper-smoothing function based local binary pattern (ILHBP), Sobel and Canny. In this paper, three tests are performed to identify the effectiveness of the proposed method. The first test is performed on synthetic images to identify the accuracy and Jaccard similarity index(JSI) of the proposed segmentation method, the second test is performed on healthy moving element images and the third test is performed on faulty moving element images. The outcome of this investigation revealed that the proposed method of LTIHLBP outperforms the conventional image segmentation-based defect detection techniques and found to be capable and accurate in diagnosing and detecting defects.

A systematic extraction of glacial lakes for satellite imagery using deep learning based technique

For outburst susceptibility assessment of glacial lakes, the accurate extraction of glacial lakes region from satellite image is essential. Several techniques are introduced to extract glacial lakes over the years, such as based on image pixel or different thresholds and object-based statistics. These methods require manual optimization parameters for accurate extraction of the glacial lake’s region. Hence, in this work, the deep learning (DL) models are utilized to segment the water and non-water bodies from satellite imagery. DL techniques are successful in many research areas for classification problems, but these techniques are relatively new for the extraction of glacial lakes. The main challenges in implementing DL techniques for the extraction of the glacial lakes are (i) preserving the semantic features of the image without loss during max-pooling operation, (ii) huge dataset requirement for proper training. To overcome the problems mentioned above, a customized architecture of the U-Net named as glacial lakes U-Net (GLU-Net) with various encoders is proposed in this work for effective segmentation of the glacial lakes. The proposed method for segmentation is also compared with the existing DL techniques in this work. The DL models are evaluated on Imja, Chandra, and Bhaga glacier regions datasets collected from Landsat 8 satellite imagery. The average sensitivity (Sen), specificity (Spe), and dice coefficient (Dice) for GLU-Net are 92.23%, 99.81%, and 93.46%. These are the three measures for an average of Imja, Chandra and Bhaga glacier regions. The qualitative and quantitative performance analysis shows the significant improvement of the proposed technique compared to other DL techniques for extraction of glacial lake’s region.

Automated Knee MR Images Segmentation of Anterior Cruciate Ligament Tears.

The anterior cruciate ligament (ACL) is one of the main stabilizer parts of the knee. ACL injury leads to causes of osteoarthritis risk. ACL rupture is common in the young athletic population. Accurate segmentation at an early stage can improve the analysis and classification of anterior cruciate ligaments tears. This study automatically segmented the anterior cruciate ligament (ACL) tears from magnetic resonance imaging through deep learning. The knee mask was generated on the original Magnetic Resonance (MR) images to apply a semantic segmentation technique with convolutional neural network architecture U-Net. The proposed segmentation method was measured by accuracy, intersection over union (IoU), dice similarity coefficient (DSC), precision, recall and F1-score of 98.4%, 99.0%, 99.4%, 99.6%, 99.6% and 99.6% on 11451 training images, whereas on the validation images of 3817 was, respectively, 97.7%, 93.8%,96.8%, 96.5%, 97.3% and 96.9%. We also provide dice loss of training and test datasets that have remained 0.005 and 0.031, respectively. The experimental results show that the ACL segmentation on JPEG MRI images with U-Nets achieves accuracy that outperforms the human segmentation. The strategy has promising potential applications in medical image analytics for the segmentation of knee ACL tears for MR images.

Bayberry segmentation in a complex environment based on a multi-module convolutional neural network

Automatic bayberry picking can substantially reduce labor costs and improve picking efficiency in an orchard management system. Nowadays, an automatic picking system mainly relies on machine vision to segment bayberry fruit from the background. Most existing methods are carried out in an environment where the light intensity is relatively fixed and the bayberries are unobstructed. However, due to the complexity of the growing environment, including variations in lighting and widespread occlusion, segmentation accuracy is quite limited, which affects the large-scale application of automatic picking systems. Aiming at these issues, in this study, a bayberry segmentation method based on a multi-module convolutional neural network is proposed. First, the bayberry images in a real scene were collected and preprocessed to form a dataset. Then, a convolutional neural network was constructed, with an image correction module to improve the network’s robustness to natural ambient lighting. Finally, a shape completion module with a puzzle algorithm was utilized to overcome the occlusion in the natural environment. The experimental results show that the average precision of the proposed method for semantic segmentation and instance segmentation of bayberry fruit can reach 0.913 and 0.755, respectively, which outperforms the existing methods and has important significance for automatic picking in orchards.

Structure attention co-training neural network for neovascularization segmentation in intravascular optical coherence tomography.

To development and validate a neovascularization (NV) segmentation model in intravascular optical coherence tomography (IVOCT) through deep learning methods. A total of 1950 2D slices of 70 IVOCT pullbacks were used in our study. We randomly selected 1273 2D slices from 44 patients as the training set, 379 2D slices from 11 patients as the validation set, and 298 2D slices from the last 15 patients as the testing set. Automatic NV segmentation is quite challenging, as it must address issues of speckle noise, shadow artifacts, high distribution variation, etc. To meet these challenges, a new deep learning-based segmentation method is developed based on a co-training architecture with an integrated structural attention mechanism. Co-training is developed to exploit the features of three consecutive slices. The structural attention mechanism comprises spatial and channel attention modules and is integrated into the co-training architecture at each up-sampling step. A cascaded fixed network is further incorporated to achieve segmentation at the image level in a coarse-to-fine manner. Extensive experiments were performed involving a comparison with several state-of-the-art deep learning-based segmentation methods. Moreover, the consistency of the results with those of manual segmentation was also investigated. Our proposed NV automatic segmentation method achieved the highest correlation with the manual delineation by interventional cardiologists (the Pearson correlation coefficient is 0.825). In this work, we proposed a co-training architecture with an integrated structural attention mechanism to segment NV in IVOCT images. The good agreement between our segmentation results and manual segmentation indicates that the proposed method has great potential for application in the clinical investigation of NV-related plaque diagnosis and treatment.

Deep Learning-Aided Automatic Contouring of Clinical Target Volumes for Radiotherapy in Breast Cancer After Modified Radical Mastectomy

Purpose: The aim of this study is to develop a practicable automatic clinical target volume (CTV) delineation method for radiotherapy of breast cancer after modified radical mastectomy.Methods: Unlike breast conserving surgery, the radiotherapy CTV for modified radical mastectomy involves several regions, including CTV in the chest wall (CTVcw), supra- and infra-clavicular region (CTVsc), and internal mammary lymphatic region (CTVim). For accurate and efficient segmentation of the CTVs in radiotherapy of breast cancer after modified radical mastectomy, a multi-scale convolutional neural network with an orientation attention mechanism is proposed to capture the corresponding features in different perception fields. A channel-specific local Dice loss, alongside several data augmentation methods, is also designed specifically to stabilize the model training and improve the generalization performance of the model. The segmentation performance is quantitatively evaluated by statistical metrics and qualitatively evaluated by clinicians in terms of consistency and time efficiency.Results: The proposed method is trained and evaluated on the self-collected dataset, which contains 110 computed tomography scans from patients with breast cancer who underwent modified mastectomy. The experimental results show that the proposed segmentation method achieved superior performance in terms of Dice similarity coefficient (DSC), Hausdorff distance (HD) and Average symmetric surface distance (ASSD) compared with baseline approaches.Conclusion: Both quantitative and qualitative evaluation results demonstrated that the specifically designed method is practical and effective in automatic contouring of CTVs for radiotherapy of breast cancer after modified radical mastectomy. Clinicians can significantly save time on manual delineation while obtaining contouring results with high consistency by employing this method.

Construction of a Computer-Aided Analysis System for Orthopedic Diseases Based on High-Frequency Ultrasound Images.

The area of medical diagnosis has been transformed by computer-aided diagnosis (CAD). With the advancement of technology and the widespread availability of medical data, CAD has gotten a lot of attention, and numerous methods for predicting different pathological diseases have been created. Ultrasound (US) is the safest clinical imaging method; therefore, it is widely utilized in medical and healthcare settings with computer-aided systems. However, owing to patient movement and equipment constraints, certain artefacts make identification of these US pictures challenging. To enhance the quality of pictures for classification and segmentation, certain preprocessing techniques are required. Hence, we proposed a three-stage image segmentation method using U-Net and Iterative Random Forest Classifier (IRFC) to detect orthopedic diseases in ultrasound images efficiently. Initially, the input dataset is preprocessed using Enhanced Wiener Filter for image denoising and image enhancement. Then, the proposed segmentation method is applied. Feature extraction is performed by transform-based analysis. Finally, obtained features are reduced to optimal subset using Principal Component Analysis (PCA). The classification is done using the proposed Iterative Random Forest Classifier. The proposed method is compared with the conventional performance measures like accuracy, specificity, sensitivity, and dice score. The proposed method is proved to be efficient for detecting orthopedic diseases in ultrasound images than the conventional methods.

An Automatic Deep Segmentation Network for Pixel-Level Welding Defect Detection

Accurate welding defect location is of great significance to modern manufacturing, which could be used for accurate quality evaluation and precise repairing decision-making basis of different products. Nevertheless, accurate welding defect location is still a challenging task due to some complex factors, such as complex backgrounds, low contrast, weak texture, and class imbalance issue. Recently, deep learning has got great development due to its strong feature expression ability, which has been widely applied into defect detection, but it still exists certain shortcomings on segmentation tasks with the class unbalanced issue or microdefects. To address these issues, with the encoder&#x2013;decoder network architecture, a novel welding defect location method is proposed with an attention-guided segmentation network. To reduce the contextual information loss of the deep encoder module after multiple convolution and pooling operations, a multiscale feature fusion block is proposed to embed into a U-shaped network (U-Net) to acquire more information. On the basis, combined with a bidirectional convolutional long short-term memory (BiConvLSTM) block, an improved attention block is integrated into the skip connections between the encoder path and the decoder path to capture the global, long-range contexts and emphasize target regions, contributing to locate welding defect areas and enhance the segmentation ability on microdefects. Meanwhile, to address the foreground&#x2013;background class imbalance issue, a hybrid loss function combined with binary cross-entropy (BCE) and <inline-formula> <tex-math notation="LaTeX">$D_{\text {ice}}$ </tex-math></inline-formula> loss functions is proposed to effectively utilize their unique excellent characteristics for accurate defect segmentation. Experiment results on the public <i>GDXray</i> dataset show that the proposed segmentation method could obtain a competitive segmentation performance compared with other advanced segmentation models.

An Efficient Breast Cancer Detection Framework for Medical Diagnosis Applications

Breast cancer is the most common type of cancer, and it is the reason for cancer death toll in women in recent years. Early diagnosis is essential to handle breast cancer patients for treatment at the right time. Screening with mammography is the preferred examination for breast cancer, as it is available worldwide and inexpensive. Computer-Aided Detection (CAD) systems are used to analyze medical images to detect breast cancer, early. The death rate of cancer patients has decreased by detecting tumors early and having appropriate treatment after operations. Processing of mammogram images has four main steps: pre-processing, segmentation of the region of interest, feature extraction and classification of the images into normal or abnormal classes. This paper presents an efficient framework for processing of mammogram images and introduces an algorithm for segmentation of the images to detect masses. The pre-processing step of mammogram images includes removal of digitization noise using a 2D median filter, removal of artifacts using morphological operations, and contrast enhancement using a fuzzy enhancement technique. The proposed fuzzy image enhancement technique is analyzed and compared with conventional techniques based on an Enhancement Measure (EME) and local contrast metrics. The comparison shows an outstanding performance of the proposed technique from the visual and numerical perspectives. The segmentation process is performed using Otsu's multiple thresholding method. This method segments the image regions into five classes with variable intensities using four thresholds. Its effectiveness is measured based on visual quality of the segmentation output, as it gives details about the image and positions of masses. The performance of the proposed framework is measured using Dice coefficient, Hausdorff, and Peak Signal-to-Noise Ratio (PSNR) metrics. The segmented tumor region with the proposed segmentation method is 81% of the ground truth region provided by an expert. Hence, the proposed framework achieves promising results for aiding radiologists in screening of mammograms, accurately.

Magnetic Resonance Imaging Image-Based Segmentation of Brain Tumor Using the Modified Transfer Learning Method.

The goal of this study was to improve overall brain tumor segmentation (BraTS) accuracy. In this study, a form of convolutional neural network called three-dimensional (3D) U-Net was utilized to segment various tumor regions on brain 3D magnetic resonance imaging images using a transfer learning technique. The dataset used for this study was obtained from the multimodal BraTS challenge. The total number of studies was 2240, obtained from BraTS 2018, BraTS 2019, BraTS 2020, and BraTS 2021 challenges, and each study had five series: T1, contrast-enhanced-T1, Flair, T2, and segmented mask file (seg), all in Neuroimaging Informatics Technology Initiative (NIFTI) format. The proposed method employs a 3D U-Net that was trained separately on each of the four datasets by transferring weights across them. The overall training accuracy, validation accuracy, mean dice coefficient, and mean intersection over union achieved were 99.35%, 98.93%, 0.9875%, and 0.8738%, respectively. The proposed method for tumor segmentation outperforms the existing method.

Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks

Medical image fusion aims to integrate the complementary information captured by images of different modalities into a more informative composite image. However, current study on medical image fusion suffers from several drawbacks: 1) Existing methods are mostly designed for 2-D slice fusion, and they tend to lose spatial contextual information when fusing medical images with volumetric structure slice by slice individually. 2) The few existing 3-D medical image fusion methods fail in considering the characteristics of source modalities sufficiently, leading to the loss of important modality information. 3) Most existing works concentrate on pursuing good performance on visual perception and objective evaluation, while there is a severe lack of clinical problem-oriented study. In this paper, to address these issues, we propose a multimodal MRI volumetric data fusion method based on an end-to-end convolutional neural network (CNN). In our network, an attention-based multimodal feature fusion (MMFF) module is presented for more effective feature learning. In addition, a specific loss function that considers the characteristics of different MRI modalities is designed to preserve the modality information. Experimental results demonstrate that the proposed method can obtain more competitive results on both visual quality and objective assessment, when compared with some representative 3-D and 2-D medical image fusion methods. We further verify the significance of the proposed method for brain tumor segmentation by enriching the input modalities, and the results show that it is helpful to improve the segmentation accuracy. The source code of our fusion method is available at https://github.com/yuliu316316/3D-CNN-Fusion.

Geometry-Aware Segmentation of Remote Sensing Images via Joint Height Estimation

Recent studies have shown the benefits of using additional elevation data [e.g., digital surface model (DSM) or normalized DSM (nDSM)] for enhancing the performance of the semantic labeling of aerial images. However, previous methods mostly adopt 3-D elevation information as additional inputs, while, in many real-world applications, one does not have the corresponding DSM images at hand, and the spatial resolution of acquired DSM images usually does not match the aerial images. To alleviate this data constraint and also take advantage of 3-D elevation information, in this letter, a geometry-aware segmentation model is introduced to achieve accurate semantic labeling of aerial images via joint height estimation. Instead of using a single-stream encoder-decoder network for semantic labeling, we design a separate decoder branch to predict the height map and use the DSM images as side supervision to train this newly designed decoder branch. With the newly designed decoder branch, our model can distill the 3-D geometric features from 2-D appearance features under the supervision of ground-truth DSM images. Moreover, we develop a new geometry-aware convolution module that fuses the 3-D geometric features from the height decoder branch and the 2-D contextual features from the semantic segmentation branch. The fused feature embeddings can produce geometry-aware segmentation maps with enhanced performance. Our model is trained with DSM images as side supervision, while, in the inference stage, it does not require DSM data and directly predicts the semantic labels. Experiments on International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen and Potsdam data sets demonstrate the effectiveness of the proposed method for the semantic segmentation of aerial images.