Deep Learning-based Segmentation Methods Research Articles

The InterVision Framework: An Enhanced Fine-Tuning Deep Learning Strategy for Auto-Segmentation in Head and Neck.

Adaptive radiotherapy (ART) workflows are increasingly adopted to achieve dose escalation and tissue sparing under dynamic anatomical conditions. However, recontouring and time constraints hinder the implementation of real-time ART workflows. Various auto-segmentation methods, including deformable image registration, atlas-based segmentation, and deep learning-based segmentation (DLS), have been developed to address these challenges. Despite the potential of DLS methods, clinical implementation remains difficult due to the need for large, high-quality datasets to ensure model generalizability. This study introduces an InterVision framework for segmentation. The InterVision framework can interpolate or create intermediate visuals between existing images to generate specific patient characteristics. The InterVision model is trained in two steps: (1) generating a general model using the dataset, and (2) tuning the general model using the dataset generated from the InterVision framework. The InterVision framework generates intermediate images between existing patient image slides using deformable vectors, effectively capturing unique patient characteristics. By creating a more comprehensive dataset that reflects these individual characteristics, the InterVision model demonstrates the ability to produce more accurate contours compared to general models. Models are evaluated using the volumetric dice similarity coefficient (VDSC) and the Hausdorff distance 95% (HD95%) for 18 structures in 20 test patients. As a result, the Dice score was 0.81 ± 0.05 for the general model, 0.82 ± 0.04 for the general fine-tuning model, and 0.85 ± 0.03 for the InterVision model. The Hausdorff distance was 3.06 ± 1.13 for the general model, 2.81 ± 0.77 for the general fine-tuning model, and 2.52 ± 0.50 for the InterVision model. The InterVision model showed the best performance compared to the general model. The InterVision framework presents a versatile approach adaptable to various tasks where prior information is accessible, such as in ART settings. This capability is particularly valuable for accurately predicting complex organs and targets that pose challenges for traditional deep learning algorithms.

UNSEG: unsupervised segmentation of cells and their nuclei in complex tissue samples

Multiplexed imaging technologies have made it possible to interrogate complex tissue microenvironments at sub-cellular resolution within their native spatial context. However, proper quantification of this complexity requires the ability to easily and accurately segment cells into their sub-cellular compartments. Within the supervised learning paradigm, deep learning-based segmentation methods demonstrating human level performance have emerged. However, limited work has been done in developing such generalist methods within the unsupervised context. Here we present an easy-to-use unsupervised segmentation (UNSEG) method that achieves deep learning level performance without requiring any training data via leveraging a Bayesian-like framework, and nucleus and cell membrane markers. We show that UNSEG is internally consistent and better at generalizing to the complexity of tissue morphology than current deep learning methods, allowing it to unambiguously identify the cytoplasmic compartment of a cell, and localize molecules to their correct sub-cellular compartment. We also introduce a perturbed watershed algorithm for stably and automatically segmenting a cluster of cell nuclei into individual nuclei that increases the accuracy of classical watershed. Finally, we demonstrate the efficacy of UNSEG on a high-quality annotated gastrointestinal tissue dataset we have generated, on publicly available datasets, and in a range of practical scenarios.

Enhanced IDOL segmentation framework using personalized hyperspace learning IDOL.

Adaptive radiotherapy (ART) workflows have been increasingly adopted to achieve dose escalation and tissue sparing under shifting anatomic conditions, but the necessity of recontouring and the associated time burden hinders a real-time or online ART workflow. In response to this challenge, approaches to auto-segmentation involving deformable image registration, atlas-based segmentation, and deep learning-based segmentation (DLS) have been developed. Despite the particular promise shown by DLS methods, implementing these approaches in a clinical setting remains a challenge, namely due to the difficulty of curating a data set of sufficient size and quality so as to achieve generalizability in a trained model. To address this challenge, we have developed an intentional deep overfit learning (IDOL) framework tailored to the auto-segmentation task. However, certain limitations were identified, particularly the insufficiency of the personalized dataset to effectively overfit the model. In this study, we introduce a personalized hyperspace learning (PHL)-IDOL segmentation framework capable of generating datasets that induce the model to overfit specific patient characteristics for medical image segmentation. The PHL-IDOL model is trained in two stages. In the first, a conventional, general model is trained with a diverse set of patient data (n=100 patients) consisting of CT images and clinical contours. Following this, the general model is tuned with a data set consisting of two components: (a) selection of a subset of the patient data (m<n) using the similarity metrics (mean square error (MSE), peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and the universal quality image index (UQI) values); (b) adjust the CT and the clinical contours using a deformed vector generated from the reference patient and the selected patients using (a). After training, the general model, the continual model, the conventional IDOL model, and the proposed PHL-IDOL model were evaluated using the volumetric dice similarity coefficient (VDSC) and the Hausdorff distance 95% (HD95%) computed for 18 structures in 20 test patients. Implementing the PHL-IDOL framework resulted in improved segmentation performance for each patient. The Dice scores increased from 0.81 0.05 with the general model, 0.83 for the continual model, 0.83 for the conventional IDOL model to an average of 0.87 with the PHL-IDOL model. Similarly, the Hausdorff distance decreased from 3.06 with the general model, 2.84 for the continual model, 2.79 for the conventional IDOL model and 2.36 for the PHL-IDOL model. All the standard deviations were decreased by nearly half of the values comparing the general model and the PHL-IDOL model. The PHL-IDOL framework applied to the auto-segmentation task achieves improved performance compared to the general DLS approach, demonstrating the promise of leveraging patient-specific prior information in a task central to online ART workflows.

Finite element models with automatic computed tomography bone segmentation for failure load computation

Bone segmentation is an important step to perform biomechanical failure load simulations on in-vivo CT data of patients with bone metastasis, as it is a mandatory operation to obtain meshes needed for numerical simulations. Segmentation can be a tedious and time consuming task when done manually, and expert segmentations are subject to intra- and inter-operator variability. Deep learning methods are increasingly employed to automatically carry out image segmentation tasks. These networks usually need to be trained on a large image dataset along with the manual segmentations to maximize generalization to new images, but it is not always possible to have access to a multitude of CT-scans with the associated ground truth. It then becomes necessary to use training techniques to make the best use of the limited available data. In this paper, we propose a dedicated pipeline of preprocessing, deep learning based segmentation method and post-processing for in-vivo human femurs and vertebrae segmentation from CT-scans volumes. We experimented with three U-Net architectures and showed that out-of-the-box models enable automatic and high-quality volume segmentation if carefully trained. We compared the failure load simulation results obtained on femurs and vertebrae using either automatic or manual segmentations and studied the sensitivity of the simulations on small variations of the automatic segmentation. The failure loads obtained using automatic segmentations were comparable to those obtained using manual expert segmentations for all the femurs and vertebrae tested, demonstrating the effectiveness of the automated segmentation approach for failure load simulations.

Effects of different ground segmentation methods on the accuracy of UAV-based canopy volume measurements.

The nonuniform distribution of fruit tree canopies in space poses a challenge for precision management. In recent years, with the development of Structure from Motion (SFM) technology, unmanned aerial vehicle (UAV) remote sensing has been widely used to measure canopy features in orchards to balance efficiency and accuracy. A pipeline of canopy volume measurement based on UAV remote sensing was developed, in which RGB and digital surface model (DSM) orthophotos were constructed from captured RGB images, and then the canopy was segmented using U-Net, OTSU, and RANSAC methods, and the volume was calculated. The accuracy of the segmentation and the canopy volume measurement were compared. The results show that the U-Net trained with RGB and DSM achieves the best accuracy in the segmentation task, with mean intersection of concatenation (MIoU) of 84.75% and mean pixel accuracy (MPA) of 92.58%. However, in the canopy volume estimation task, the U-Net trained with DSM only achieved the best accuracy with Root mean square error (RMSE) of 0.410 m3, relative root mean square error (rRMSE) of 6.40%, and mean absolute percentage error (MAPE) of 4.74%. The deep learning-based segmentation method achieved higher accuracy in both the segmentation task and the canopy volume measurement task. For canopy volumes up to 7.50 m3, OTSU and RANSAC achieve an RMSE of 0.521 m3 and 0.580 m3, respectively. Therefore, in the case of manually labeled datasets, the use of U-Net to segment the canopy region can achieve higher accuracy of canopy volume measurement. If it is difficult to cover the cost of data labeling, ground segmentation using partitioned OTSU can yield more accurate canopy volumes than RANSAC.

WSPolyp-SAM: Weakly Supervised and Self-Guided Fine-Tuning of SAM for Colonoscopy Polyp Segmentation

Ensuring precise segmentation of colorectal polyps holds critical importance in the early diagnosis and treatment of colorectal cancer. Nevertheless, existing deep learning-based segmentation methods are fully supervised, requiring extensive, precise, manual pixel-level annotation data, which leads to high annotation costs. Additionally, it remains challenging to train large-scale segmentation models when confronted with limited colonoscopy data. To address these issues, we introduce the general segmentation foundation model—the Segment Anything Model (SAM)—into the field of medical image segmentation. Fine-tuning the foundation model is an effective approach to tackle sample scarcity. However, current SAM fine-tuning techniques still rely on precise annotations. To overcome this limitation, we propose WSPolyp-SAM, a novel weakly supervised approach for colonoscopy polyp segmentation. WSPolyp-SAM utilizes weak annotations to guide SAM in generating segmentation masks, which are then treated as pseudo-labels to guide the fine-tuning of SAM, thereby reducing the dependence on precise annotation data. To improve the reliability and accuracy of pseudo-labels, we have designed a series of enhancement strategies to improve the quality of pseudo-labels and mitigate the negative impact of low-quality pseudo-labels. Experimental results on five medical image datasets demonstrate that WSPolyp-SAM outperforms current fully supervised mainstream polyp segmentation networks on the Kvasir-SEG, ColonDB, CVC-300, and ETIS datasets. Furthermore, by using different amounts of training data in weakly supervised and fully supervised experiments, it is found that weakly supervised fine-tuning can save 70% to 73% of annotation time costs compared to fully supervised fine-tuning. This study provides a new perspective on the combination of weakly supervised learning and SAM models, significantly reducing annotation time and offering insights for further development in the field of colonoscopy polyp segmentation.

An automated two-stage approach to kidney and tumor segmentation in CT imaging.

The incidence of kidney tumors is progressively increasing each year. The precision of segmentation for kidney tumors is crucial for diagnosis and treatment. To enhance accuracy and reduce manual involvement, propose a deep learning-based method for the automatic segmentation of kidneys and kidney tumors in CT images. The proposed method comprises two parts: object detection and segmentation. We first use a model to detect the position of the kidney, then narrow the segmentation range, and finally use an attentional recurrent residual convolutional network for segmentation. Our model achieved a kidney dice score of 0.951 and a tumor dice score of 0.895 on the KiTS19 dataset. Experimental results show that our model significantly improves the accuracy of kidney and kidney tumor segmentation and outperforms other advanced methods. The proposed method provides an efficient and automatic solution for accurately segmenting kidneys and renal tumors on CT images. Additionally, this study can assist radiologists in assessing patients' conditions and making informed treatment decisions.

Synthetic Optical Coherence Tomography Angiographs for Detailed Retinal Vessel Segmentation Without Human Annotations.

Optical coherence tomography angiography (OCTA) is a non-invasive imaging modality that can acquire high-resolution volumes of the retinal vasculature and aid the diagnosis of ocular, neurological and cardiac diseases. Segmenting the visible blood vessels is a common first step when extracting quantitative biomarkers from these images. Classical segmentation algorithms based on thresholding are strongly affected by image artifacts and limited signal-to-noise ratio. The use of modern, deep learning-based segmentation methods has been inhibited by a lack of large datasets with detailed annotations of the blood vessels. To address this issue, recent work has employed transfer learning, where a segmentation network is trained on synthetic OCTA images and is then applied to real data. However, the previously proposed simulations fail to faithfully model the retinal vasculature and do not provide effective domain adaptation. Because of this, current methods are unable to fully segment the retinal vasculature, in particular the smallest capillaries. In this work, we present a lightweight simulation of the retinal vascular network based on space colonization for faster and more realistic OCTA synthesis. We then introduce three contrast adaptation pipelines to decrease the domain gap between real and artificial images. We demonstrate the superior segmentation performance of our approach in extensive quantitative and qualitative experiments on three public datasets that compare our method to traditional computer vision algorithms and supervised training using human annotations. Finally, we make our entire pipeline publicly available, including the source code, pretrained models, and a large dataset of synthetic OCTA images.

Comparative assessment of established and deep learning-based segmentation methods for hippocampal volume estimation in brain magnetic resonance imaging analysis.

In this study, our objective was to assess the performance of two deep learning-based hippocampal segmentation methods, SynthSeg and TigerBx, which are readily available to the public. We contrasted their performance with that of two established techniques, FreeSurfer-Aseg and FSL-FIRST, using three-dimensional T1-weighted MRI scans (n = 1447) procured from public databases. Our evaluation focused on the accuracy and reproducibility of these tools in estimating hippocampal volume. The findings suggest that both SynthSeg and TigerBx are on a par with Aseg and FIRST in terms of segmentation accuracy and reproducibility, but offer a significant advantage in processing speed, generating results in less than 1 min compared with several minutes to hours for the latter tools. In terms of Alzheimer's disease classification based on the hippocampal atrophy rate, SynthSeg and TigerBx exhibited superior performance. In conclusion, we evaluated the capabilities of two deep learning-based segmentation techniques. The results underscore their potential value in clinical and research environments, particularly when investigating neurological conditions associated with hippocampal structures.

Feature Sparse Choosing VIT Model for Efficient Concrete Crack Segmentation in Portable Crack Measuring Devices

Concrete crack measurement is important for concrete buildings. Deep learning-based segmentation methods have achieved state-of-art results. However, the model size of these models is extremely large which is impossible to use in portable crack measuring devices. To address this problem, a light-weight concrete crack segmentation model based on the Feature Sparse Choosing VIT (LTNet) is proposed by us. In our proposed model, a Feature Sparse Choosing VIT (FSVIT) is used to reduce computational complexity in VIT as well as reducing the number of channels for crack features. In addition, a Feature Channel Selecting Module (FCSM) is proposed by us to reduce channel features as well as suppressing the influence of interfering features. Finally, Depthwise Separable Convolutions are used to substitute traditional convolutions for further reducing computational complexity. As a result, the model size of our LTNet is extremely small. Experimental results show that our LTNet could achieve an accuracy of 0.887, 0.817 and 0.693, and achieve a recall of 0.882, 0.805 and 0.681 on three datasets, respectively, which is 3–8% higher than current mainstream algorithms. However, the model size of our LTNet is only 2 M.

Estimation of Damage Levels in Masonry Structures Following Earthquake Impact Using Deep Learning-based Segmentation Method

This study presents an innovative and automated methodology for assessing damage levels in masonry structures following earthquakes, utilizing a deep learning-based segmentation approach. Central to this research is the use of a U-Net convolutional neural network (CNN) model, which facilitates automated damage detection with a focus on smartphone-enabled, real-time analysis. A key feature of this method is a novel damage index (DI), calculated by normalizing the surface area of detected damages against the total area of the structure, as viewed from the same perspective. The findings indicate a marked improvement in damage detection capabilities, with the U-Net model achieving a precision of approximately 92% and a recall of around 93%. These figures highlight the model’s proficiency in accurately identifying damaged areas and reducing false positives, an essential aspect of post-earthquake evaluations. While this methodology represents a significant step forward in enabling rapid and cost-effective post-earthquake inspections, it is accompanied by certain constraints, particularly in terms of dataset diversity and computational requirements. Despite these challenges, the high accuracy and effectiveness of the damage detection and indexing process demonstrate strong potential for future applications in structural health monitoring, especially in scenarios that demand prompt action and are limited by resources.

CSAM: A 2.5D Cross-Slice Attention Module for Anisotropic Volumetric Medical Image Segmentation.

A large portion of volumetric medical data, especially magnetic resonance imaging (MRI) data, is anisotropic, as the through-plane resolution is typically much lower than the in-plane resolution. Both 3D and purely 2D deep learning-based segmentation methods are deficient in dealing with such volumetric data since the performance of 3D methods suffers when confronting anisotropic data, and 2D methods disregard crucial volumetric information. Insufficient work has been done on 2.5D methods, in which 2D convolution is mainly used in concert with volumetric information. These models focus on learning the relationship across slices, but typically have many parameters to train. We offer a Cross-Slice Attention Module (CSAM) with minimal trainable parameters, which captures information across all the slices in the volume by applying semantic, positional, and slice attention on deep feature maps at different scales. Our extensive experiments using different network architectures and tasks demonstrate the usefulness and generalizability of CSAM. Associated code is available at https://github.com/aL3x-O-o-Hung/CSAM.

Exploring Unlabeled Data in Multiple Aspects for Semi-Supervised MRI Segmentation.

Background: MRI segmentation offers crucial insights for automatic analysis. Although deep learning-based segmentation methods have attained cutting-edge performance, their efficacy heavily relies on vast sets of meticulously annotated data. Methods: In this study, we propose a novel semi-supervised MRI segmentation model that is able to explore unlabeled data in multiple aspects based on various semi-supervised learning technologies. Results: We compared the performance of our proposed method with other deep learning-based methods on 2 public datasets, and the results demonstrated that we have achieved Dice scores of 90.3% and 89.4% on the LA and ACDC datasets, respectively. Conclusions: We explored the synergy of various semi-supervised learning technologies for MRI segmentation, and our investigation will inspire research that focuses on designing MRI segmentation models.

Comparative Analysis of Baseline Vnet and Unet Architectures on Pancreas Segmentation

The pancreas is one of the vital organs in the human body. It has an essential role in the digestive system and endocrine system. Diseases such as cancer, diabetes, hormonal problems, pancreatitis, and digestive problems occur in pancreatic disorders. In detecting pancreatic disorders, first blood and urine tests are requested. If further examination is needed, CT (Computed Tomography), MR (Magnetic Resonance), and EUS (Endoscopic Ultrasonography) imaging methods are used. Pancreas segmentation is generally the process of defining and drawing the lines of the pancreas from medical images such as CT and MRI. The size and shape of the pancreas varies from person to person. Manual segmentation of the pancreas is time-consuming and varies between physicians. Recently, deep learning-based segmentation methods that achieve high-performance results in organ segmentation have become trendy. In this study, Unet and Vnet architectures were comparatively analyzed on the NIH-CT-82 dataset. As a result of the ablation studies, a validation sensitivity of 0.9978 and a validation loss of 0.041 were obtained in the Unet architecture. In the training with the Vnet architecture, 0.9975 validation sensitivity and 0.046 validation loss values were obtained, respectively.

Learning with limited annotations: A survey on deep semi-supervised learning for medical image segmentation

Medical image segmentation is a fundamental and critical step in many image-guided clinical approaches. Recent success of deep learning-based segmentation methods usually relies on a large amount of labeled data, which is particularly difficult and costly to obtain, especially in the medical imaging domain where only experts can provide reliable and accurate annotations. Semi-supervised learning has emerged as an appealing strategy and been widely applied to medical image segmentation tasks to train deep models with limited annotations. In this paper, we present a comprehensive review of recently proposed semi-supervised learning methods for medical image segmentation and summarize both the technical novelties and empirical results. Furthermore, we analyze and discuss the limitations and several unsolved problems of existing approaches. We hope this review can inspire the research community to explore solutions to this challenge and further advance the field of medical image segmentation.

Semi-Supervised Representation Learning for Segmentation on Medical Volumes and Sequences.

Benefiting from the massive labeled samples, deep learning-based segmentation methods have achieved great success for two dimensional natural images. However, it is still a challenging task to segment high dimensional medical volumes and sequences, due to the considerable efforts for clinical expertise to make large scale annotations. Self/semi-supervised learning methods have been shown to improve the performance by exploiting unlabeled data. However, they are still lack of mining local semantic discrimination and exploitation of volume/sequence structures. In this work, we propose a semi-supervised representation learning method with two novel modules to enhance the features in the encoder and decoder, respectively. For the encoder, based on the continuity between slices/frames and the common spatial layout of organs across subjects, we propose an asymmetric network with an attention-guided predictor to enable prediction between feature maps of different slices of unlabeled data. For the decoder, based on the semantic consistency between labeled data and unlabeled data, we introduce a novel semantic contrastive learning to regularize the feature maps in the decoder. The two parts are trained jointly with both labeled and unlabeled volumes/sequences in a semi-supervised manner. When evaluated on three benchmark datasets of medical volumes and sequences, our model outperforms existing methods with a large margin of 7.3% DSC on ACDC, 6.5% on Prostate, and 3.2% on CAMUS when only a few labeled data is available. Further, results on the M&M dataset show that the proposed method yields improvement without using any domain adaption techniques for data from unknown domain. Intensive evaluations reveal the effectiveness of representation mining, and superiority on performance of our method. The code is available at https://github.com/CcchenzJ/BootstrapRepresentation.

FastCellpose: A Fast and Accurate Deep-Learning Framework for Segmentation of All Glomeruli in Mouse Whole-Kidney Microscopic Optical Images.

Automated evaluation of all glomeruli throughout the whole kidney is essential for the comprehensive study of kidney function as well as understanding the mechanisms of kidney disease and development. The emerging large-volume microscopic optical imaging techniques allow for the acquisition of mouse whole-kidney 3D datasets at a high resolution. However, fast and accurate analysis of massive imaging data remains a challenge. Here, we propose a deep learning-based segmentation method called FastCellpose to efficiently segment all glomeruli in whole mouse kidneys. Our framework is based on Cellpose, with comprehensive optimization in network architecture and the mask reconstruction process. By means of visual and quantitative analysis, we demonstrate that FastCellpose can achieve superior segmentation performance compared to other state-of-the-art cellular segmentation methods, and the processing speed was 12-fold higher than before. Based on this high-performance framework, we quantitatively analyzed the development changes of mouse glomeruli from birth to maturity, which is promising in terms of providing new insights for research on kidney development and function.

A dynamic semantic segmentation algorithm with encoder-crossor-decoder structure for pixel-level building cracks

A large number of newly built infrastructures as well as those constructed in the early stage are faced with the problems of detection and maintenance. However, it is difficult to detect building cracks because of its small size and complex background noise. In this study, a crack segmentation network based on Encoder-Crossor-Decoder structure is innovatively proposed to solve the problems of small cracks and easy to be disturbed by background. Then, a loss function is proposed to address the problem of large differences in the ratio of cracks to background pixels in architectural crack segmentation. The experiments show that the loss function can effectively improve the training effect of the model and make the model obtain better semantic segmentation ability. Finally, according to the requirements of building crack detection, a large dataset of concrete pavement cracks is produced, which fills the gap of large dataset of semantic segmentation of cracks. The excellent effect of the model and loss function is verified with three datasets containing most of the major material and structural scenes. In addition, we compare the model with other deep learning segmentation models to validate its effectiveness. The results show that the mIoU of the model of this study reaches 84.04%, 77.56% and 87.38% in the bridge non-steel crack dataset, steel surface crack dataset and our concrete crack dataset, respectively. The accuracy reaches 99.14%, 98.62% and 99.37%. F1 reaches 0.911, 0.873 and 0.963 respectively. It outperforms other deep learning based segmentation methods.

GA-Net: Ghost convolution adaptive fusion skin lesion segmentation network

Automatic segmentation of skin lesions is a pivotal task in computer-aided diagnosis, playing a crucial role in the early detection and treatment of skin cancer. Despite the existence of numerous deep learning-based segmentation methods, the extraction of lesion features remains inadequate as a result of the segmentation process. Consequently, skin lesion image segmentation continues to face challenges regarding missing detailed information and inaccurate segmentation of the lesion region. In this paper, we propose a ghost convolution adaptive fusion network for skin lesion segmentation. First, the neural network incorporates a ghost module instead of the ordinary convolution layer, generating a rich skin lesion feature map for comprehensive target feature extraction. Subsequently, the network employs an adaptive fusion module and bilateral attention module to connect the encoding and decoding layers, facilitating the integration of shallow and deep network information. Moreover, multi-level output patterns are used for pixel prediction. Layer feature fusion effectively combines output features of different scales, thus improving image segmentation accuracy. The proposed network was extensively evaluated on three publicly available datasets: ISIC2016, ISIC2017, and ISIC2018. The experimental results demonstrated accuracies of 96.42%, 94.07%, and 95.03%, and kappa coefficients of 90.41%, 81.08%, and 86.96%, respectively. The overall performance of our network surpassed that of existing networks. Simulation experiments further revealed that the ghost convolution adaptive fusion network exhibited superior segmentation results for skin lesion images, offering new possibilities for the diagnosis of skin diseases.

Automatic brain structure segmentation for 18F-fluorodeoxyglucose positron emission tomography/magnetic resonance images via deep learning.

Brain structure segmentation is of great value in diagnosing brain disorders, allowing radiologists to quickly acquire regions of interest and assist in subsequent analyses, diagnoses and treatment. Current brain structure segmentation methods are usually applied to magnetic resonance (MR) images, which provide higher soft tissue contrast and better spatial resolution. However, fewer segmentation methods are conducted on a positron emission tomography/magnetic resonance imaging (PET/MRI) system that combines functional and structural information to improve analysis accuracy. In this paper, we explore a dual-modality image segmentation model to segment brain 18F-fluorodeoxyglucose (18F-FDG) PET/MR images based on the U-Net architecture. This model takes registered PET and MR images as parallel inputs, and four evaluation metrics (Dice score, Jaccard coefficient, precision and sensitivity) are used to evaluate segmentation performance. Moreover, we also compared the proposed approach with other single-modality segmentation strategies, including PET-only segmentation and MRI-only segmentation. The experiments were conducted on the clinical head data of 120 patients, and the results show that the proposed algorithm accurately delineates brain volumes of interest (VOIs), achieving superior performance with 84.24%±1.44% Dice score, 74.36%±2.40% Jaccard, 84.33%±1.56% precision and 84.73%±1.56% sensitivity. Furthermore, compared with directly using the FreeSurfer toolkit, the proposed method reduced the segmentation time, which only needs 20 seconds to segment the whole brain for each patient. We present a deep learning-based method for the joint segmentation of anatomical and functional PET/MR images. Compared with other single-modality methods, our method greatly improved the accuracy of brain structure delineation, which shows great potential for brain analysis.