Patch-based Label Fusion Method Research Articles

Multi-atlas based neonatal brain extraction using a two-level patch-based label fusion strategy

Compared to adults, brain extraction from magnetic resonance (MR) images of newborn infants is particularly challenging due to their smaller brain size, which causes lower spatial resolution, lower tissue contrast and ambiguous tissue intensity distribution. In this work, a multi-atlas patch-based label fusion method is presented for automatic brain extraction from neonatal head MR images. In this method, a number of atlases are first selected uniformly among a set of training images. After nonlinear alignment of the selected atlas images to the target image, a probabilistic gray level-coded brain mask is created and used to assign brain/non-brain labels to voxels with different degree of uncertainty using a modified non-local patch-based label fusion method based on the integration of low-level and in-depth search patch selection strategies. Experiments with 40 neonates aged between 37 and 44 weeks showed an average Dice, Jaccard and Conformity coefficients of 0.993, 0.986 and 0.986, respectively. We compared the performance of the proposed method with two multi atlas-based methods, i.e. NLPB and MASS, and two popular non-learning-based methods, i.e. BSE and BET. Compared to these methods, our method achieved higher accuracy with brain masks very close to manually extracted ones and produced lower false negative and false positive rates. Our proposed method allows for accurate and efficient brain extraction, a crucial step in brain MRI applications such as brain tissue segmentation and volume estimation in neonates.

Sparse patch‐based representation with combined information of atlas for multi‐atlas label fusion

To obtain a higher accuracy in the multi-atlas patch-based label fusion method, it is essential to have the accurate similarity measure of selected patches. In this study, the authors propose a new sparse patch-based representation method using a local binary texture (LBT) in the atlas image and atlas label information for the multi-atlas label fusion. In the proposed method, the intensity information in a patch is converted into a LBT which is then combined with the labels of corresponding patches from the atlas to form an atom of a dictionary. The initial labels of target images are estimated through a rough segmentation. The voxel in a patch to be labelled is also constructed as a vector similar to the atom. The voxel vector is then modelled as a sparse linear combination of the atoms in the dictionary. Experimental results on two MR brain data sets demonstrated that the proposed method is efficient in the segmentation which can achieve competitive performance compared with the state-of-the-art methods.

Label fusion method based on sparse patch representation for the brain MRI image segmentation

The multi-Atlas patch-based label fusion method (MAS-PBM) has emerged as a promising technique for the magnetic resonance imaging (MRI) image segmentation. The state-of-the-art MAS-PBM approach measures the patch similarity between the target image and each atlas image using the features extracted from images intensity only. It is well known that each atlas consists of both MRI image and labelled image (which is also called the map). In other words, the map information is not used in calculating the similarity in the existing MAS-PBM. To improve the segmentation result, the authors propose an enhanced MAS-PBM in which the maps will be used for similarity measure. The first component of the proposed method is that an initial segmentation result (i.e. an appropriate map for the target) is obtained by using either the non-local-patch-based label fusion method (NPBM) or the sparse patch-based label fusion method (SPBM) based on the grey scales of patches. Then, the SPBM is applied again to obtain the finer segmentation based on the labels of patches. The authors called these two versions of the proposed fusion method as MAS-PBM-NPBM and MAS-PBM-SPBM. Experimental results show that more accurate segmentation results are achieved compared with those of the majority voting, NPBM, SPBM, STEPS and the hierarchical multi-atlas label fusion with multi-scale feature representation and label-specific patch partition.

Combining a Patch-based Approach with a Non-rigid Registration-based Label Fusion Method for the Hippocampal Segmentation in Alzheimer's Disease.

We provide and evaluate an open-source software solution for automatically hippocampal segmentation from T1-weighted (T1w) magnetic resonance imaging (MRI). The method is applied for measuring the hippocampal volume, which allows discriminate between patients with Alzheimer's disease (AD) or mild cognitive impairment (MCI) and elderly controls (NC). The method is based on a fast patch-based label fusion method, whose selected patches and their weights are calculated from a combination of similarity measures between patches using intensity-based distances and labeling-based distances. These combined similarity measures produces better selection of the patches, and their weights are more robust. The algorithm is trained with the Harmonized Hippocampal Protocol (HarP). The proposal is compared with FreeSurfer and other label fusion methods. To evaluate the performance and the robustness of the proposed label fusion method, we employ two databases of T1w MRI of human brains. For AD vs NC, we obtain a high degree of accuracy, approximately 90 %. For MCI vs NC, we obtain accuracies around 75 %. The average time for the hippocampal segmentation from a T1w MRI is less than 17 minutes.

Progressive multi-atlas label fusion by dictionary evolution.

Accurate segmentation of anatomical structures in medical images is important in recent imaging based studies. In the past years, multi-atlas patch-based label fusion methods have achieved a great success in medical image segmentation. In these methods, the appearance of each input image patch is first represented by an atlas patch dictionary (in the image domain), and then the latent label of the input image patch is predicted by applying the estimated representation coefficients to the corresponding anatomical labels of the atlas patches in the atlas label dictionary (in the label domain). However, due to the generally large gap between the patch appearance in the image domain and the patch structure in the label domain, the estimated (patch) representation coefficients from the image domain may not be optimal for the final label fusion, thus reducing the labeling accuracy. To address this issue, we propose a novel label fusion framework to seek for the suitable label fusion weights by progressively constructing a dynamic dictionary in a layer-by-layer manner, where the intermediate dictionaries act as a sequence of guidance to steer the transition of (patch) representation coefficients from the image domain to the label domain. Our proposed multi-layer label fusion framework is flexible enough to be applied to the existing labeling methods for improving their label fusion performance, i.e., by extending their single-layer static dictionary to the multi-layer dynamic dictionary. The experimental results show that our proposed progressive label fusion method achieves more accurate hippocampal segmentation results for the ADNI dataset, compared to the counterpart methods using only the single-layer static dictionary.

Head and Neck Auto Segmentation Challenge based on Non-Local Generative Models

A new patch based label fusion method based on generative approach is proposed for segmentation of mandible, brainstem, parotid and submandibular glands, optic nerves and the optic chiasm in head and neck CT images. The proposal constructs local classifiers from a dictionary of patches and weights their contribution using a generative probabilistic criterion. Also, a gaussian slide window is used to weight the multiples estimations of neighboring voxels. The proposed method was evaluated on a set of 15 CT images (10 off-site and 5 onsite) provided by the organizers of the Head and neck Auto-Segmentation challenge(MICCAI 2015), where the obtained results are comparable to many of the other methods used in the challenge.

Head and Neck Auto Segmentation Challenge based on Non-Local Generative Models

A new patch based label fusion method based on generative approach is proposed for segmentation of mandible, brainstem, parotid and submandibular glands, optic nerves and the optic chiasm in head and neck CT images. The proposal constructs local classifiers from a dictionary of patches and weights their contribution using a generative probabilistic criterion. Also, a gaussian slide window is used to weight the multiples estimations of neighboring voxels. The proposed method was evaluated on a set of 15 CT images (10 off-site and 5 onsite) provided by the organizers of the Head and neck Auto-Segmentation challenge(MICCAI 2015), where the obtained results are comparable to many of the other methods used in the challenge.

Automatic Segmentation of Hippocampus for Longitudinal Infant Brain MR Image Sequence by Spatial-Temporal Hypergraph Learning.

Accurate segmentation of infant hippocampus from Magnetic Resonance (MR) images is one of the key steps for the investigation of early brain development and neurological disorders. Since the manual delineation of anatomical structures is time-consuming and irreproducible, a number of automatic segmentation methods have been proposed, such as multi-atlas patch-based label fusion methods. However, the hippocampus during the first year of life undergoes dynamic appearance, tissue contrast and structural changes, which pose substantial challenges to the existing label fusion methods. In addition, most of the existing label fusion methods generally segment target images at each time-point independently, which is likely to result in inconsistent hippocampus segmentation results along different time-points. In this paper, we treat a longitudinal image sequence as a whole, and propose a spatial-temporal hypergraph based model to jointly segment infant hippocampi from all time-points. Specifically, in building the spatial-temporal hypergraph, (1) the atlas-to-target relationship and (2) the spatial/temporal neighborhood information within the target image sequence are encoded as two categories of hyperedges. Then, the infant hippocampus segmentation from the whole image sequence is formulated as a semi-supervised label propagation model using the proposed hypergraph. We evaluate our method in segmenting infant hippocampi from T1-weighted brain MR images acquired at the age of 2 weeks, 3 months, 6 months, 9 months, and 12 months. Experimental results demonstrate that, by leveraging spatial-temporal information, our method achieves better performance in both segmentation accuracy and consistency over the state-of-the-art multi-atlas label fusion methods.

Multi-Atlas Based Segmentation of Brainstem Nuclei from MR Images by Deep Hyper-Graph Learning.

Accurate segmentation of brainstem nuclei (red nucleus and substantia nigra) is very important in various neuroimaging applications such as deep brain stimulation and the investigation of imaging biomarkers for Parkinson's disease (PD). Due to iron deposition during aging, image contrast in the brainstem is very low in Magnetic Resonance (MR) images. Hence, the ambiguity of patch-wise similarity makes the recently successful multi-atlas patch-based label fusion methods have difficulty to perform as competitive as segmenting cortical and sub-cortical regions from MR images. To address this challenge, we propose a novel multi-atlas brainstem nuclei segmentation method using deep hyper-graph learning. Specifically, we achieve this goal in three-fold. First, we employ hyper-graph to combine the advantage of maintaining spatial coherence from graph-based segmentation approaches and the benefit of harnessing population priors from multi-atlas based framework. Second, besides using low-level image appearance, we also extract high-level context features to measure the complex patch-wise relationship. Since the context features are calculated on a tentatively estimated label probability map, we eventually turn our hyper-graph learning based label propagation into a deep and self-refining model. Third, since anatomical labels on some voxels (usually located in uniform regions) can be identified much more reliably than other voxels (usually located at the boundary between two regions), we allow these reliable voxels to propagate their labels to the nearby difficult-to-label voxels. Such hierarchical strategy makes our proposed label fusion method deep and dynamic. We evaluate our proposed label fusion method in segmenting substantia nigra (SN) and red nucleus (RN) from 3.0 T MR images, where our proposed method achieves significant improvement over the state-of-the-art label fusion methods.

A transversal approach for patch-based label fusion via matrix completion.

Recently, multi-atlas patch-based label fusion has received an increasing interest in the medical image segmentation field. After warping the anatomical labels from the atlas images to the target image by registration, label fusion is the key step to determine the latent label for each target image point. Two popular types of patch-based label fusion approaches are (1) reconstruction-based approaches that compute the target labels as a weighted average of atlas labels, where the weights are derived by reconstructing the target image patch using the atlas image patches; and (2) classification-based approaches that determine the target label as a mapping of the target image patch, where the mapping function is often learned using the atlas image patches and their corresponding labels. Both approaches have their advantages and limitations. In this paper, we propose a novel patch-based label fusion method to combine the above two types of approaches via matrix completion (and hence, we call it transversal). As we will show, our method overcomes the individual limitations of both reconstruction-based and classification-based approaches. Since the labeling confidences may vary across the target image points, we further propose a sequential labeling framework that first labels the highly confident points and then gradually labels more challenging points in an iterative manner, guided by the label information determined in the previous iterations. We demonstrate the performance of our novel label fusion method in segmenting the hippocampus in the ADNI dataset, subcortical and limbic structures in the LONI dataset, and mid-brain structures in the SATA dataset. We achieve more accurate segmentation results than both reconstruction-based and classification-based approaches. Our label fusion method is also ranked 1st in the online SATA Multi-Atlas Segmentation Challenge.

Progressive Label Fusion Framework for Multi-atlas Segmentation by Dictionary Evolution.

Accurate segmentation of anatomical structures in medical images is very important in neuroscience studies. Recently, multi-atlas patch-based label fusion methods have achieved many successes, which generally represent each target patch from an atlas patch dictionary in the image domain and then predict the latent label by directly applying the estimated representation coefficients in the label domain. However, due to the large gap between these two domains, the estimated representation coefficients in the image domain may not stay optimal for the label fusion. To overcome this dilemma, we propose a novel label fusion framework to make the weighting coefficients eventually to be optimal for the label fusion by progressively constructing a dynamic dictionary in a layer-by-layer manner, where a sequence of intermediate patch dictionaries gradually encode the transition from the patch representation coefficients in image domain to the optimal weights for label fusion. Our proposed framework is general to augment the label fusion performance of the current state-of-the-art methods. In our experiments, we apply our proposed method to hippocampus segmentation on ADNI dataset and achieve more accurate labeling results, compared to the counterpart methods with single-layer dictionary.

Multi-Atlas and Multi-Modal Hippocampus Segmentation for Infant MR Brain Images by Propagating Anatomical Labels on Hypergraph.

Accurate segmentation of hippocampus from infant magnetic resonance (MR) images is very important in the study of early brain development and neurological disorder. Recently, multi-atlas patch-based label fusion methods have shown a great success in segmenting anatomical structures from medical images. However, the dramatic appearance change from birth to 1-year-old and the poor image contrast make the existing label fusion methods less competitive to handle infant brain images. To alleviate these difficulties, we propose a novel multi-atlas and multi-modal label fusion method, which can unanimously label for all voxels by propagating the anatomical labels on a hypergraph. Specifically, we consider not only all voxels within the target image but also voxels across the atlas images as the vertexes in the hypergraph. Each hyperedge encodes a high-order correlation, among a set of vertexes, in different perspectives which incorporate 1) feature affinity within the multi-modal feature space, 2) spatial coherence within target image, and 3) population heuristics from multiple atlases. In addition, our label fusion method further allows those reliable voxels to supervise the label estimation on other difficult-to-label voxels, based on the established hyperedges, until all the target image voxels reach the unanimous labeling result. We evaluate our proposed label fusion method in segmenting hippocampus from T1 and T2 weighted MR images acquired from at 2-week-old, 3-month-old, 6-month-old, 9-month-old, and 12-month-old. Our segmentation results achieves improvement of labeling accuracy over the conventional state-of-the-art label fusion methods, which shows a great potential to facilitate the early infant brain studies.

Hierarchical multi-atlas label fusion with multi-scale feature representation and label-specific patch partition

Multi-atlas patch-based label fusion methods have been successfully used to improve segmentation accuracy in many important medical image analysis applications. In general, to achieve label fusion a single target image is first registered to several atlas images. After registration a label is assigned to each target point in the target image by determining the similarity between the underlying target image patch (centered at the target point) and the aligned image patch in each atlas image. To achieve the highest level of accuracy during the label fusion process it's critical for the chosen patch similarity measurement to accurately capture the tissue/shape appearance of the anatomical structure. One major limitation of existing state-of-the-art label fusion methods is that they often apply a fixed size image patch throughout the entire label fusion procedure. Doing so may severely affect the fidelity of the patch similarity measurement, which in turn may not adequately capture complex tissue appearance patterns expressed by the anatomical structure. To address this limitation, we advance state-of-the-art by adding three new label fusion contributions: First, each image patch is now characterized by a multi-scale feature representation that encodes both local and semi-local image information. Doing so will increase the accuracy of the patch-based similarity measurement. Second, to limit the possibility of the patch-based similarity measurement being wrongly guided by the presence of multiple anatomical structures in the same image patch, each atlas image patch is further partitioned into a set of label-specific partial image patches according to the existing labels. Since image information has now been semantically divided into different patterns, these new label-specific atlas patches make the label fusion process more specific and flexible. Lastly, in order to correct target points that are mislabeled during label fusion, a hierarchical approach is used to improve the label fusion results. In particular, a coarse-to-fine iterative label fusion approach is used that gradually reduces the patch size. To evaluate the accuracy of our label fusion approach, the proposed method was used to segment the hippocampus in the ADNI dataset and 7.0T MR images, sub-cortical regions in LONI LBPA40 dataset, mid-brain regions in SATA dataset from MICCAI 2013 segmentation challenge, and a set of key internal gray matter structures in IXI dataset. In all experiments, the segmentation results of the proposed hierarchical label fusion method with multi-scale feature representations and label-specific atlas patches are more accurate than several well-known state-of-the-art label fusion methods.

Patch-based label fusion segmentation of brainstem structures with dual-contrast MRI for Parkinson's disease.

Parkinson's disease (PD) is a neurodegenerative disorder that impairs the motor functions. Both surgical treatment and study of PD require delineation of basal ganglia nuclei morphology. While many automatic volumetric segmentation methods have been proposed for the lentiform nucleus, few have attempted to identify the key brainstem substructures including the subthalamic nucleus (STN), substantia nigra (SN), and red nucleus (RN) due to their small size and poor contrast in conventional T1W MRI. A dual-contrast patch-based label fusion method was developed to segment the SN, STN, and RN using multivariate cross-correlation. Two different MRI contrasts (T2*w and phase) are produced from a multi-contrast multi-echo FLASH MRI sequence, enabling visualization of these nuclei. T1-T2* fusion MRI was used to resolve the issue of poor nuclei (i.e., the STN, SN, and RN) contrast on T1w MRI, and to mitigate susceptibility artifacts that may hinder accurate nonlinear registration on T2*w MRI. Unbiased group-wise registration was used for anatomical normalization between the atlas library and the target subject. The performance of the proposed method was compared with a state-of-the-art single-contrast label fusion technique. The proposed method outperformed a state-of-the-art single-contrast patch-based method in segmenting the STN, RN and SN, and the results were better than those reported in previous literature. Our dual-contrast patch-based label fusion method was superior to a single-contrast method for segmenting brainstem nuclei using a multi-contrast multi-echo FLASH MRI sequence. The method is promising for the treatment and research of Parkinson's disease. This method can be extended for multiple alternative image contrasts and other fields of applications.

Hierarchical label fusion with multiscale feature representation and label-specific patch partition.

Recently, patch-based label fusion methods have achieved many successes in medical imaging area. After registering atlas images to the target image, the label at each target image point can be subsequently determined by checking the patchwise similarities between the underlying target image patch and all atlas image patches. Apparently, the definition of patchwise similarity is critical in label fusion. However, current methods often simply use entire image patch with fixed patch size throughout the entire label fusion procedure, which could be insufficient to distinguish complex shape/appearance patterns of anatomical structures in medical imaging scenario. In this paper, we address the above limitations at three folds. First, we assign each image patch with multiscale feature representations such that both local and semi-local image information can be encoded to increase robustness of measuring patchwise similarity in label fusion. Second, since multiple variable neighboring structures could present in one image patch, simply computing patchwise similarity based on the entire image patch is not specific to the particular structure of interest under labeling and can be easily misled by the surrounding variable structures in the same image patch. Thus, we partition each atlas patch into a set of new label-specific atlas patches according to the existing label information in the atlas mages. Then, the new label-specific atlas patches can be more specific and flexible for label fusion than using the entire image patch, since the complex image patch has now been semantically divided into several distinct patterns. Finally, in order to correct the possible mis-labeling, we hierarchically improve the label fusion result in a coarse-to-fine manner by iteratively repeating the label fusion procedure with the gradually-reduced patch size. More accurate label fusion results have been achieved by our hierarchical label fusion method with multiscale feature presentations upon label-specific atlas patches.

Minimizing joint risk of mislabeling for iterative Patch-based label fusion.

Automated labeling of anatomical structures in medical images is very important in many neuroscience studies. Recently, patch-based labeling in the non-local manner has been widely investigated to alleviate the possible misalignment when registering atlases to the target image. However, the weights used for label fusion from the registered atlases in conventional methods are generally computed independently and thus lack the capability of preventing the ambiguous atlas patches from contributing to the label fusion. More critically, these weights are often calculated based only on the simple patch similarity, thus not necessarily providing optimal solution for label fusion. To address these issues, we present a novel patch-based label fusion method in multi-atlas scenario, for the goal of labeling each voxel in the target image by the best representative atlas patches that also have the lowest joint risk of mislabeling. Specifically, sparse coding is used to select a small number of atlas patches which best represent the underlying patch at each point of the target image, thus minimizing the chance of including the misleading atlas patches for labeling. Furthermore, we examine the joint risk of any pair of atlas patches in making similar labeling error, by analyzing the correlation of their morphological error patterns and also the labeling consensus among atlases. This joint risk will be further recursively updated based on the latest labeling results to correct the possible labeling errors. To demonstrate the performance of our proposed method, we have evaluated it on both whole brain parcellation and hippocampus segmentation, and achieved promising labeling results, compared with the state-of-the-art methods.

Influence of the training library composition on a patch-based label fusion method: Application to hippocampus segmentation on the ADNI dataset

The atrophy of medial temporal lobe structures is may serve as early biomarkers of Alzheimer's disease. The evaluation of hippocampus (HC) atrophy is estimated by volumetric studies requiring a segmentation step that can be very time consuming when done manually. This limitation can be overcome by using automatic segmentation methods. In this study, we propose to validate our nonlocal patch-based method on the Alzheimer's Disease Neuroimaging Initiative (ADNI) database by segmenting the HC of Cognitively Normal (CN) subjects and patients with early Alzheimer's Disease (AD).

Influence of the training library composition on a patch-based label fusion method: Application to hippocampus segmentation on the ADNI dataset

Influence of the training library composition on a patch-based label fusion method: Application to hippocampus segmentation on the ADNI dataset