Large Shape Variations Research Articles

CA-Net: Comprehensive Attention Convolutional Neural Networks for Explainable Medical Image Segmentation.

Accurate medical image segmentation is essential for diagnosis and treatment planning of diseases. Convolutional Neural Networks (CNNs) have achieved state-of-the-art performance for automatic medical image segmentation. However, they are still challenged by complicated conditions where the segmentation target has large variations of position, shape and scale, and existing CNNs have a poor explainability that limits their application to clinical decisions. In this work, we make extensive use of multiple attentions in a CNN architecture and propose a comprehensive attention-based CNN (CA-Net) for more accurate and explainable medical image segmentation that is aware of the most important spatial positions, channels and scales at the same time. In particular, we first propose a joint spatial attention module to make the network focus more on the foreground region. Then, a novel channel attention module is proposed to adaptively recalibrate channel-wise feature responses and highlight the most relevant feature channels. Also, we propose a scale attention module implicitly emphasizing the most salient feature maps among multiple scales so that the CNN is adaptive to the size of an object. Extensive experiments on skin lesion segmentation from ISIC 2018 and multi-class segmentation of fetal MRI found that our proposed CA-Net significantly improved the average segmentation Dice score from 87.77% to 92.08% for skin lesion, 84.79% to 87.08% for the placenta and 93.20% to 95.88% for the fetal brain respectively compared with U-Net. It reduced the model size to around 15 times smaller with close or even better accuracy compared with state-of-the-art DeepLabv3+. In addition, it has a much higher explainability than existing networks by visualizing the attention weight maps. Our code is available at https://github.com/HiLab-git/CA-Net.

Liver segmentation in abdominal CT images via auto-context neural network and self-supervised contour attention

Objective: Accurate image segmentation of the liver is a challenging problem owing to its large shape variability and unclear boundaries. Although the applications of fully convolutional neural networks (CNNs) have shown groundbreaking results, limited studies have focused on the performance of generalization. In this study, we introduce a CNN for liver segmentation on abdominal computed tomography (CT) images that focus on the performance of generalization and accuracy. Methods: To improve the generalization performance, we initially propose an auto-context algorithm in a single CNN. The proposed auto-context neural network exploits an effective high-level residual estimation to obtain the shape prior. Identical dual paths are effectively trained to represent mutual complementary features for an accurate posterior analysis of a liver. Further, we extend our network by employing a self-supervised contour scheme. We trained sparse contour features by penalizing the ground-truth contour to focus more contour attentions on the failures. Results: We used 180 abdominal CT images for training and validation. Two-fold cross-validation is presented for a comparison with the state-of-the-art neural networks. The experimental results show that the proposed network results in better accuracy when compared to the state-of-the-art networks by reducing 10.31% of the Hausdorff distance. Novel multiple N-fold cross-validations are conducted to show the best performance of generalization of the proposed network. Conclusion and significance: The proposed method minimized the error between training and test images more than any other modern neural networks. Moreover, the contour scheme was successfully employed in the network by introducing a self-supervising metric.

Towards accurate coronary artery calcium segmentation with multi‐scale attention mechanism

Coronary artery calcium is a strong and independent marker of atherosclerosis and cardiovascular disease. Typically, the accurate segmentation of computed tomography images of the chest is an important prerequisite and basis for coronary artery calcium identification and analysis. However, this is very challenging in practice because the boundaries of coronary artery calcium, the small lesions with large shape variation, are very blurry, resulting in poor performance in existing studies. To tackle this challenge, we present a novel Attention-based Multi-Scale Network called AMSN, which can process information through both the main and boundary branches in parallel. Key to our AMSN is a new non-local multi-scale context encoder module, which is mainly composed of the multi-scale attention mechanism and local global long short-term memory module. By aggregating the multi-scale context information, i.e. high-resolution low-level and low-resolution high-level features, the model's feature representative capability and deployment ability are improved effectively. Besides, we introduce a new boundary preserving loss, which can consider the boundary information of all coronary artery calcium together and establish links for the segmentation of different coronary artery calcium simultaneously. Extensive experiments demonstrate our AMSN enables reliable accurate coronary artery calcium segmentation for assisted cardiovascular disease diagnosis clinically.

Boundary Coding Representation for Organ Segmentation in Prostate Cancer Radiotherapy.

Accurate segmentation of the prostate and organs at risk (OARs, e.g., bladder and rectum) in male pelvic CT images is a critical step for prostate cancer radiotherapy. Unfortunately, the unclear organ boundary and large shape variation make the segmentation task very challenging. Previous studies usually used representations defined directly on unclear boundaries as context information to guide segmentation. Those boundary representations may not be so discriminative, resulting in limited performance improvement. To this end, we propose a novel boundary coding network (BCnet) to learn a discriminative representation for organ boundary and use it as the context information to guide the segmentation. Specifically, we design a two-stage learning strategy in the proposed BCnet: 1) Boundary coding representation learning. Two sub-networks under the supervision of the dilation and erosion masks transformed from the manually delineated organ mask are first separately trained to learn the spatial-semantic context near the organ boundary. Then we encode the organ boundary based on the predictions of these two sub-networks and design a multi-atlas based refinement strategy by transferring the knowledge from training data to inference. 2) Organ segmentation. The boundary coding representation as context information, in addition to the image patches, are used to train the final segmentation network. Experimental results on a large and diverse male pelvic CT dataset show that our method achieves superior performance compared with several state-of-the-art methods.

Fully automated 3D segmentation of MR-imaged calf muscle compartments: Neighborhood relationship enhanced fully convolutional network

Automated segmentation of individual calf muscle compartments from 3D magnetic resonance (MR) images is essential for developing quantitative biomarkers for muscular disease progression and its prediction. Achieving clinically acceptable results is a challenging task due to large variations in muscle shape and MR appearance. In this paper, we present a novel fully convolutional network (FCN) that utilizes contextual information in a large neighborhood and embeds edge-aware constraints for individual calf muscle compartment segmentations. An encoder–decoder architecture is used to systematically enlarge convolution receptive field and preserve information at all resolutions. Edge positions derived from the FCN output muscle probability maps are explicitly regularized using kernel-based edge detection in an end-to-end optimization framework. Our method was evaluated on 40 T1-weighted MR images of 10 healthy and 30 diseased subjects by fourfold cross-validation. Mean DICE coefficients of 88.00–91.29% and mean absolute surface positioning errors of 1.04–1.66 mm were achieved for the five 3D muscle compartments.

SymmetryNet

We study the problem of symmetry detection of 3D shapes from single-view RGB-D images, where severely missing data renders geometric detection approach infeasible. We propose an end-to-end deep neural network which is able to predict both reflectional and rotational symmetries of 3D objects present in the input RGB-D image. Directly training a deep model for symmetry prediction, however, can quickly run into the issue of overfitting. We adopt a multi-task learning approach. Aside from symmetry axis prediction, our network is also trained to predict symmetry correspondences. In particular, given the 3D points present in the RGB-D image, our network outputs for each 3D point its symmetric counterpart corresponding to a specific predicted symmetry. In addition, our network is able to detect for a given shape multiple symmetries of different types. We also contribute a benchmark of 3D symmetry detection based on single-view RGB-D images. Extensive evaluation on the benchmark demonstrates the strong generalization ability of our method, in terms of high accuracy of both symmetry axis prediction and counterpart estimation. In particular, our method is robust in handling unseen object instances with large variation in shape, multi-symmetry composition, as well as novel object categories.

Using additive manufactured parametric models for wind tunnel test-based aerodynamic shape optimization

PurposeThe purpose of this paper is to propose a new approach of using additively manufactured parametric models in the wind tunnel test-based aerodynamic shape optimization (ASO) framework and to present its applicability test results obtained from a realistic aircraft design problem.Design/methodology/approachFor aircraft shape optimization, the following three methodologies were used. First, as a validation study, the possibility of using rapid prototyping (RP) model in the wind tunnel test was verified. Second, through the wind tunnel test-based ASO, the application and feasibility of the real fighter aircraft shape optimization were verified. A generic fighter configuration is parameterized to generate various test models using additive manufacturing. Wind tunnel tests are conducted to measure their stability criteria in high angle of attack (AOA). Finally, a computational fluid dynamics (CFD) study was performed and analysis procedures, costs and results compared to the wind tunnel test were compared and reviewed.FindingsRP technology can significantly reduce the time and cost of generating parametric wind tunnel models and can open up new possibilities for wind tunnel tests to be used in the rigorous aerodynamic design loop. There was a slight difference between the results of the RP model and the metallic model because of rigidity and surface roughness. However, the tendency of the aerodynamic characteristics was very similarly predictable. Although there are limitations to obtaining precise aerodynamic data, it is a suitable method to be applied to comparative studies on various shapes with large geo-metric changes in the early phase of design. The CFD analysis indicates that the wind tunnel-based ASO using the RP model shows the efficiency corresponding to the CFD shape optimization.Research limitations/implicationsThe RP parametric models may have various assembly error sources and rigidity problems. The proposed methodology may not be suitable for collecting the accurate aerodynamic database of a final design; rather, the methodology is more suitable to screen out many configurations having fairly large shape variation in the early stage of the design process.Practical implicationsThe wind tunnel test-based ASO can replace or supplement CFD-based ASO. In areas where CFD accuracy is low, such as high AOA flight characteristics, RP model wind tunnel-based ASO can be a research method that can secure both efficiency and accuracy advantages, providing ten times more effective in terms of cost and time. The wind tunnel test is used to obtain aerodynamic data at the final stage of shape design. It can be extended to a comparative study of several shapes in the early design phase. This procedure can be applied for both industrial level and educational aircraft design activities.Originality/valueThis study is the application to be applied as a parametric study on the whole aircraft, rather than using the RP model applying a simple partial control surface or configuration change of a part of the wing. The possibility of using the RP model was confirmed by comparing and verifying each other in a medium-sized wind tunnel using a relatively large RP model and a metallic model. It was verified that it can be applied in the shape design process, not the shape verification in the traditional design procedure, and a comparison with the CFD method was also performed. With further development and validation efforts, the new design framework may become an industrial standard for future aircraft development.

Learning Semantic Keypoint Representations for Door Opening Manipulation

We consider a scenario where a robot is capable of autonomously opening previously unseen doors. Prior works either use model-based methods that rely strongly on accurate kinematic models, or learn a policy from scratch through trial-and-error, which cannot generalize to large variations in shape, and location of doors. In this letter, we propose a novel method for opening unseen doors with no prior knowledge of door model, which leverages semantic 3D keypoints as door handle representations to generate the end-effector trajectory from a motion planner. The keypoint representations are predicted from raw visual input by a deep neural network, which can provide a concise, and semantic description of the handle to determine the grasp pose, and subsequent motion planning. In contrast to existing works that require known object models or significant manual effort on data collection, we present a data augmentation technique to automatically generate large amounts of realistic-looking synthetic data with almost no human labeling effort. An augmented dataset, consisting of large amounts of synthetic data, and small amounts of real data, is used to train the network. Qualitative results show that our proposed method outperforms the state-of-the-art pose-based methods on real test sets in terms of perception metrics. Hardware experiments demonstrate that our proposed method can achieve 94.2% success rate on opening 6 previously unseen doors with significant shape variations under different environments, and conditions.

An improved boundary-aware face alignment using stacked dense U-Nets

Facial landmark localization is still a challenge task in the unconstrained environment with influences of significant variation conditions such as facial pose, shape, expression, illumination, and occlusions. In this work, we present an improved boundary-aware face alignment method by using stacked dense U-Nets. The proposed method consists of two stages: a boundary heatmap estimation stage to learn the facial boundary lines and a facial landmark localization stage to predict the final face alignment result. With the constraint of boundary lines, facial landmarks are unified as a whole facial shape. Hence, the unseen landmarks in a shape with occlusions can be better estimated by message passing with other landmarks. By introducing the stacked dense U-Nets for feature extraction, the capacity of the model is improved. Experiments and comparisons on public datasets show that the proposed method obtains better performance than the baselines, especially for facial images with large pose variation, shape variation, and occlusions.

HIFUNet: Multi-Class Segmentation of Uterine Regions From MR Images Using Global Convolutional Networks for HIFU Surgery Planning.

Accurate segmentation of uterus, uterine fibroids, and spine from MR images is crucial for high intensity focused ultrasound (HIFU) therapy but remains still difficult to achieve because of 1) the large shape and size variations among individuals, 2) the low contrast between adjacent organs and tissues, and 3) the unknown number of uterine fibroids. To tackle this problem, in this paper, we propose a large kernel Encoder-Decoder Network based on a 2D segmentation model. The use of this large kernel can capture multi-scale contexts by enlarging the valid receptive field. In addition, a deep multiple atrous convolution block is also employed to enlarge the receptive field and extract denser feature maps. Our approach is compared to both conventional and other deep learning methods and the experimental results conducted on a large dataset show its effectiveness.

Development and Characterization of a Benchtop Corneal Puncture Injury Model

During recent military operations, eye-related injuries have risen in frequency due to increased use of explosive weaponry which often result in corneal puncture injuries. These have one of the poorest visual outcomes for wounded soldiers, often resulting in blindness due to the large variations in injury shape, size, and severity. As a result, improved therapeutics are needed which can stabilize the injury site and promote wound healing. Unfortunately, current corneal puncture injury models are not capable of producing irregularly shaped, large, high-speed injuries as seen on the battlefield, making relevant therapeutic development challenging. Here, we present a benchtop corneal puncture injury model for use with enucleated eyes that utilizes a high-speed solenoid device suitable for creating military-relevant injuries. We first established system baselines and ocular performance metrics, standardizing the different aspects of the benchtop model to ensure consistent results and properly account for tissue variability. The benchtop model was evaluated with corneal puncture injury objects up to 4.2 mm in diameter which generated intraocular pressure levels exceeding 1500 mmHg. Overall, the created benchtop model provides an initial platform for better characterizing corneal puncture injuries as seen in a military relevant clinical setting and a realistic approach for assessing potential therapeutics.

Encoding features from multi-layer Gabor filtering for visual smoke recognition

It is a challenging task to accurately recognize smoke from visual scenes due to large variations in smoke shape, color and texture. To improve recognition accuracy, we propose a framework mainly with a robust local feature extraction module based on Gabor convolutional networks. We first propose a Gabor convolutional network, each layer of which mainly consists of Gabor convolution and feature fusion. To fuse feature maps generated by Gabor convolution, we present three Basic Grouping Methods, which divide the feature maps into several groups along orientation axis, scale axis and both of them. To avoid exponential growth of feature maps and preserve discriminative information simultaneously, we propose three element-wise aggregation functions, which are mean, min and max, to combine feature maps in each group. To further improve efficiency, we use local binary pattern to encode hidden and output maps of Gabor convolutional layers. In addition, we use a weight vector to optimize concatenation of histograms for further improvement. Experiments show that our method achieves very outstanding results on smoke, texture and material image datasets. Although the feature extraction step of our method is training-free, our framework consistently outperforms state-of-the-art methods on small smoke datasets, even including deep learning-based methods.

Grape detection, segmentation, and tracking using deep neural networks and three-dimensional association

Agricultural applications such as yield prediction, precision agriculture and automated harvesting need systems able to infer the crop state from low-cost sensing devices. Proximal sensing using affordable cameras combined with computer vision has seen a promising alternative, strengthened after the advent of convolutional neural networks (CNNs) as an alternative for challenging pattern recognition problems in natural images. Considering fruit growing monitoring and automation, a fundamental problem is the detection, segmentation and counting of individual fruits in orchards. Here we show that for wine grapes, a crop presenting large variability in shape, color, size and compactness, grape clusters can be successfully detected, segmented and tracked using state-of-the-art CNNs. In a test set containing 408 grape clusters from images taken on a trellis-system based vineyard, we have reached an F1-score up to 0.91 for instance segmentation, a fine separation of each cluster from other structures in the image that allows a more accurate assessment of fruit size and shape. We have also shown as clusters can be identified and tracked along video sequences recording orchard rows. We also present a public dataset containing grape clusters properly annotated in 300 images and a novel annotation methodology for segmentation of complex objects in natural images. The presented pipeline for annotation, training, evaluation and tracking of agricultural patterns in images can be replicated for different crops and production systems. It can be employed in the development of sensing components for several agricultural and environmental applications.

Deep pancreas segmentation with uncertain regions of shadowed sets

Pancreas segmentation is a challenging task in medical image analysis especially for the patients with pancreatic cancer. First, the images often have poor contrast and blurred boundaries. Second, there exist large variations in gray scale, texture, location, shape and size among pancreas images. It becomes even worse with cases of pancreatic cancer. Besides, as an inevitable phenomenon, some of the slices have disconnected topology in pancreas part. All these problems lead to high segmentation uncertainties and make the results inaccurate. Existing pancreas segmentation methods rarely achieve sufficiently accurate and robust results especially for cancer cases. To tackle these problems, we propose a 2D deep learning-based method which can involve uncertainties in the process of segmentation iteratively. The proposed method describes the uncertain regions of pancreatic MRI images based on shadowed sets theory. The results are further corrected through increasing the weights of uncertain regions in iterative training. We evaluate our approach on a challenging pancreatic cancer MRI images dataset collected from the Changhai Hospital, and also validate our approach on the NIH pancreas segmentation dataset. The experimental results demonstrate that our proposed method outperforms the state-of-the-art methods in terms of the Dice similarity coefficient of 73.88% on cancer MRI dataset and 84.37% on NIH dataset respectively.

CT Male Pelvic Organ Segmentation via Hybrid Loss Network With Incomplete Annotation.

Sufficient data with complete annotation is essential for training deep models to perform automatic and accurate segmentation of CT male pelvic organs, especially when such data is with great challenges such as low contrast and large shape variation. However, manual annotation is expensive in terms of both finance and human effort, which usually results in insufficient completely annotated data in real applications. To this end, we propose a novel deep framework to segment male pelvic organs in CT images with incomplete annotation delineated in a very user-friendly manner. Specifically, we design a hybrid loss network derived from both voxel classification and boundary regression, to jointly improve the organ segmentation performance in an iterative way. Moreover, we introduce a label completion strategy to complete the labels of the rich unannotated voxels and then embed them into the training data to enhance the model capability. To reduce the computation complexity and improve segmentation performance, we locate the pelvic region based on salient bone structures to focus on the candidate segmentation organs. Experimental results on a large planning CT pelvic organ dataset show that our proposed method with incomplete annotation achieves comparable segmentation performance to the state-of-the-art methods with complete annotation. Moreover, our proposed method requires much less effort of manual contouring from medical professionals such that an institutional specific model can be more easily established.

Unified model for interpreting multi-view echocardiographic sequences without temporal information

The robust and fully automatic interpretation of multi-view echocardiographic sequences across multi-vendor and multi-center is a challenging task due to abounding artifacts, low signal-to-noise ratio, large shape variations among different views, and large gaps across different centers and vendors. In this paper, a dense pyramid and deep supervision network (DPSN) is proposed to tackle this challenging task. DPSN incorporates the advantages of the densely connected network, feature pyramid network, and deeply supervised network, which help to extract and fuse multi-level and multi-scale holistic semantic information. This capability endows DPSN with prominent generalization and robustness, enabling it to yield a precise interpretation. To reduce the computational complexity and avoid the frequent information loss in temporal modeling, DPSN processes all frames independently (i.e., without utilizing temporal information) but can still obtain stable and coherent performance in the sequence. Adequate experiments on the heterogeneous (multi-view, multi-center, and multi-vendor) dataset (10858 labeled images) corroborate that DPSN achieves not only superior segmentation results but also prominent computational efficiency and stable performance. Estimation of the ejection fraction also shows good clinical correlation, revealing the clinical potential of DPSN.

Automatic Pancreas Segmentation via Coarse Location and Ensemble Learning

Automatic and reliable segmentation of the pancreas is an important but difficult task for various clinical applications, such as pancreatic cancer radiotherapy and computer-aided diagnosis (CAD). The main challenges for accurate CT pancreas segmentation lie in two aspects: (1) large shape variation across different patients, and (2) low contrast and blurring around the pancreas boundary. In this paper, we propose a two-stage, ensemble-based fully convolutional neural network (FCN) to solve the challenging pancreas segmentation problem in CT images. First, candidate region generation is performed by classifying patches generated by superpixels. Second, five FCNs based on the U-Net architecture are trained with different objective functions. For each network, 2.5D slices are used as the input to provide 3D image information complementarily without the need for computationally expensive 3D convolutions. Then, an ensemble model is utilized to combine the five output segmentation maps and achieve the final segmentation. The proposed method is extensively evaluated on a publicly available dataset of 82 manually segmented CT volumes via 4-fold cross-validation. Experimental results show its superior performance compared with several state-of-the-art methods with a Dice coefficient of 84.10±4.91% and Jaccard coefficient of 72.86±6.89%.

PoSeg: Pose-Aware Refinement Network for Human Instance Segmentation

Human instance segmentation is a core problem for human-centric scene understanding and segmenting human instances poses a unique challenge to vision systems due to large intra-class variations in both appearance and shape, and complicated occlusion patterns. In this paper, we propose a new pose-aware human instance segmentation method. Compared to the previous pose-aware methods which first predict bottom-up poses and then estimate instance segmentation on top of predicted poses, our method integrates both top-down and bottom-up cues for an instance: it adopts detection results as human proposals and jointly estimates human pose and instance segmentation for each proposal. We develop a modular recurrent deep network that utilizes pose estimation to refine instance segmentation in an iterative manner. Our refinement modules exploit pose cues in two levels: as a coarse shape prior and local part attention. We evaluate our approach on two public multi-person benchmarks: OCHuman dataset and COCOPersons dataset. The proposed method surpasses the state-of-the-art methods on OCHuman dataset by 3.0 mAP and on COCOPersons by 6.4 mAP, demonstrating the effectiveness of our approach.

Disentangling phylogenetic and functional components of shape variation among shoot apical meristems of a wide range of herbaceous angiosperms.

The shoot apical meristem (SAM) is the basic determinant of plant body organization, but interspecific variation in SAM shape and its relationship to stem and leaf morphological traits is not well known. Here we tested the hypothesis that different SAM shapes are associated with specific shoot traits of the plant body and examined the phylogenetic conservatism of these relationships. We used geometric morphometrics of SAM outlines for a phylogenetically representative set of 110 herbaceous angiosperms and examined their relationship to a number of shoot traits. We found large variations in SAM shapes across angiosperm lineages, but covering only a subset of geometrically possible shapes. Part of this variation was allometric (due to SAM size), but the dominant shape variation (dome-shaped vs. flat surface) was size-independent and strongly phylogenetically conserved. SAM shapes were largely independent of their cell size and therefore of the number of cells involved. Different patterns in shape variation of outer and inner SAM boundaries were associated with stem thickness, leaf area, and leafiness of the stem. The findings show that geometric interdependence of meristem zones gives rise to correlations among organ numbers, sizes, and their proportions. Phylogenetic conservatism in these correlations indicates conservatism in regulatory processes that underlie the correlations, or the individual traits, that give rise to plant architecture.

Treetop detection using convolutional neural networks trained through automatically generated pseudo labels

ABSTRACTUsing remote sensing techniques to detect trees at the individual level is crucial for forest management while finding the treetop is an initial and important first step. However, due to the large variations of tree size and shape, traditional unsupervised treetop detectors need to be carefully designed with heuristic knowledge making an efficient and versatile treetop detection still challenging. Currently, the deep convolutional neural networks (CNNs) have shown powerful capabilities to classify and segment images, but the required volume of labelled data for the training impedes their applications. Considering the strengths and limitations of the unsupervised and deep learning methods, we propose a framework using the automatically generated pseudo labels from unsupervised treetop detectors to train the CNNs, which saves the manual labelling efforts. In this study, we use multi-view satellite imagery derived digital surface model (DSM) and multispectral orthophoto as research data and train the fully convolutional networks (FCN) with pseudo labels separately generated from two unsupervised treetop detectors: top-hat by reconstruction (THR) operation and local maxima filter with a fixed window (FFW). The experiments show the FCN detectors trained by pseudo labels, have much better detection accuracies than the unsupervised detectors (6.5% for THR and 11.1% for FFW), especially in the densely forested area (more than 20% of improvement). In addition, our comparative experiments when using manually labelled samples show the proposed treetop detection framework has the potential to significantly reduce the need for training samples while keep a comparable performance.