Multi-path Convolutional Neural Network Research Articles

Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition

Facial Expression Recognition (FER) has long been a challenging task in the field of computer vision. In this paper, we present a novel model, named Deep Attentive Multi-path Convolutional Neural Network (DAM-CNN), for FER. Different from most existing models, DAM-CNN can automatically locate expression-related regions in an expressional image and yield a robust image representation for FER. The proposed model contains two novel modules: an attention-based Salient Expressional Region Descriptor (SERD) and the Multi-Path Variation-Suppressing Network (MPVS-Net). SERD can adaptively estimate the importance of different image regions for FER task, while MPVS-Net disentangles expressional information from irrelevant variations. By jointly combining SERD and MPVS-Net, DAM-CNN is able to highlight expression-relevant features and generate a variation-robust representation for expression classification. Extensive experimental results on both constrained datasets (CK+, JAFFE, TFEID) and unconstrained datasets (SFEW, FER2013, BAUM-2i) demonstrate the effectiveness of our DAM-CNN model.

Deep Learning for Sensor-Based Rehabilitation Exercise Recognition and Evaluation.

In this paper, a multipath convolutional neural network (MP-CNN) is proposed for rehabilitation exercise recognition using sensor data. It consists of two novel components: a dynamic convolutional neural network (D-CNN) and a state transition probability CNN (S-CNN). In the D-CNN, Gaussian mixture models (GMMs) are exploited to capture the distribution of sensor data for the body movements of the physical rehabilitation exercises. Then, the input signals and the GMMs are screened into different segments. These form multiple paths in the CNN. The S-CNN uses a modified Lempel–Ziv–Welch (LZW) algorithm to extract the transition probabilities of hidden states as discriminate features of different movements. Then, the D-CNN and the S-CNN are combined to build the MP-CNN. To evaluate the rehabilitation exercise, a special evaluation matrix is proposed along with the deep learning classifier to learn the general feature representation for each class of rehabilitation exercise at different levels. Then, for any rehabilitation exercise, it can be classified by the deep learning model and compared to the learned best features. The distance to the best feature is used as the score for the evaluation. We demonstrate our method with our collected dataset and several activity recognition datasets. The classification results are superior when compared to those obtained using other deep learning models, and the evaluation scores are effective for practical applications.

Abstract WP11: Semi-automated Quantitative Assessment of Reperfusion in Acute Ischemic Stroke Patients

Introduction: Accurate assessment of brain reperfusion during endovascular treatment helps determining whether adjuvant intervention is necessary. The gold standard for reperfusion scales are the modified and extended Treatment in Cerebral Ischemia (mTICI and eTICI) scale. However, these scales are coarse and highly dependent on the observer, which may result in imprecise rating of reperfusion status. In this study, we developed a semi-automated quantitative reperfusion measure (qTICI) that was based on eTICI methodology using deep learning. Methods: Digital subtraction angiography (DSA) images of 67 acute ischemic stroke patients with proximal occlusions were collected from the MRCLEAN registry, including only patients who were eTICI graded by an independent core lab and had follow-up 90 days-modified Rankin Scale (mRS) scores. Both DSA projections, anteroposterior and lateral, were used. We trained a patch-based multi-path convolutional neural network to segment the perfusion area. The ground truths of perfusion segmentation were constructed using heuristic, intensity thresholding after vessel removal, which were inspected for their accuracy by a neuroradiologist. The target downstream territory was manually segmented based on the treated occlusion location by two neurointerventionalists. qTICI was formulated as a ratio between the area of perfusion and target downstream territory. We compared performance between qTICI and eTICI in relation with functional outcome, dichotomized as favorable (mRS 0-2) and unfavorable outcome (mRS 3-6). Results: Our study showed that the multi-path convolutional neural network was able to reproduce the ground truth perfusion segmentation with an average Dice score of 0.91. qTICI correlated significantly with eTICI (p<0.001) and the ordinal 90-days mRS (p<0.05). qTICI had comparable discriminative power to distinguish between favorable and unfavorable functional outcome (AUC = 0.80) with eTICI (AUC = 0.81). Conclusions: qTICI provided a quantitative and semi-automated perfusion assessment with strong association with functional outcome.

Multi-path convolutional neural network for lung cancer detection

Lung cancer is the leading cause of death among cancer-related death. Like other cancers, the finest solution for lung cancer diagnosis and treatment is early screening. Automatic CAD system of lung cancer screening from Computed Tomography scan mainly involves two steps: detect all suspicious pulmonary nodules and evaluate the malignancy of the nodules. Recently, there are many works about the first step, but rare about the second step. Since the presence of pulmonary nodules does not absolutely specify cancer, the morphology of nodules such as shape, size, and contextual information has a sophisticated relationship with cancer, the screening of lung cancer needs a careful investigation on each suspicious nodule and integration of information of all nodules. We propose deep CNN architecture which differs from those traditionally used in computer vision to solve this problem. First, the suspicious nodules are generated with the modified version of U-Net and then the generated nodules become an input data for our model. The proposed model is a multi-path CNN which exploits both local features as well as more global contextual features simultaneously to automatically detect lung cancer. To this end, the model used three paths, each path employed different receptive field size which helps to model distant dependencies (short and long-range dependencies of the neighboring pixels). Then, to further upgrade our model performance, we concatenate features from the three paths. This balance the receptive field size effect and makes our model more adaptable to the variability of shape, size, and contextual information among nodules. Finally, we also introduce a retraining phase system that permits us to tackle difficulties related to the imbalance of image labels. Experimental results on Kaggle Data Science Bowl 2017 challenge shows that our model is better adaptable to the described inconsistency among nodules size and shape, and also obtained better detection results compared to the recently published state of the art methods.

Deep dense multi-path neural network for prostate segmentation in magnetic resonance imaging.

We propose an approach of 3D convolutional neural network to segment the prostate in MR images. A 3D deep dense multi-path convolutional neural network that follows the framework of the encoder-decoder design is proposed. The encoder is built based upon densely connected layers that learn the high-level feature representation of the prostate. The decoder interprets the features and predicts the whole prostate volume by utilizing a residual layout and grouped convolution. A set of sub-volumes of MR images, centered at the prostate, is generated and fed into the proposed network for training purpose. The performance of the proposed network is compared to previously reported approaches. Two independent datasets were employed to assess the proposed network. In quantitative evaluations, the proposed network achieved 95.11 and 89.01 Dice coefficients for the two datasets. The segmentation results were robust to variations in MR images. In comparison experiments, the segmentation performance of the proposed network was comparable to the previously reported approaches. In qualitative evaluations, the segmentation results by the proposed network were well matched to the ground truth provided by human experts. The proposed network is capable of segmenting the prostate in an accurate and robust manner. This approach can be applied to other types of medical images.

Learning multi-path CNN for mural deterioration detection

Mural deterioration easily destroys valuable paintings and must be monitored frequently for preventive protection. Deterioration detection in mural images is often manually labeled and is a preprocessing step for mural protection and restoration. Many deterioration forms are commonly invisible with only one lighting condition because mural deterioration is caused by changes in the material and plaster layer. This study addresses mural deterioration detection through a multi-path convolutional neural network (CNN), which takes images of a scene with multiple lightings as inputs and generates a binary map that indicates deterioration regions. We design an eight-path CNN in which seven paths are utilized for basic feature extraction from lighted images, and the remaining path is responsible for cross feature fusion. This mechanism enables our method to not only identify suitable features for different lightings but also utilize these features collaboratively through cross feature fusion. Furthermore, we build two realistic mural deterioration datasets of real-world mural deterioration and briquettes that simulate the cave deterioration. Extensive experiments verify the effectiveness and efficiency of our method.