3D Convolutional Neural Network Model Research Articles

An ensemble 3D deep-learning model to predict protein metal-binding site

Predicting metal-binding sites in proteins is critical for understanding the protein’s biological function. Here, we develop an ensemble deep convolutional neural network (CNN) method for predicting metal-binding sites based on their three-dimensional (3D) structure. We build multi-channel 3D voxels based on biophysical characteristics obtained from raw atom coordinates of each protein-binding pocket. Then, we use these 3D voxels as the input of an ensemble 3D CNN model. We train and evaluate the model using a curated dataset of 3D protein structures. Our proposed model shows high performance in predicting metal-binding sites for Zn, Fe, Mg, Mn, Ca, and Na. Our approach offers a framework to use 3D spatial features to train 3D-CNN, which may be used to predict complicated metal-binding sites directly from their biophysical characteristics. The source code and webserver of the model are publicly available.

The multiscale 3D convolutional network for emotion recognition based on electroencephalogram.

Emotion recognition based on EEG (electroencephalogram) has become a research hotspot in the field of brain-computer interfaces (BCI). Compared with traditional machine learning, the convolutional neural network model has substantial advantages in automatic feature extraction in EEG-based emotion recognition. Motivated by the studies that multiple smaller scale kernels could increase non-linear expression than a larger scale, we propose a 3D convolutional neural network model with multiscale convolutional kernels to recognize emotional states based on EEG signals. We select more suitable time window data to carry out the emotion recognition of four classes (low valence vs. low arousal, low valence vs. high arousal, high valence vs. low arousal, and high valence vs. high arousal). The results using EEG signals in the DEAP and SEED-IV datasets show accuracies for our proposed emotion recognition network model (ERN) of 95.67 and 89.55%, respectively. The experimental results demonstrate that the proposed approach is potentially useful for enhancing emotional experience in BCI.

Single Camera Face Position-Invariant Driver's Gaze Zone Classifier Based on Frame-Sequence Recognition Using 3D Convolutional Neural Networks.

Estimating the driver’s gaze in a natural real-world setting can be problematic for different challenging scenario conditions. For example, faces will undergo facial occlusions, illumination, or various face positions while driving. In this effort, we aim to reduce misclassifications in driving situations when the driver has different face distances regarding the camera. Three-dimensional Convolutional Neural Networks (CNN) models can make a spatio-temporal driver’s representation that extracts features encoded in multiple adjacent frames that can describe motions. This characteristic may help ease the deficiencies of a per-frame recognition system due to the lack of context information. For example, the front, navigator, right window, left window, back mirror, and speed meter are part of the known common areas to be checked by drivers. Based on this, we implement and evaluate a model that is able to detect the head direction toward these regions having various distances from the camera. In our evaluation, the 2D CNN model had a mean average recall of 74.96% across the three models, whereas the 3D CNN model had a mean average recall of 87.02%. This result show that our proposed 3D CNN-based approach outperforms a 2D CNN per-frame recognition approach in driving situations when the driver’s face has different distances from the camera.

Identification of slightly sprouted wheat kernels using hyperspectral imaging technology and different deep convolutional neural networks

Sprouted wheat kernels have a great impact on the quality of flour, bread and other wheat products, but they are hard to be identified by human eyes, especially that sprouted slightly. This study utilized hyperspectral imaging technology, combined with deep learning algorithms to classify slightly sprouted and sound wheat kernels. Hyperspectral images collected from wheat kernels placed randomly will be pre-processed by a series of commonly used pre-processing methods. And then four deep convolutional neural network (CNN) models, 1D, 2D, 3D and mixed CNNs, would be established and used as classifiers to identify the two types of wheat kernels through the spectral or hyperspectral data. Data augmentation method was also used on the training set, and the proposed 2D, 3D and mixed CNN models were trained again, and the performances on the validation set were compared to the results got before data augmentation. The results showed that data augmentation did beneficial to the performance of the proposed 2D, 3D and mixed CNN models, and the four proposed models performed well when classifying slightly sprouted wheat kernels and sound wheat kernels, getting accuracy rate of 96.81% (1D CNN model), 96.02% (2D CNN model), 98.40% (3D CNN model), and 98.12% (mixed CNN model) on testing set. The results indicated that models proposed in this paper, especially the 3D CNN model with the highest accuracy rate and the mixed CNN model with the least trainable parameters, have a good prospect of being used as classifiers to detect sprouted and sound wheat kernels.

Buoy Light Pattern Classification for Autonomous Ship Navigation Using Recurrent Neural Networks

In near coast navigation, buoys and beacons convey essential information about dangers. At night-time, selected buoys send out individual blink-sequences that are marked in sea charts. International regulations require that navigation officer on watch makes visual confirmation of objects and their type in order to navigate safely. With rapid developments of highly automated vessels, this duty needs be carried out by algorithms that detect and locate objects without human intervention. At night-time, this requires algorithms that decode blink sequences and are able to classify from this information. The paper investigates this problem and suggests an algorithm that solves the problem. Convolutional Neural Networks (CNN) with Gated Recurrent Units (GRU) are developed for classification. A dedicated architecture is suggested that includes both temporal and color decoding to obtain unique precision. We demonstrate how networks are trained on synthetically generated data, and the paper shows, on real-world data, how the suggested approach yields 100.0% accurate results on 44 real-world recordings while being robust to inaccuracy in actual blink sequences. Comparison with baseline signal processing and with a recent state-of-the-art 3D CNN model shows that the new blink-sequence classifier outperforms alternative algorithms. A showcase of the results of this work is available in this video: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://youtu.be/KEi8qNnKV2w</uri> .

3D convolutional neural network model from contrast-enhanced CT to predict spread through air spaces in non-small cell lung cancer

PurposeThe purpose of this study was to compare the efficacy of five non-invasive models, including three-dimensional (3D) convolutional neural network (CNN) model, to predict the spread through air spaces (STAS) status of non-small cell lung cancer (NSCLC), and to obtain the best prediction model to provide a basis for clinical surgery planning. Materials and methodsA total of 203 patients (112 men, 91 women; mean age, 60 years; age range 22–80 years) with NSCLC were retrospectively included. Of these, 153 were used for training cohort and 50 for validation cohort. According to the image biomarker standardization initiative reference manual, the image processing and feature extraction were standardized using PyRadiomics. The logistic regression classifier was used to build the model. Five models (clinicopathological/CT model, conventional radiomics model, computer vision (CV) model, 3D CNN model and combined model) were constructed to predict STAS by NSCLC. Area under the receiver operating characteristic curves (AUC) were used to validate the capability of the five models to predict STAS. ResultsFor predicting STAS, the 3D CNN model was superior to the clinicopathological/CT model, conventional radiomics model, CV model and combined model and achieved satisfactory discrimination performance, with an AUC of 0.93 (95% CI: 0.70–0.82) in the training cohort and 0.80 (95% CI: 0.65–0.86) in the validation cohort. Decision curve analysis indicated that, when the probability of the threshold was over 10%, the 3D CNN model was beneficial for predicting STAS status compared to either treating all or treating none of the patients within certain ranges of risk threshold ConclusionThe 3D CNN model can be used for the preoperative prediction of STAS in patients with NSCLC, and was superior to the other four models in predicting patients' risk of developing STAS.

Vegetation Type Classification Based on 3D Convolutional Neural Network Model: A Case Study of Baishuijiang National Nature Reserve

Efficient and accurate vegetation type extraction from remote sensing images can provide decision makers with basic forest cover and land use information, and provides a reliable basis for long-term monitoring. With the development of deep learning, the convolutional neural network (CNN) has been used successfully to classify tree species in many studies, but CNN models have rarely been applied in the classification of vegetation types on larger scales. To evaluate the performance of CNN models in the classification of vegetation types, this paper compared the classification accuracy of nine dominant land cover types in Baishuijiang National Nature Reserve with four models: 3D-CNN, 2D-CNN, JSSAN (joint spatial–spectral attention network) and Resnet18, using sentinel-2A data. Comparing the difference in classification accuracy between the direct use of raw sentinel images and fused feature indices sentinel images, the results showed that adding feature indices can improve the overall accuracy of the model. After fusing the characteristic bands, the accuracy of the four models was improved significantly, by 5.46–19.33%. The best performing 3D-CNN model achieved the highest classification accuracy with an overall accuracy of 95.82% and a kappa coefficient of 95.07%. In comparison, 2D-CNN achieved an overall accuracy of 79.07% and a kappa coefficient of 75.44%, JSSAN achieved an overall accuracy of 81.67% and a kappa coefficient of 78.56%, and Resnet18 achieved an overall accuracy of 93.61% and a kappa coefficient of 92.45%. The results showed that the 3D-CNN model can effectively capture vegetation type cover changes from broad-leaved forests at lower elevation, to shrublands and grasslands at higher elevation, across a range spanning 542–4007 m. In experiments using a small amount of sample data, 3D-CNN can better incorporate spatial–spectral information and is more effective in distinguishing the performance of spectrally similar vegetation types, providing an efficient and novel approach to classifying vegetation types in nature reserves with complex conditions.

Using CNN in Surveillance Videos for Recognizing Human Actions Based on Machine Learning in Examination

Video analytics is the process of processing a video, obtaining data, and analyzing the data to derive domain-specific information. In recent years, examining live surveillance films for recognizing events that occur within its coverage area has become increasingly important, in addition to analyzing any video for information retrieval. Face identification from recordings is easy when training models are used. Estimating skin colour aids hand detection. This study project aims to detect suspicious acts such as object exchange, new person admission, peeping into another's answer sheet, and person exchange using footage captured by a security camera during exam. People are paying greater attention to the fairness of exams these days, therefore detecting anomalous conduct to ensure examination order is important. Most of the current methodologies propose a fraudulent model. We extract the optical flow of video data in this system and present a 3D convolutional neural networks model to solve the problem. Face recognition, hand recognition, and detecting contact between the face and hands of the same person and that of distinct people are all required. The automation of suspicious activity detection' will aid in lowering the mistake rate associated with manual monitoring.

Convolutional neural network for automatic maxillary sinus segmentation on cone-beam computed tomographic images

An accurate three-dimensional (3D) segmentation of the maxillary sinus is crucial for multiple diagnostic and treatment applications. Yet, it is challenging and time-consuming when manually performed on a cone-beam computed tomography (CBCT) dataset. Recently, convolutional neural networks (CNNs) have proven to provide excellent performance in the field of 3D image analysis. Hence, this study developed and validated a novel automated CNN-based methodology for the segmentation of maxillary sinus using CBCT images. A dataset of 264 sinuses were acquired from 2 CBCT devices and randomly divided into 3 subsets: training, validation, and testing. A 3D U-Net architecture CNN model was developed and compared to semi-automatic segmentation in terms of time, accuracy, and consistency. The average time was significantly reduced (p-value < 2.2e−16) by automatic segmentation (0.4 min) compared to semi-automatic segmentation (60.8 min). The model accurately identified the segmented region with a dice similarity co-efficient (DSC) of 98.4%. The inter-observer reliability for minor refinement of automatic segmentation showed an excellent DSC of 99.6%. The proposed CNN model provided a time-efficient, precise, and consistent automatic segmentation which could allow an accurate generation of 3D models for diagnosis and virtual treatment planning.

A novel 3D convolutional neural network model with supervised spectral regression for recognition of hyperspectral images of colored wool fiber

AbstractPrior knowledge of textile fiber colors in blends is useful for color recipe assessment. There have been various methods to improve the accuracy of color recognition from the dataset of colored textile fibers in recent years. However, numerical assessments based on spectral feature and color difference is insufficient, in which the accuracy of color recognition can be affected by morphology and the uneven coloration on a single fiber. This paper proposes a novel 3D convolutional neural network model (3D‐CNN) with supervised spectral regression for the color recognition of hyperspectral images (HSI) of colored textile fiber. The proposed method obtained spatial‐spectral features based on 3D‐CNN, and the true spectrum of each class was used for supervised spectral regression to improve the accuracy. The loss function used was the sum of the supervised classification loss function and the spectral regression loss function model are optimized by mini‐batch‐based backpropagation. The proposed method was trained and tested on the HSI dataset composed of 100 colors of wool fibers acquired through a microscopic hyperspectral imaging system at a ×3.375 optical magnification. The experimental results showed that the proposed method exhibited better performance compared to numerical assessments and other deep learning models, except for efficiency. Specifically, it achieved better recognition performance on sub‐datasets of similar colored and light‐colored wool fiber where subtle inter‐class and large intra‐class variance existed.

Generalizable Permeability Prediction of Digital Porous Media via a Novel Multi‐Scale 3D Convolutional Neural Network

AbstractSubsurface characterization is critical in understanding and controlling many natural and industrial processes including groundwater movement, oil extraction, and geological carbon dioxide sequestration. While recent advances in three‐dimensional (3D) imaging of core samples have enabled digital subsurface characterization, the exorbitant computational cost associated with direct numerical simulation in 3D remains a persistent challenge. In contrast, machine learning models are much more efficient, though their use in subsurface characterization is still in its infancy. Here, we introduce a novel 3D convolutional neural network (CNN) for end‐to‐end prediction of permeability, which is a fundamental characteristic of subsurface porous media. We show that increasing the dataset size and diversity, utilizing multi‐scale feature aggregation, and optimizing the network architecture elevate the model accuracy beyond that of existing state‐of‐the‐art 3D CNN models for permeability prediction. We demonstrate that the model is generalizable, and it is capable of predicting the permeability of previously unseen samples with an excellent accuracy.

A Disjoint Samples-Based 3D-CNN With Active Transfer Learning for Hyperspectral Image Classification

Convolutional Neural Networks (CNNs) have been extensively studied for Hyperspectral Image Classification (HSIC). However, CNNs are critically attributed to a large number of labeled training samples, which outlays high costs in terms of time and resources. Moreover, CNNs are trained on some samples and have been tested on the entire HSI. Perhaps, the entire HSI is taken into account at test time to appropriately generate the ground truth maps. In order to obtain a higher accuracy while considering the limited availability of training samples and disjoint validation and test samples, this work proposes a fast and compact 3D CNN-based Active Learning (AL) for HSIC that integrates both deep transfer learning and AL into a unified framework. In the proposed methodology, a 3D CNN model is trained with very few training samples (i.e., 5%, only) and in the next phase, the most informative and heterogeneous samples are queried from the validation set (candidate set) based on the fuzziness, mutual information and breaking ties of the trained model. The 3D CNN model is later fine-tuned (rather retraining from scratch) with the new training samples (i.e., 200 samples are selected in each iteration) to reduce the computational cost. The proposed method has been compared with the state-of-the-art traditional and deep models proposed for HSIC. Experimental results proved the superiority of our proposed method on several benchmark HSI datasets with significantly fewer labeled samples. Matlab demo can be accessed on GitHub: github.com/mahmad00.

TimeDistributed-CNN-LSTM: A Hybrid Approach Combining CNN and LSTM to Classify Brain Tumor on 3D MRI Scans Performing Ablation Study

Identification of brain tumors and accurate grading at an early stage are crucial in cancer diagnosis, as a timely diagnosis can increase the chances of survival. Considering the challenges and risks of tumor biopsies, noninvasive imaging procedures such as Magnetic Resonance Imaging (MRI) are extensively used in analyzing brain tumors. Recent advances in the field of medical imaging with deep learning using three dimensional (3D) MRI is aiding the clinical experts significantly in the diagnosis of brain tumor. In this study, three BraTS MRI datasets named BraTS 2018, BraTS 2019 and BraTS 2020 are employed to classify brain tumor into high-grade glioma (HGG) and low-grade glioma (LGG) where each of the datasets contains four different sequences of 3D MRI brain images named T1-weighted MRI (T1), T1-weighted MRI with contrast enhancement (T1ce), T2-weighted MRI (T2), and Fluid Attenuated Inversion Recovery (FLAIR) for a single patient. This research is composed of two approaches where, in the first part, we propose a hybrid deep learning model named TimeDistributed-CNN-LSTM (TD-CNN-LSTM) combining 3D Convolutional Neural Network (CNN) and Long Short Term Memory (LSTM) where each layer of the architecture is wrapped with a TimeDistributed function. The objective of developing the hybrid model is to consider all the four 3D MRI sequences of each patient as a single input data because every sequence contains necessary information on tumor that can have an impact on improving cancer detection performance. However, interpreting all the MRI sequences together with optimal performance especially in 3D is quite challenging. Therefore, the model is developed with optimal configuration based on highest accuracy performing ablation study for layer architecture and hyper-parameters. In the second part, a 3D CNN model is trained respectively with each of the MRI sequences to compare the performance with TD-CNN-LSTM model. In this regard, for both of the models, BraTS 2018 and BraTS 2019 are combined to increase the number of images and used as train dataset where BraTS 2020 dataset is employed as the test dataset. Moreover, before training the models the datasets is preprocessed to ensure the highest performance. Our results of these two approaches demonstrate that the TD-CNN-LSTM network outperforms 3D CNN achieving the highest test accuracy of 98.90%. Later, to evaluate the performance consistency, the TD-CNN-LSTM model is evaluated with K-fold cross validation using different k values. The approach of putting together all the MRI sequences at a time using hybrid CNN-LSTM approach with good generalization capability to classify brain tumor can be used effectively in future Computer Aided Diagnosis (CAD) based research which can aid radiologists in medical diagnostics.

Joint Lymphoma Lesion Segmentation and Prognosis Prediction From Baseline FDG-PET Images via Multitask Convolutional Neural Networks

<i>Objective:</i> Lymphoma lesion segmentation and prognosis prediction from baseline FDG-PET images are valuable for tailoring and adapting a treatment plan for patients with Diffuse Large B-cell Lymphoma (DLBCL). However, the tasks are challenging due to the fact that DLBCL is a highly heterogeneous group of neoplasms and that the lymphoma cells are large and arranged in a diffuse pattern. <i>Methods:</i> We propose a novel multi-task 3D convolutional neural network model for simultaneous lymphoma lesion segmentation and prognosis prediction from baseline FDG-PET images. In our model, the learned image features of one task are shared and thereby mutually reinforce the learning of the other task. Since the dataset is limited, to reduce overfitting and to facilitate network convergence, we further introduce deep supervision into both the segmentation task and the prognosis prediction task. <i>Results:</i> Evaluated on a dataset of 269 patients, our method achieved an average Dice similarity coefficient of 0.868 for lesion segmentation, an average AUC (area under the curve) of 0.823 and an average accuracy of 0.821 for prognosis prediction. Its predictions can differentiate patients with different PFS (progression-free survival) and OS (overall survival) (<i>p</i> &#x02C2; 0.0001). <i>Conclusion:</i> Our method achieves joint lymphoma lesion segmentation and prognosis prediction from baseline FDG-PET scans. <i>Significance:</i> Our model may be used to asset the physician as a second opinion while making the final decision.

Evaluating Clinical Acceptability of Organs-at-Risk Segmentation in Head & Neck Cancer (HNC) by Open-Source 3D Convolutional Neural Networks (CNNs)

<h3>Purpose/Objective(s)</h3> To evaluate the performance and clinical acceptability of the highest objective performing 3D convolutional neural network models developed to contour regions of interest (ROIs) in medical images. <h3>Materials/Methods</h3> Radiation treatment planning data from 582 HNC patients treated at a large tertiary cancer center were used for this study. Eleven open-source 3D segmentation models (6 UNETs, 3 DenseNets, 1 3D pyramid abstraction network, and 1 3D recurrent slice-wise attention network), originally engineered for medical image segmentation, were trained to segment 19 OARs. The same 80/10/10 split was used for training, validating and testing corresponding to 479, 44 and 59 scans. A model's performance was ranked by volumetric Dice similarity coefficient (DSC) and Hausdorff distance (HD). The best performing model was selected and re-trained on full-resolution patient scans and validated using sliding-window inference. With the best model, two radiation oncologists assessed clinical acceptability, using 5-scale ratings of 5 for "anatomically perfect, not requiring editing" and score 1 for "anatomically incorrect, unusable for planning purpose." At least 10 cases were randomly selected and deep learning-based (DLCs) and manual contours (MCs), for the entire range of the OAR, were presented to the blinded observers. The mean ratings between DLC and MC were compared using Wilcoxon signed rank test and Student's T-test for the groups where significant differences between the groups represented the ability to distinguish between the MC and DLC contours by the observers. <h3>Results</h3> A simple 3D Unet performed the best among the 11 models when comparing average DSC (0.75 ± 0.12) and HD (mean HD 0.44 ± 0.09) for every OAR. Mandible (0.90 ± 0.03), brainstem (0.89 ± 0.04), and spinal cord (0.85 ± 0.05) received the highest mean DSC, and chiasm (0.38 ± 0.19), optic nerves (0.65 ± 0.15), and lips (0.70 ± 0.09) the lowest mean DSC. Lens (0.052 ± 0.019), optic nerve (0.069 ± 0.026), and chiasm (0.075 ± 0.02) received the lowest mean HD, and spinal cord (1.15 ± 0.15), mandible (1.11 ± 0.18), and brachial plexus (0.90 ± 0.12) the highest mean HD. For clinical acceptability, MC received a significantly higher rating than DLC (3.75 ± 0.77 vs. 3.23 ± 0.86) when all OARs were considered (<i>P</i> < 0.01). When evaluating OARs individually, MC showed significantly higher ratings for brainstem, esophagus, larynx, eyes, optic nerves, while lips, whereas parotids, acoustics, and lenses were indistinguishable. <h3>Conclusion</h3> Simple 3D architectures consistently outcompete more complex networks by quantitative measures. Qualitative assessment for clinical acceptability may not agree with quantitative performance, especially when the entire range of OARs is evaluated.

Automated segmentation of knee articular cartilage: Joint deep and hand-crafted learning-based framework using diffeomorphic mapping

Segmentation of knee articular cartilage tissue (ACT) from 3D magnetic resonance images (MRIs) is a fundamental task in assessing knee osteoarthritis (KOA). However, automated ACT segmentation of knee cartilage is complicated by (1) the variability of pathological structures in terms of shape, size, spatial resolution and (2) the uncertainties in delineating inter- and intra-cartilage boundaries (possibly touching inter-cartilage surfaces, intensity inhomogeneity, cartilage loss due to disease progression). To address these problems, we propose a novel joint deep and hand-crafted learning-based (JD-HCl) framework. The fully automated segmentation pipeline is a combination of three main steps: 3D convolutional neural networks (CNNs), diffeomorphic mapping, and a conventional hand-crafted feature-based classification model. Firstly, the initial segmentation of knee ACT was drawn from preprocessed knee MRIs using 3D U-Net. Secondly, the spatial alignment in the image domain was carried out using diffeomorphic image registration derived from an estimate of the anatomical correspondences between the subjects and population-specific template. Finally, a tissue-specific hand-crafted feature set was extracted from the spatially aligned pre-segmented ACT region and combined with the semantic context priors obtained from 3D CNN model to develop the second-stage learning model. The proposed method was evaluated with two publicly available Osteoarthritis Initiative (OAI) datasets and one in-house MRI sequences from the Alfred hospital, Melbourne, Australia. The proposed JD-HCl framework produced strong Dice similarity coefficient (DSC), ranging between 97.4% and 98.5% (95% confidence interval) for all cartilage compartments at baseline to 97.2% and 98.5% (the 1-year follow-up) for OAI(-Imorphics) dataset, and 89.3% and 94.6% for OAI-ZIB validation dataset, respectively. Average correlations of cartilage volumes between manual and automatic segmentations were 0.9 and 0.83 for OAI-ZIB dataset, respectively, and 0.975, 0.995, and 0.99 for OAI(-Imorphics) validation datasets, respectively. In longitudinal studies, accurate segmentation of knee ACT using the proposed method produces reproducible cartilage volume, thickness measurements valuable for the study of KOA progression.

Decoding Auditory Attentional States by a 3D Convolutional Neural Network Model

In previous research, the flower-gusset composite joint (FGC joint), which can replace traditional gusset joints, was proposed. The results of previous tests and numerical analysis show that the effect of the axial force and bolt clearance on the semi-rigid performance of FGC joints cannot be ignored. In this paper, the M-θ model of FGC joints considering the effect of the axial force and bolt clearance is proposed through theoretical analysis and test verification. First, the corresponding assumptions are proposed based on the effect of the axial force and bolt clearance on the semi-rigid performance of FGC joints. Second, the calculation formulas of the M-θ model of the FGC joint are proposed through theoretical analysis. The calculation formulas are modified by many numerical analysis results. Then, two groups of bending tests on FGC joints under different axial forces and bolt clearances are completed, and the M-θ curve of FGC joints is obtained. Finally, the M-θ curves from the tests are compared with the calculation results of the M-θ model, which show that they are very similar. Therefore, the proposed M-θ model in this paper can fully consider the effect of the axial force and bolt clearance.

An end-to-end 3D convolutional neural network for decoding attentive mental state

The detection of attentive mental state plays an essential role in the neurofeedback process and the treatment of Attention Deficit and Hyperactivity Disorder (ADHD). However, the performance of the detection methods is still not satisfactory. One of the challenges is to find a proper representation for the electroencephalogram (EEG) data, which could preserve the temporal information and maintain the spatial topological characteristics. Inspired by the deep learning (DL) methods in the research of brain–computer interface (BCI) field, a 3D representation of EEG signal was introduced into attention detection task, and a 3D convolutional neural network model with cascade and parallel convolution operations was proposed. The model utilized three cascade blocks, each consisting of two parallel 3D convolution branches, to simultaneously extract the multi-scale features. Evaluated on a public dataset containing twenty-six subjects, the proposed model achieved better performance compared with the baseline methods under the intra-subject, inter-subject and subject-adaptive classification scenarios. This study demonstrated the promising potential of the 3D CNN model for detecting attentive mental state.

Segmentation of focal cortical dysplasia lesions from magnetic resonance images using 3D convolutional neural networks

Computer-aided diagnosis using advanced Artific ial Intelligence (AI) techniques has become much popular over the last few years. This work automates the segmentation of Focal Cortical Dysplasia (FCD) lesions from three-dimensional (3D) Magnetic Resonance (MR) images. FCD is a type of neuronal malformation in the brain cortex and is the leading cause of intractable epilepsy, irrespective of gender or age differences. Since the neuron related abnormalities are usually resistant to drug therapy, surgical resection has been the main treatment approach for patients with intractable epilepsy. Automating the identification and segmentation of FCD is useful for neuroradiologists in pre-surgical evaluations. Convolutional Neural Networks (CNNs) have the ability to learn appropriate features from the training data without any human intervention. But, most of the state-of-the-art FCD segmentation approaches use two-dimensional (2D) CNN models despite the availability of 3D Magnetic resonance imaging (MRI) volumes, and hence fail to leverage the inter-slice information present in the MRI volumes. The major hurdles in considering a 3D CNN model are the need for a large 3D dataset, big memory, and high computation cost. A deep 3D CNN segmentation model, which can extract inter-slice information and overcomes the drawbacks of conventional 3D CNN methods to an extent, is proposed in this paper. The model uses a 3D version of U-Net with residual blocks that works on shallow depth 3D sub-volumes generated from MRI volumes. The proposed method shows superior performance over the state-of-the-art FCD segmentation methods in both qualitative and quantitative analysis.

Structural and diffusion MRI based schizophrenia classification using 2D pretrained and 3D naive Convolutional Neural Networks

The ability of automatic feature learning makes Convolutional Neural Network (CNN) potentially suitable to uncover the complex and widespread brain changes in schizophrenia. Despite that, limited studies have been done on schizophrenia identification using interpretable deep learning approaches on multimodal neuroimaging data. Here, we developed a deep feature approach based on pre-trained 2D CNN and naive 3D CNN models trained from scratch for schizophrenia classification by integrating 3D structural and diffusion magnetic resonance imaging (MRI) data. We found that the naive 3D CNN models outperformed the pretrained 2D CNN models and the handcrafted feature-based machine learning approach using support vector machine during both cross-validation and testing on an independent dataset. Multimodal neuroimaging-based models accomplished performance superior to models based on a single modality. Furthermore, we identified brain grey matter and white matter regions critical for illness classification at the individual- and group-level which supported the salience network and striatal dysfunction hypotheses in schizophrenia. Our findings underscore the potential of CNN not only to automatically uncover and integrate multimodal 3D brain imaging features for schizophrenia identification, but also to provide relevant neurobiological interpretations which are crucial for developing objective and interpretable imaging-based probes for prognosis and diagnosis in psychiatric disorders.