Multiple Deep Convolutional Neural Networks Research Articles

Flow Field Estimation with Distortion Correction Based on Multiple Input Deep Convolutional Neural Networks and Hartmann–Shack Wavefront Sensing

The precise estimation of fluid motion is critical across various fields, including aerodynamics, hydrodynamics, and industrial fluid mechanics. However, refraction at complex interfaces in the light path can cause image deterioration and lead to severe measurement errors if the aberration changes with time, e.g., at fluctuating air–water interfaces. This challenge is particularly pronounced in technical energy conversion processes such as bubble formation in electrolysis, droplet formation in fuel cells, or film flows. In this paper, a flow field estimation algorithm that can perform the aberration correction function is proposed, which integrates the flow field distribution estimation algorithm based on the Particle Image Velocimetry (PIV) technique and the novel actuator-free adaptive optics technique. Two different multi-input convolutional neural network (CNN) structures are established, with two frames of distorted PIV images and measured wavefront distortion information as inputs. The corrected flow field results are directly output, which are divided into two types based on different network structures: dense estimation and sparse estimation. Based on a series of models, a corresponding dataset synthesis model is established to generate training datasets. Finally, the algorithm performance is evaluated from different perspectives. Compared with traditional algorithms, the two proposed algorithms achieves reductions in the root mean square value of velocity residual error by 84% and 89%, respectively. By integrating both flow field measurement and novel adaptive optics technique into deep CNNs, this method lays a foundation for future research aimed at exploring more intricate distortion phenomena in flow field measurement.

A novel fusion algorithm for benign-malignant lung nodule classification on CT images

The accurate recognition of malignant lung nodules on CT images is critical in lung cancer screening, which can offer patients the best chance of cure and significant reductions in mortality from lung cancer. Convolutional Neural Network (CNN) has been proven as a powerful method in medical image analysis. Radiomics which is believed to be of interest based on expert opinion can describe high-throughput extraction from CT images. Graph Convolutional Network explores the global context and makes the inference on both graph node features and relational structures. In this paper, we propose a novel fusion algorithm, RGD, for benign-malignant lung nodule classification by incorporating Radiomics study and Graph learning into the multiple Deep CNNs to form a more complete and distinctive feature representation, and ensemble the predictions for robust decision-making. The proposed method was conducted on the publicly available LIDC-IDRI dataset in a 10-fold cross-validation experiment and it obtained an average accuracy of 93.25%, a sensitivity of 89.22%, a specificity of 95.82%, precision of 92.46%, F1 Score of 0.9114 and AUC of 0.9629. Experimental results illustrate that the RGD model achieves superior performance compared with the state-of-the-art methods. Moreover, the effectiveness of the fusion strategy has been confirmed by extensive ablation studies. In the future, the proposed model which performs well on the pulmonary nodule classification on CT images will be applied to increase confidence in the clinical diagnosis of lung cancer.

Pathological brain classification using multiple kernel-based deep convolutional neural network

Pathological brain classification using multiple kernel-based deep convolutional neural network

Chest radiography deep radiomics-enabled aortic arch calcification interpretation across different populations

Earlier detection of aortic calcification can facilitate subsequent cardiovascular care planning. Opportunistic screening based on plain chest radiography is potentially feasible in various population. We used multiple deep convolutional neural network (CNN) transfer learning by fine-tuning pre-trained models followed by ensemble technique for aortic arch calcification on chest radiographs from a derivation and two external databases with distinct features. Our ensemble approach achieved 84.12% precision, 84.70% recall, and an area under the receiver-operating-characteristic curve (AUC) of 0.85 in the general population/older adult's dataset. We also obtained 87.5% precision, 85.56% recall, and an AUC of 0.86 in the pre-end-stage kidney disease (pre-ESKD) cohort. We identified discriminative regions for identifying aortic arch calcification between patients without and with pre-ESKD. These findings are expected to optimize cardiovascular risk prediction if our model is incorporated into the process of routine care.

Can artificial intelligence accelerate the diagnosis of inherited retinal diseases? Protocol for a data-only retrospective cohort study (Eye2Gene)

IntroductionInherited retinal diseases (IRD) are a leading cause of visual impairment and blindness in the working age population. Mutations in over 300 genes have been found to be associated with...

Speech Emotion Recognition Based on Multiple Acoustic Features and Deep Convolutional Neural Network

Speech emotion recognition (SER) plays a vital role in human–machine interaction. A large number of SER schemes have been anticipated over the last decade. However, the performance of the SER systems is challenging due to the high complexity of the systems, poor feature distinctiveness, and noise. This paper presents the acoustic feature set based on Mel frequency cepstral coefficients (MFCC), linear prediction cepstral coefficients (LPCC), wavelet packet transform (WPT), zero crossing rate (ZCR), spectrum centroid, spectral roll-off, spectral kurtosis, root mean square (RMS), pitch, jitter, and shimmer to improve the feature distinctiveness. Further, a lightweight compact one-dimensional deep convolutional neural network (1-D DCNN) is used to minimize the computational complexity and to represent the long-term dependencies of the speech emotion signal. The overall effectiveness of the proposed SER systems’ performance is evaluated on the Berlin Database of Emotional Speech (EMODB) and the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) datasets. The proposed system gives an overall accuracy of 93.31% and 94.18% for the EMODB and RAVDESS datasets, respectively. The proposed MFCC and 1-D DCNN provide greater accuracy and outpace the traditional SER techniques.

A hybrid deep learning approach for musical difficulty estimation of piano symbolic music

Musical difficulty estimation is an essential part of musical learning. Without a precise estimate, a music learner cannot choose a piece to play according to their current level. This problem is highly complicated for its subjectivity and data scarcity. In this study, we investigate deep learning approaches to solve this problem. Our pipeline can be summarized as follows: firstly, we convert the symbolic music MIDI file of a piano performance to piano roll representation. Secondly, the piano roll is divided into smaller parts. Finally, a model is trained on parts accompanied by the corresponding difficulty labels. We test our models on both complete and partial track difficulty classification problems. Multiple deep convolutional neural networks are proposed and evaluated. Accompanied with handcrafted features, the proposed hybrid deep model yields a relative F1 score improvement of more than 10% compared to previous studies, achieving a state-of-the-art F1 score of 76.26%. Besides the direct application of the work to classify musical pieces, the promising results can be a starting point for more complicated applications, including automated difficulty-controlled music generation.

Comparison of Fine-Tuned Deep Convolutional Neural Networks for the Automated Classification of Lung Cancer Cytology Images with Integration of Additional Classifiers.

Objective:It is essential to accurately diagnose and classify histological subtypes into adenocarcinoma (ADC), squamous cell carcinoma (SCC), and small cell lung carcinoma (SCLC) for the appropriate treatment of lung cancer patients. However, improving the accuracy and stability of diagnosis is challenging, especially for non-small cell carcinomas. The purpose of this study was to compare multiple deep convolutional neural network (DCNN) technique with subsequent additional classifiers in terms of accuracy and characteristics in each histology. Methods:Lung cancer cytological images were classified into ADC, SCC, and SCLC with four fine-tuned DCNN models consisting of AlexNet, GoogLeNet (Inception V3), VGG16 and ResNet50 pretrained by natural images in ImageNet database. For more precise classification, the figures of 3 histological probabilities were further applied to subsequent machine learning classifiers using Naïve Bayes (NB), Support vector machine (SVM), Random forest (RF), and Neural network (NN). Results:The classification accuracies of the AlexNet, GoogLeNet, VGG16 and ResNet50 were 74.0%, 66.8%, 76.8% and 74.0%, respectively. Well differentiated typical morphologies were tended to be correctly judged by all four architectures. However, poorly differentiated non-small cell carcinomas lacking typical structures were inclined to be misrecognized in some DCNNs. Regarding the histological types, ADC were best judged by AlexNet and SCC by VGG16. Subsequent machine learning classifiers of NB, SVV, RF, and NN improved overall accuracies to 75.1%, 77.5%, 78.2%, and 78.9%, respectively. Conclusion:Fine-tuning DCNNs in combination with additional classifiers improved classification of cytological diagnosis of lung cancer, although classification bias could be indicated among DCNN architectures.

Operational Scheduling of Behind-the-Meter Storage Systems Based on Multiple Nonstationary Decomposition and Deep Convolutional Neural Network for Price Forecasting.

In the competitive electricity market, electricity price reflects the relationship between power supply and demand and plays an important role in the strategic behavior of market players. With the development of energy storage systems after watt-hour meter, accurate price prediction becomes more and more crucial in the energy management and control of energy storage systems. Due to the great uncertainty of electricity price, the performance of the general electricity price forecasting models is not satisfactory to be adopted in practice. Therefore, in this paper, we propose a novel electricity price forecasting strategy applied in optimization for the scheduling of battery energy storage systems. At first, multiple nonstationary decompositions are presented to extract the most significant components in price series, which express remarkably discriminative features in price fluctuation for regression prediction. In addition, all extracted components are delivered to a devised deep convolution neural network with multiscale dilated kernels for multistep price forecasting. At last, more advanced price fluctuation detection serves the optimized operation of the battery energy storage system within Ontario grid-connected microgrids. Sufficient ablation studies showed that our proposed price forecasting strategy provides predominant performances compared with the state-of-the-art methods and implies a promising prospect in economic benefits of battery energy storage systems.

An efficient framework for zero-shot sketch-based image retrieval

Zero-shot sketch-based image retrieval (ZS-SBIR) has recently attracted the attention of the computer vision community due to its real-world applications, and the more realistic and challenging setting that it presents over SBIR. ZS-SBIR inherits the main challenges of multiple computer vision problems including content-based Image Retrieval (CBIR), zero-shot learning and domain adaptation. The majority of previous studies using deep neural networks have achieved improved results by either projecting sketch and images into a common low-dimensional space, or transferring knowledge from seen to unseen classes. However, those approaches are trained with complex frameworks composed of multiple deep convolutional neural networks (CNNs) and are dependent on category-level word labels. This increases the requirements for training resources and datasets. In comparison, we propose a simple and efficient framework that does not require high computational training resources, and learns the semantic embedding space from a vision model rather than a language model, as is done by related studies. Furthermore, at training and inference stages our method only uses a single CNN. In this work, a pre-trained ImageNet CNN (i.e., ResNet50) is fine-tuned with three proposed learning objects: domain-balanced quadruplet loss, semantic classification loss, and semantic knowledge preservation loss. The domain-balanced quadruplet and semantic classification losses are introduced to learn discriminative, semantic and domain invariant features by considering ZS-SBIR as an object detection and verification problem. To preserve semantic knowledge learned with ImageNet and exploit it for unseen categories, the semantic knowledge preservation loss is proposed. To reduce computational cost and increase the accuracy of the semantic knowledge distillation process, ground-truth semantic knowledge is prepared in a class-oriented fashion prior to training. Extensive experiments are conducted on three challenging ZS-SBIR datasets: Sketchy Extended, TU-Berlin Extended and QuickDraw Extended. The proposed method achieves state-of-the-art results, and outperforms the majority of related works by a substantial margin.

EpistoNet: an ensemble of Epistocracy-optimized mixture of experts for detecting COVID-19 on chest X-ray images

The Coronavirus has spread across the world and infected millions of people, causing devastating damage to the public health and global economies. To mitigate the impact of the coronavirus a reliable, fast, and accurate diagnostic system should be promptly implemented. In this study, we propose EpistoNet, a decision tree-based ensemble model using two mixtures of discriminative experts to classify COVID-19 lung infection from chest X-ray images. To optimize the architecture and hyper-parameters of the designed neural networks, we employed Epistocracy algorithm, a recently proposed hyper-heuristic evolutionary method. Using 2500 chest X-ray images consisting of 1250 COVID-19 and 1250 non-COVID-19 cases, we left out 500 images for testing and partitioned the remaining 2000 images into 5 different clusters using K-means clustering algorithm. We trained multiple deep convolutional neural networks on each cluster to help build a mixture of strong discriminative experts from the top-performing models supervised by a gating network. The final ensemble model obtained 95% accuracy on COVID-19 images and 93% accuracy on non-COVID-19. The experimental results show that EpistoNet can accurately, and reliably be used to detect COVID-19 infection in the chest X-ray images, and Epistocracy algorithm can be effectively used to optimize the hyper-parameters of the proposed models.

Smartphone-based food recognition system using multiple deep CNN models

People with blindness or low vision utilize mobile assistive tools for various applications such as object recognition, text recognition, etc. Most of the available applications are focused on recognizing generic objects. And they have not addressed the recognition of food dishes and fruit varieties. In this paper, we propose a smartphone-based system for recognizing the food dishes as well as fruits for children with visual impairments. The Smartphone application utilizes a trained deep CNN model for recognizing the food item from the real-time images. Furthermore, we develop a new deep convolutional neural network (CNN) model for food recognition using the fusion of two CNN architectures. The new deep CNN model is developed using the ensemble learning approach. The deep CNN food recognition model is trained on a customized food recognition dataset.The customized food recognition dataset consists of 29 varieties of food dishes and fruits. Moreover, we analyze the performance of multiple state of art deep CNN models for food recognition using the transfer learning approach. The ensemble model performed better than state of art CNN models and achieved a food recognition accuracy of 95.55 % in the customized food dataset. In addition to that, the proposed deep CNN model is evaluated in two publicly available food datasets to display its efficacy for food recognition tasks.

Multispectral Face Recognition Using Transfer Learning with Adaptation of Domain Specific Units.

Facial recognition is a method of identifying or authenticating the identity of people through their faces. Nowadays, facial recognition systems that use multispectral images achieve better results than those that use only visible spectral band images. In this work, a novel architecture for facial recognition that uses multiple deep convolutional neural networks and multispectral images is proposed. A domain-specific transfer-learning methodology applied to a deep neural network pre-trained in RGB images is shown to generalize well to the multispectral domain. We also propose a skin detector module for forgery detection. Several experiments were planned to assess the performance of our methods. First, we evaluate the performance of the forgery detection module using face masks and coverings of different materials. A second study was carried out with the objective of tuning the parameters of our domain-specific transfer-learning methodology, in particular which layers of the pre-trained network should be retrained to obtain good adaptation to multispectral images. A third study was conducted to evaluate the performance of support vector machines (SVM) and k-nearest neighbor classifiers using the embeddings obtained from the trained neural network. Finally, we compare the proposed method with other state-of-the-art approaches. The experimental results show performance improvements in the Tufts and CASIA NIR-VIS 2.0 multispectral databases, with a rank-1 score of 99.7% and 99.8%, respectively.

A comparative analysis between late fusion of features approach and ensemble of multiple classifiers approach for image classification

SummaryIn recent times the late fusion of features approach for high‐level features extracted by multiple deep convolutional neural networks (DCNNs) has proven to be very effective in the computer vision field, especially for object classification problems. Pretrained DCNNs DenseNet‐121 and ResNet‐18 are retrained, keeping the number of output nodes equal to the number of classes present in the dataset. The last fully connected layers of these networks thereby get adapted to transform the high‐level features to a low‐dimensional feature map. Then these maps are fused to improve the performance of the model. On the other hand, an ensemble of multiple classifiers reduces the overfitting problem by combining multiple models prediction matrices. In this work, the prediction matrices of two Logsoftmax multiclass classifiers are combined. The feature maps for these two classifiers are extracted using pretrained DenseNet and ResNet. This study compares the late fusion of high‐level features approach and ensemble of multiple classifiers approach for object classification problems. Experimentation has been carried out on two benchmark datasets, such as CIFAR‐10 and CIFAR‐100, and it achieves 96.48% and 83.33% of test accuracy for ensemble of multiple classifiers and the late feature fusion approach. The proposed method has been compared with other deep architectures and datasets.

Spatio-Temporal Downscaling of Climate Data Using Convolutional and Error-Predicting Neural Networks

Numerical weather and climate simulations nowadays produce terabytes of data, and the data volume continues to increase rapidly since an increase in resolution greatly benefits the simulation of weather and climate. In practice, however, data is often available at lower resolution only, for which there are many practical reasons, such as data coarsening to meet memory constraints, limited computational resources, favoring multiple low-resolution ensemble simulations over few high-resolution simulations, as well as limits of sensing instruments in observations. In order to enable a more insightful analysis, we investigate the capabilities of neural networks to reconstruct high-resolution data from given low-resolution simulations. For this, we phrase the data reconstruction as a super-resolution problem from multiple data sources, tailored toward meteorological and climatological data. We therefore investigate supervised machine learning using multiple deep convolutional neural network architectures to test the limits of data reconstruction for various spatial and temporal resolutions, low-frequent and high-frequent input data, and the generalization to numerical and observed data. Once such downscaling networks are trained, they serve two purposes: First, legacy low-resolution simulations can be downscaled to reconstruct high-resolution detail. Second, past observations that have been taken at lower resolutions can be increased to higher resolutions, opening new analysis possibilities. For the downscaling of high-frequent fields like precipitation, we show that error-predicting networks are far less suitable than deconvolutional neural networks due to the poor learning performance. We demonstrate that deep convolutional downscaling has the potential to become a building block of modern weather and climate analysis in both research and operational forecasting, and show that the ideal choice of the network architecture depends on the type of data to predict, i.e., there is no single best architecture for all variables.

Analysis of tumor nuclear features using artificial intelligence to predict response to neoadjuvant chemotherapy in high-risk breast cancer patients.

Neoadjuvant chemotherapy (NAC) is used to treat patients with high-risk breast cancer. The tumor response to NAC can be classified as either a pathological partial response (pPR) or pathological complete response (pCR), defined as complete eradication of invasive tumor cells, with a pCR conferring a significantly lower risk of recurrence. Predicting the response to NAC, however, remains a significant clinical challenge. The objective of this study was to determine if analysis of nuclear features on core biopsies using artificial intelligence (AI) can predict response to NAC. Fifty-eight HER2-positive or triple-negative breast cancer patients were included in this study (pCR n = 37, pPR n = 21). Multiple deep convolutional neural networks were developed to automate tumor detection and nuclear segmentation. Nuclear count, area, and circularity, as well as image-based first- and second-order features including mean pixel intensity and correlation of the gray-level co-occurrence matrix (GLCM-COR) were determined. In univariate analysis, the pCR group had fewer multifocal/multicentric tumors, higher nuclear intensity, and lower GLCM-COR compared to the pPR group. In multivariate binary logistic regression, tumor multifocality/multicentricity (OR = 0.14, p = 0.012), nuclear intensity (OR = 1.23, p = 0.018), and GLCM-COR (OR = 0.96, p = 0.043) were each independently associated with likelihood of achieving a pCR, and the model was able to successful classify 79% of cases (62% for pPR and 89% for pCR). Analysis of tumor nuclear features using digital pathology/AI can significantly improve models to predict pathological response to NAC.

Composite deep neural network with gated-attention mechanism for diabetic retinopathy severity classification

Diabetic Retinopathy (DR) is a micro vascular complication caused by long-term diabetes mellitus. Unidentified diabetic retinopathy leads to permanent blindness. Early identification of this disease requires frequent complex diagnostic procedure which is expensive and time consuming. In this article, we propose a composite deep neural network architecture with gated-attention mechanism for automated diagnosis of diabetic retinopathy. The feature descriptors obtained from multiple pre-trained deep Convolutional Neural Networks (CNNs) are used to represent color fundus retinal images. Spatial pooling methods are introduced to get the reduced versions of these representations without loosing much information. The proposed composite DNN learns independently from each of these reduced representations through different channels and contributes to improving the model generalization. In addition, model also includes gated attention blocks which allows the model to emphasize more on lesion portions of the retinal images while reduced attention to the non-lesion regions. Our experiments on APTOS-2019 Kaggle blindness detection challenge reveal that, the proposed approach leads to improved performance when compared to the existing best models. Our empirical studies also reveal that, the proposed approach leads to more generalised predictions with multi-modal representations when compared to those of uni-modal representations. The proposed composite deep neural network model recorded an accuracy of 82.54% ( $$\uparrow $$ 2%), and a Kappa score of 79 ( $$\uparrow 9$$ points) for diabetic retinopathy severity level prediction.

Learning deep multimodal affective features for spontaneous speech emotion recognition

Recently, spontaneous speech emotion recognition has become an active and challenging research subject. This paper proposes a new method of spontaneous speech emotion recognition by using deep multimodal audio feature learning based on multiple deep convolutional neural networks (multi-CNNs). The proposed method initially generates three different audio inputs for multi-CNNs so as to learn deep multimodal segment-level features from the original 1D audio signal in three aspects: 1) a 1D CNN for 1D raw waveform modeling, 2) a 2D CNN for 2D time-frequency Mel-spectrogram modeling, and 3) a 3D CNN for temporal-spatial dynamic modeling. Then, an average-pooling is performed on the obtained segment-level classification results from 1D, 2D, and 3D CNN networks, to produce utterance-level classification results. Finally, a score-level fusion strategy is adopted as a multi-CNN fusion method to integrate different utterance-level classification results for final emotion classification. The learned deep multimodal audio features are shown to be complementary to each other so that they are combined in a multi-CNN fusion network to achieve significantly improved emotion classification performance. Experiments are conducted on two challenging spontaneous emotional speech datasets, i.e., the AFEW5.0 and BAUM-1 s databases, demonstrating the promising performance of our proposed method.

Multi-frequency and multi-domain human activity recognition based on SFCW radar using deep learning

Deep learning and radar make it feasible to automatically recognize human activities in various lighting conditions, even occlusion case, which significantly promotes the application of activity recognition in the fields of security surveillance, health care, and so on. In this paper, an approach for human activity recognition (HAR) using deep learning is proposed based on stepped frequency continues wave (SFCW) radar. Specifically, SFCW radar is utilized to generate two types of characteristic representation domains, namely multiple frequencies of spectrograms in time–frequency domain and range maps in range domain. On the one hand, spectrograms and range maps provide different types of features. On the other hand, multi-frequency spectrograms furnish same type of features while with different scattering properties and frequency resolutions. Then a specific deep learning network including multiple parallel deep convolutional neural networks (DCNNs) and a sparse autoencoder is designed to extract and fuse these features associated with human activities from the multi-frequency spectrograms and rang map. In particular, each DCNN is aimed at extracting the detailed micro-Doppler features from a spectrogram, while sparse autoencoder learns prime range distribution features by compressing each range map to reduce complexity and improve robustness. Experimental results verify that the proposed deep learning scheme achieves 96.42% recognition accuracy about six types of activities by incorporating three frequencies of spectrograms and range map, and surpasses two existed methods depending on single-frequency spectrogram and combination of single-frequency spectrogram and range map.

Local minima found in the subparameter space can be effective for ensembles of deep convolutional neural networks

Ensembles of deep convolutional neural networks (CNNs), which integrate multiple deep CNN models to achieve better generalization for an artificial intelligence application, now play an important role in ensemble learning due to the dominant position of deep learning. However, the usage of ensembles of deep CNNs is still not adequate because the increasing complexity of deep CNN architectures and the emerging data with large dimensionality have made the training stage and testing stage of ensembles of deep CNNs inevitably expensive. To alleviate this situation, we propose a new approach that finds multiple models converging to local minima in subparameter space for ensembles of deep CNNs. The subparameter space here refers to the space constructed by a partial selection of parameters, instead of the entire set of parameters, of a deep CNN architecture. We show that local minima found in the subparameter space of a deep CNN architecture can in fact be effective for ensembles of deep CNNs to achieve better generalization. Moreover, finding local minima in the subparameter space of a deep CNN architecture is more affordable at the training stage, and the multiple models at the found local minima can also be selectively fused to achieve better ensemble generalization while limiting the expense to a single deep CNN model at the testing stage. Demonstrations of MobilenetV2, Resnet50 and InceptionV4 (deep CNN architectures from lightweight to complex) on ImageNet, CIFAR-10 and CIFAR-100, respectively, lead us to believe that finding local minima in the subparameter space of a deep CNN architecture could be leveraged to broaden the usage of ensembles of deep CNNs.