Image-based Convolutional Neural Networks Research Articles

A learnable continuous wavelet-based multi-branch attentive convolutional neural network for spatio–spectral–temporal EEG signal decoding

Deep learning methods, particularly convolutional neural networks (CNNs), have advanced the field of motor imagery (MI) by effectively extracting spatio-spectral–temporal (SST) features from electroencephalography (EEG) signals. Image-based CNNs using time–frequency analysis such as wavelet transform have outperformed signal-based CNNs in MI-EEG classification. However, the spatial characteristics of EEG signals can be neglected due to the high dimensionality of spectral–temporal feature space. Moreover, the performance hinges on whether the predefined transformation can capture the class-discriminative spectral–temporal representations of EEG signals well. Despite attempts to alleviate these challenges by selecting EEG electrodes or restricting spectral–temporal feature space, large inter-subject variabilities have impeded successful MI-EEG decoding. In this paper, we propose a learnable continuous wavelet-based multi-branch attentive CNN framework for decoding MI-EEG signals. Our method is designed to automatically generate class-discriminative SST representations of EEG signals within subject-specific sub-bands by utilizing learnable wavelet-based convolutions. Additionally, it employs multi-branch attentive CNNs for the effective and efficient extraction of local and global SST features. Our comprehensive evaluation on two public datasets demonstrates that the proposed method significantly outperformed state-of-the-art methods. We also conduct an ablation study under various configuration settings within the proposed framework to demonstrate the effectiveness of our method. The proposed method provides a powerful and innovative approach to MI-EEG signal decoding, with implications for personalized EEG decoding.

Ultrasound image-based deep learning to differentiate tubal-ovarian abscess from ovarian endometriosis cyst.

Objectives: We developed ultrasound (US) image-based convolutional neural networks (CNNs) to distinguish between tubal-ovarian abscess (TOA) and ovarian endometriosis cyst (OEC). Methods: A total of 202 patients who underwent US scanning and confirmed tubal-ovarian abscess or ovarian endometriosis cyst by pathology were enrolled in retrospective research, in which 171 patients (from January 2014 to September 2021) were considered the primary cohort (training, validation, and internal test sets) and 31 patients (from September 2021 to December 2021) were considered the independent test cohort. There were 68 tubal-ovarian abscesses and 89 OEC, 4 TOA and 10 OEC, and 10 TOA and 21 OEC patients belonging to training and validation sets, internal sets, and independent test sets, respectively. For the model to gain better generalization, we applied the geometric image and color transformations to augment the dataset, including center crop, random rotation, and random horizontal flip. Three convolutional neural networks, namely, ResNet-152, DenseNet-161, and EfficientNet-B7 were applied to differentiate tubal-ovarian abscess from ovarian endometriosis cyst, and their performance was compared with three US physicians and a clinical indicator of carbohydrate antigen 125 (CA125) on the independent test set. The area under the receiver operating characteristic curves (AUROCs) of accuracy, sensitivity, and specificity were used to evaluate the performance. Results: Among the three convolutional neural networks, the performance of ResNet-152 was the highest, with AUROCs of 0.986 (0.954-1). The AUROCs of the three physicians were 0.781 (0.620-0.942), 0.738 (0.629-848), and 0.683 (0.501-0.865), respectively. The clinical indicator CA125 achieved only 0.564 (0.315-0.813). Conclusion: We demonstrated that the CNN model based on the US image could discriminate tubal-ovarian abscess and ovarian endometriosis cyst better than US physicians and CA125. This method can provide a valuable predictive reference for physicians to screen tubal-ovarian abscesses and ovarian endometriosis cysts in time.

WITHDRAWN: UML diagram classification model based on convolution neural network

Nowadays, the amount of data generated daily in various fields is constantly expanding, and the research field of UML diagram is no exception. It has strong expressiveness for complex systems, so it has become the best choice for modeling in many fields. Accordingly, this paper obtains and classifies UML diagram data based on convolution neural network diagram classification algorithm. This paper first analyzes the content and concept of UML, then further describes the importance of building a graphical UML classification model, the complete form convolutional neural network based on image is constructed by analyzing its theory, and then deduces the graphical UML classification model. Finally, a set of UML diagram data set is constructed by using the collected UML diagram design. After data preprocessing, the overall iteration times of the model and the accuracy and average time impact of UML diagram classification model and classification results are obtained. The graph classification algorithm model designed in this study has a good analysis and classification effect for UML object diagrams.Compared with the existing image-based convolutional neural networks and graph neural networks, the proposed UML diagram classification modeling method and GPC-GCN graph neural network have better performance for UML diagram classification. In this study, convolutional neural network technology is applied to the field of UML diagram classification model, thus promoting the development of related technologies in the field of UML diagram classification model.

A novel network architecture combining central-peripheral deviation with image-based convolutional neural networks for diffusion tensor imaging studies

Brain imaging research is a very challenging topic due to complex structure and lack of explicitly identifiable features in the image. With the advancement of magnetic resonance imaging (MRI) technologies, such as diffusion tensor imaging (DTI), developing classification methods to improve clinical diagnosis is crucial. This paper proposes a classification method for DTI data based on a novel neural network strategy that combines a convolutional neural network (CNN) with a multilayer neural network using central-peripheral deviation (CPD), which reflects diffusion dynamics in the white matter by spatially evaluating the deviation of diffusion coefficients between the inner and outer parts of the brain. In our method, a multilayer perceptron (MLP) using CPD is combined with the final layers for classification after reducing the dimensions of images in the convolutional layers of the neural network architecture. In terms of training loss and the classification error, the proposed classification method improves the existing image classification with CNN. For real data analysis, we demonstrate how to process raw DTI image data sets obtained from a traumatic brain injury study (MagNeTS) and a brain atlas construction study (ICBM), and apply the proposed approach to the data, successfully improving classification performance with two age groups.

Laboratory water surface elevation estimation using image-based convolutional neural networks

With the rapid development of artificial intelligence, estimation of water surface elevation from optical imagery using various deep learning approaches becomes popular. In this study, the convolutional neural network (CNN) is applied to estimate the instantaneous water surface elevation based on the laboratory recorded images. Three typical CNN models, i.e., Baseline, VGGNet and ResNet, are considered and evaluated using four performance metrics (RMSE, R2, MAE, and WI). All three models generally show satisfactory estimations for the stable water surface elevation, whereas less satisfying estimations for the unstable water surface elevation are confirmed. It is found that water surface elevation estimation is sensitive to the choice of the CNN network structure, while insensitive to the CNN network depth. Subsequently, influence of four hyperparameters is discussed to optimize the CNN performance, and selection strategies are recommended accordingly. Considering the variation of attention map with respect to the epoch and network depth, logics of CNN's learning behavior and process are further specified, upon which the attention hotspots in CNN algorithm are confirmed as the area where significant changes of the image intensity occur owing to the wave propagation. In addition, image rectification can improve the present estimation accuracy of the water surface elevation.

Detection Method for Classifying Malicious Firmware

A malicious firmware update may prove devastating to the embedded devices both that make up the Internet of Things (IoT) and that typically lack the same security verifications now applied to full operating systems. This work converts the binary headers of 40,000 firmware examples from bytes into 1024-pixel thumbnail images to train a deep neural network. The aim is to distinguish benign and malicious variants using modern deep learning methods without needing detailed functional or forensic analysis tools. One outcome of this image conversion enables contact with the vast machine learning literature already applied to handle digit recognition (MNIST). Another result indicates that greater than 90% accurate classifications prove possible using image-based convolutional neural networks (CNN) when combined with transfer learning methods. The envisioned CNN application would intercept firmware updates before their distribution to IoT networks and score their likelihood of containing malicious variants. To explain how the model makes classification decisions, the research applies traditional statistical methods such as both single and ensembles of decision trees with identifiable pixel or byte values that contribute the malicious or benign determination.

Functional Group Identification for FTIR Spectra Using Image-Based Machine Learning Models.

Fourier transform infrared spectroscopy (FTIR) is a ubiquitous spectroscopic technique. Spectral interpretation is a time-consuming process, but it yields important information about functional groups present in compounds and in complex substances. We develop a generalizable model via a machine learning (ML) algorithm using convolutional neural networks (CNNs) to identify the presence of functional groups in gas-phase FTIR spectra. The ML models reduce the amount of time required to analyze functional groups and facilitate interpretation of FTIR spectra. Through web scraping, we acquire intensity-frequency data from 8728 gas-phase organic molecules within the NIST spectral database and transform the data into spectral images. We successfully train models for 15 of the most common organic functional groups, which we then determine via identification from previously untrained spectra. These models serve to expand the application of FTIR measurements for facile analysis of organic samples. Our approach was done such that we have broad functional group models that infer in tandem to provide full interpretation of a spectrum. We present the first implementation of ML using image-based CNNs for predicting functional groups from a spectroscopic method.

Semi-supervised learning for the study of structural formation in colloidal systems via image recognition

The analysis of the structural formation of colloidal systems using machine learning techniques has recently attracted much attention. In many of these studies, local bond-order parameters (LBOPs) were employed as descriptors, where such LBOPs are suitable mainly for the detection of crystal structures. On the other hand, image-based convolutional neural networks (CNNs) are quite effective in detecting not only crystals but also random structures, and the author demonstrated their efficiency in a previous paper. However, in supervised learning, it is difficult to obtain a correct result when there is an unexpected new phase that was unknown when training the CNN. In this paper, we propose a hybrid scheme that consists of supervised and unsupervised learning techniques, employing two different approaches: image-based CNN and generalized LBOP. The proposed method was applied to two-dimensional colloidal systems, and its efficiency was demonstrated.

Breadth-first piston diagnosing approach for segmented mirrors through supervised learning of multiple-wavelength images.

Piston diagnosing approaches for segmented mirrors via machine-learning have shown great success. However, they are inevitably challenged with 2π ambiguity, and the accuracy is usually influenced by the location and number of submirrors. A piston diagnosing approach for segmented mirrors, which employs the breadth-first search (BFS) algorithm and supervised learning strategies of multi-wavelength images, is investigated. An original kind of object-independent and normalized dataset is generated by the in-focal and defocused images at different wavelengths. Additionally, the segmented mirrors are divided into several sub-models of binary tree and are traversed through the BFS algorithm. Furthermore, two deep image-based convolutional neural networks are constructed for predicting the ranges and values of piston aberrations. Finally, simulations are performed, and the accuracy is independent of the location and number of submirrors. The Pearson correlation coefficients for test sets are above 0.99, and the average root mean square error of segmented mirrors is approximately 0.01λ. This technique allows the piston error between segmented mirrors to be measured without 2π ambiguity. Moreover, it can be used for data collected by a real setup. Furthermore, it can be applied to segmented mirrors with different numbers of submirrors based on the sub-model of a binary tree.

Integrating SIFT and CNN Feature Matching for Partial-Duplicate Image Detection

With the increasing popularity of various deep neural networks in the area of computational intelligence, the research attention for content-based image detection/retrieval has been shifted from the handcrafted local features such as scale invariant feature transform (SIFT) to the features derived from convolutional neural networks (CNN). However, the existing image-based CNN features, directly extracted from the entire images, are not suitable for detecting small duplicate regions, while region-based CNN features show limited robustness to a variety of image modifications such as rescaling, occlusion, and noise adding. These will affect the performance of partial-duplicate image detection. To address these issues, we propose an integrated feature matching scheme, which integrates the matching of SIFT features and CNN features between images for partial-duplicate image detection. In this scheme, we first implement SIFT feature matching based on the bag-of-visual-words model to detect the potential duplicate region pairs between images, and then match the CNN features of these regions extracted from the deep convolutional layer of CNN to compute image similarity. Since both the good robustness of SIFT features and the high discriminative power of CNN features are sufficiently explored, our scheme allows an accurate detection. Experimental results show that the proposed approach provides superior accuracy than the state of the arts, while achieves comparable efficiency.

Compressed-Sensing Magnetic Resonance Image Reconstruction Using an Iterative Convolutional Neural Network Approach

Convolutional neural networks (CNNs) demonstrate excellent performance when employed to reconstruct the images obtained by compressed-sensing magnetic resonance imaging (CS-MRI). Our study aimed to enhance image quality by developing a novel iterative reconstruction approach that utilizes image-based CNNs and k-space correction to preserve original k-space data. In the proposed method, CNNs represent a priori information concerning image spaces. First, the CNNs are trained to map zero-filling images onto corresponding full-sampled images. Then, they recover the zero-filled part of the k-space data. Subsequently, k-space corrections, which involve the replacement of unfilled regions by original k-space data, are implemented to preserve the original k-space data. The above-mentioned processes are used iteratively. The performance of the proposed method was validated using a T2-weighted brain-image dataset, and experiments were conducted with several sampling masks. Finally, the proposed method was compared with other noniterative approaches to demonstrate its effectiveness. The aliasing artifacts in the reconstructed images obtained using the proposed approach were reduced compared to those using other state-of-the-art techniques. In addition, the quantitative results obtained in the form of the peak signal-to-noise ratio and structural similarity index demonstrated the effectiveness of the proposed method. The proposed CS-MRI method enhanced MR image quality with high-throughput examinations.

Learning Facial Expressions with 3D Mesh Convolutional Neural Network

Making machines understand human expressions enables various useful applications in human-machine interaction. In this article, we present a novel facial expression recognition approach with 3D Mesh Convolutional Neural Networks (3DMCNN) and a visual analytics-guided 3DMCNN design and optimization scheme. From an RGBD camera, we first reconstruct a 3D face model of a subject with facial expressions and then compute the geometric properties of the surface. Instead of using regular Convolutional Neural Networks (CNNs) to learn intensities of the facial images, we convolve the geometric properties on the surface of the 3D model using 3DMCNN. We design a geodesic distance-based convolution method to overcome the difficulties raised from the irregular sampling of the face surface mesh. We further present interactive visual analytics for the purpose of designing and modifying the networks to analyze the learned features and cluster similar nodes in 3DMCNN. By removing low-activity nodes in the network, the performance of the network is greatly improved. We compare our method with the regular CNN-based method by interactively visualizing each layer of the networks and analyze the effectiveness of our method by studying representative cases. Testing on public datasets, our method achieves a higher recognition accuracy than traditional image-based CNN and other 3D CNNs. The proposed framework, including 3DMCNN and interactive visual analytics of the CNN, can be extended to other applications.

A picture tells a thousand…exposures: Opportunities and challenges of deep learning image analyses in exposure science and environmental epidemiology

BackgroundArtificial intelligence (AI) is revolutionizing our world, with applications ranging from medicine to engineering. ObjectivesHere we discuss the promise, challenges, and probable data sources needed to apply AI in the fields of exposure science and environmental health. In particular, we focus on the use of deep convolutional neural networks to estimate environmental exposures using images and other complementary data sources such as cell phone mobility and social media information. DiscussionCharacterizing the health impacts of multiple spatially-correlated exposures remains a challenge in environmental epidemiology. A shift toward integrated measures that simultaneously capture multiple aspects of the urban built environment could improve efficiency and provide important insights into how our collective environments influence population health. The widespread adoption of AI in exposure science is on the frontier. This will likely result in new ways of understanding environmental impacts on health and may allow for analyses to be efficiently scaled for broad coverage. Image-based convolutional neural networks may also offer a cost-effective means of estimating local environmental exposures in low and middle-income countries where monitoring and surveillance infrastructure is limited. However, suitable databases must first be assembled to train and evaluate these models and these novel approaches should be complemented with traditional exposure metrics. ConclusionsThe promise of deep learning in environmental health is great and will complement existing measurements for data-rich settings and could enhance the resolution and accuracy of estimates in data poor scenarios. Interdisciplinary partnerships will be needed to fully realize this potential.

Using convolutional neural networks on perceptual and spectral features: Classification experiments on the TREX’13 dataset

Distinguishing objects of interest on the seafloor from clutter remains a key problem facing the automatic target recognition (ATR) community. Because scattering at high frequencies relates more to the geometry of the scatterer, traditional ATR on high frequency imaging sonars is known to suffer in areas of high clutter. Recently, there has been interest in low frequency (1–50 kHz) wideband imaging sonars, because the complex combination of external (geometric) and internal (elastic) scattering that occurs in this frequency range is thought to improve classification in areas of high clutter. This work investigates the classification problem of distinguishing unexploded ordnance (UXO) from clutter using image-based Convolutional Neural Networks on the TREX’13 dataset. This dataset consists of experimental and modelled acoustic backscatter for objects interrogated acoustically at 3–30 kHz across a range of aspects. The model data are used exclusively for training, and the experimental data are used exclusively for testing purposes. The generated feature-set is a combination of acoustic colour and perceptual features, which are derived from modelling how humans perceive timbre. Further in an effort to address the differing amplitude modulations between sets of model and experimental data and improve classified performance, several moving window normalizations will be investigated.

Patch-based Convolutional Neural Network for Whole Slide Tissue Image Classification.

Convolutional Neural Networks (CNN) are state-of-the-art models for many image classification tasks. However, to recognize cancer subtypes automatically, training a CNN on gigapixel resolution Whole Slide Tissue Images (WSI) is currently computationally impossible. The differentiation of cancer subtypes is based on cellular-level visual features observed on image patch scale. Therefore, we argue that in this situation, training a patch-level classifier on image patches will perform better than or similar to an image-level classifier. The challenge becomes how to intelligently combine patch-level classification results and model the fact that not all patches will be discriminative. We propose to train a decision fusion model to aggregate patch-level predictions given by patch-level CNNs, which to the best of our knowledge has not been shown before. Furthermore, we formulate a novel Expectation-Maximization (EM) based method that automatically locates discriminative patches robustly by utilizing the spatial relationships of patches. We apply our method to the classification of glioma and non-small-cell lung carcinoma cases into subtypes. The classification accuracy of our method is similar to the inter-observer agreement between pathologists. Although it is impossible to train CNNs on WSIs, we experimentally demonstrate using a comparable non-cancer dataset of smaller images that a patch-based CNN can outperform an image-based CNN.