Complex Deep Learning Research Articles

On-FPGA Spiking Neural Networks for End-to-End Neural Decoding

In the last decades, deep learning neural decoding algorithms have gained momentum in the field of neural interfaces and neural processing systems. However, to be deployed on low-budget portable devices while maintaining real-time operability, these models must withstand strict computational and power limitations. This work presents a spike decoding system implemented on a low-end Zynq-7010 FPGA, which includes a multiplier-less spike detection pipeline and a spiking-neural-network-based decoder mapped in the programmable logic. We tested the system on two publicly available datasets and achieved comparable results with state-of-the-art neural decoders that use more complex deep learning models. The system required 7.36 times fewer parameters than the smallest architecture tested on the same dataset. Moreover, by exploiting the spike sparsity property of the neural signal, the total amount of computations is reduced by about 90% during a test carried out on real recorded data. The low computational complexity of the chosen spike detection setup, combined with the power efficiency of spiking neural networks, makes this prototype a well-suited choice for low-power real-time neural decoding at the edge.

Spectral pre-processing and non-linear calibration with convolutional kernel partial least-squares. Teaching new tricks to an old dog

A recent trend in multivariate calibration of non-linear systems is to simplify data processing models, avoiding, if possible, some complex deep learning approaches. Contributing to this line of work, convolutional kernel partial least-squares (CKPLS) is introduced both for finding the best spectral pre-processing procedure for reducing the impact of radiation scattering and for handling non-linearities in the data. CKPLS is a combination of a previous convolutional step for pre-processing the spectra with the well-known kernel PLS regression model for coping with non-linear relationships between spectral signatures and analyte concentrations or target sample properties. The convolutional step is driven by particle swarm optimization (PSO), which estimates the coefficients of a moving window spectral pre-processing. This convolutional phase, previous to KPLS, is a viable alternative to the few available methods for finding the best mathematical pre-processing of the spectra, which is usually performed by trial and error. Analytical results concerning the calibration of selected analytes from partially selective spectra are employed to illustrate the performance of CKPLS. For this purpose, both simulated and experimental data sets have been employed, showing that automatic pre-processing of spectra is possible, with a success which is comparable to classical methods such as computing the spectral derivatives.

Exploring global attention mechanism on fault detection and diagnosis for complex engineering processes

Considering about slow drift and complicated relationships among process variables caused by corrosion, fatigue, and so on in complex chemical engineering processes, an Industrial Process Optimization ViT (IPO-ViT) method was proposed to explore the global receptive field provided by self-attention mechanism of Vision Transformer (ViT) on fault detection and diagnosis (FDD). The applications on data sampled from both a real industrial process and the Tennessee Eastman (TE) process showed superior performance of the global attention-based method (IPO-ViT) over other typical local receptive fields deep learning methods without increasing sample and computation requirements. Moreover, results on six different variants in combing local, shallow filtering and global receptive field mechanisms unravel that the local attention explosion, the information alignment, and the expression capability are three major challenges for further improving on industrial applications of complex deep learning network structures.

Do not sleep on traditional machine learning: Simple and interpretable techniques are competitive to deep learning for sleep scoring

Over the last few years, research in automatic sleep scoring has mainly focused on developing increasingly complex deep learning architectures. However, recently these approaches achieved only marginal improvements, often at the expense of requiring more data and more expensive training procedures. Despite all these efforts and their satisfactory performance, automatic sleep staging solutions are not widely adopted in a clinical context yet. We argue that most deep learning solutions for sleep scoring are limited in their real-world applicability as they are hard to train, deploy, and reproduce. Moreover, these solutions lack interpretability and transparency, which are often key to increase adoption rates. In this work, we revisit the problem of sleep stage classification using classical machine learning. Results show that competitive performance can be achieved with a conventional machine learning pipeline consisting of preprocessing, feature extraction, and a simple machine learning model. In particular, we analyze the performance of a linear model and a non-linear (gradient boosting) model. Our approach surpasses state-of-the-art (that uses the same data) on two public datasets: Sleep-EDF SC-20 (MF1 0.810) and Sleep-EDF ST (MF1 0.795), while achieving competitive results on Sleep-EDF SC-78 (MF1 0.775) and MASS SS3 (MF1 0.817). We show that, for the sleep stage scoring task, the expressiveness of an engineered feature vector is on par with the internally learned representations of deep learning models. This observation opens the door to clinical adoption, as a representative feature vector allows to leverage both the interpretability and successful track record of traditional machine learning models.

Exploring Lightweight Deep Learning Solution for Malware Detection in IoT Constraint Environment

The present era is facing the industrial revolution. Machine-to-Machine (M2M) communication paradigm is becoming prevalent. Resultantly, the computational capabilities are being embedded in everyday objects called things. When connected to the internet, these things create an Internet of Things (IoT). However, the things are resource-constrained devices that have limited computational power. The connectivity of the things with the internet raises the challenges of the security. The user sensitive information processed by the things is also susceptible to the trusability issues. Therefore, the proliferation of cybersecurity risks and malware threat increases the need for enhanced security integration. This demands augmenting the things with state-of-the-art deep learning models for enhanced detection and protection of the user data. Existingly, the deep learning solutions are overly complex, and often overfitted for the given problem. In this research, our primary objective is to investigate a lightweight deep-learning approach maximizes the accuracy scores with lower computational costs to ensure the applicability of real-time malware monitoring in constrained IoT devices. We used state-of-the-art Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), and Bi-directional LSTM deep learning algorithm on a vanilla configuration trained on a standard malware dataset. The results of the proposed approach show that the simple deep neural models having single dense layer and a few hundred trainable parameters can eliminate the model overfitting and achieve up to 99.45% accuracy, outperforming the overly complex deep learning models.

Vision-Based Semantic Segmentation in Scene Understanding for Autonomous Driving: Recent Achievements, Challenges, and Outlooks

Scene understanding plays a crucial role in autonomous driving by utilizing sensory data for contextual information extraction and decision making. Beyond modeling advances, the enabler for vehicles to become aware of their surroundings is the availability of visual sensory data, which expand the vehicular perception and realizes vehicular contextual awareness in real-world environments. Research directions for scene understanding pursued by related studies include person/vehicle detection and segmentation, their transition analysis, lane change, and turns detection, among many others. Unfortunately, these tasks seem insufficient to completely develop fully-autonomous vehicles i.e., achieving level-5 autonomy, travelling just like human-controlled cars. This latter statement is among the conclusions drawn from this review paper: scene understanding for autonomous driving cars using vision sensors still requires significant improvements. With this motivation, this survey defines, analyzes, and reviews the current achievements of the scene understanding research area that mostly rely on computationally complex deep learning models. Furthermore, it covers the generic scene understanding pipeline, investigates the performance reported by the state-of-the-art, informs about the time complexity analysis of avant garde modeling choices, and highlights major triumphs and noted limitations encountered by current research efforts. The survey also includes a comprehensive discussion on the available datasets, and the challenges that, even if lately confronted by researchers, still remain open to date. Finally, our work outlines future research directions to welcome researchers and practitioners to this exciting domain.

Exposing the Limitations of Molecular Machine Learning with Activity Cliffs.

Machine learning has become a crucial tool in drug discovery and chemistry at large, e.g., to predict molecular properties, such as bioactivity, with high accuracy. However, activity cliffs─pairs of molecules that are highly similar in their structure but exhibit large differences in potency─have received limited attention for their effect on model performance. Not only are these edge cases informative for molecule discovery and optimization but also models that are well equipped to accurately predict the potency of activity cliffs have increased potential for prospective applications. Our work aims to fill the current knowledge gap on best-practice machine learning methods in the presence of activity cliffs. We benchmarked a total of 24 machine and deep learning approaches on curated bioactivity data from 30 macromolecular targets for their performance on activity cliff compounds. While all methods struggled in the presence of activity cliffs, machine learning approaches based on molecular descriptors outperformed more complex deep learning methods. Our findings highlight large case-by-case differences in performance, advocating for (a) the inclusion of dedicated "activity-cliff-centered" metrics during model development and evaluation and (b) the development of novel algorithms to better predict the properties of activity cliffs. To this end, the methods, metrics, and results of this study have been encapsulated into an open-access benchmarking platform named MoleculeACE (Activity Cliff Estimation, available on GitHub at: https://github.com/molML/MoleculeACE). MoleculeACE is designed to steer the community toward addressing the pressing but overlooked limitation of molecular machine learning models posed by activity cliffs.

Comparative analysis of deep convolution neural network models on small scale datasets

Deep Learning is a representational model that proves significant impact in computer vision applications by automating the feature extraction process with the advancement and availability of high computing power. Conventional machine learning algorithm works efficiently on large scale data and trained for specific applications that cannot be used for other task. Deep neural networks require large amounts of data, resources, time and high computing devices. Complex deep learning models for a new application domain could be developed using transfer learning with less resources and few labeled training data. An important challenge is negative transfer, which means the initial and target problems are similar. Also, there are no research findings on choosing the right pre-trained model which are closely related to the target dataset and there is no proper measure to analyze the performance. Convolutional Neural Networks models impart significant advancement to the ImageNet visual recognition challenge in the last decade. This work focuses on the use of pre-trained models for small datasets. It analyzes the initialization of parameters, tuning of hyperparameters, parameter optimization, regularization methods, and computation complexity of pre-trained convolutional neural network models. Experiments are performed on three different data sets COREL1k, COIL20 and Garbage images. It shows that the time to re-train the existing pre-trained models was greatly reduced by transfer learning. Transferring weights, instances and layers from models trained in ImageNet dataset could produce better performance in similar small scale datasets, but the performance of dataset which differs in ImageNet dataset was not satisfactory.

An Effective Method for Lung Cancer Diagnosis from CT Scan Using Deep Learning-Based Support Vector Network.

The diagnosis of early-stage lung cancer is challenging due to its asymptomatic nature, especially given the repeated radiation exposure and high cost of computed tomography(CT). Examining the lung CT images to detect pulmonary nodules, especially the cell lung cancer lesions, is also tedious and prone to errors even by a specialist. This study proposes a cancer diagnostic model based on a deep learning-enabled support vector machine (SVM). The proposed computer-aided design (CAD) model identifies the physiological and pathological changes in the soft tissues of the cross-section in lung cancer lesions. The model is first trained to recognize lung cancer by measuring and comparing the selected profile values in CT images obtained from patients and control patients at their diagnosis. Then, the model is tested and validated using the CT scans of both patients and control patients that are not shown in the training phase. The study investigates 888 annotated CT scans from the publicly available LIDC/IDRI database. The proposed deep learning-assisted SVM-based model yields 94% accuracy for pulmonary nodule detection representing early-stage lung cancer. It is found superior to other existing methods including complex deep learning, simple machine learning, and the hybrid techniques used on lung CT images for nodule detection. Experimental results demonstrate that the proposed approach can greatly assist radiologists in detecting early lung cancer and facilitating the timely management of patients.

A descriptive human visual cognitive strategy using graph neural network for facial expression recognition

In the period of rapid development on the new information technologies, computer vision has become the most common application of artificial intelligence, which is represented by deep learning in the current society. As the most direct and effective application of computer vision, facial expression recognition (FER) has become a hot topic and used in many studies and domains. However, the existing FER methods focus on deep learning to generate increasingly complex attention structures, so they are unable to consider the connotative relationship between different parts of facial expressions. Moreover, the human expression recognition method based on complex deep learning network has serious interpretability issues. Therefore, in this paper, a novel Graph Neural Network (GNN) model is proposed to consider the systematic process of FER in human visual perception. Firstly, a region division mechanism is proposed, which divides the face region into six parts to unify the selection of key facial features. On this basis, in order to better consider the connotative relationship between different parts of facial expression, a human visual cognition strategy is proposed, which uses the divided six regions to learn facial expression features, and evenly selects the key features with high reliability as graph nodes. In combination with the human regional cooperative recognition process, the connotative relationship (such as relative position and similar structure) between graph nodes is extracted, so as to construct the GNN model. Finally, the effect of FER is obtained by the modeled GNN model. The experimental results compared with other related algorithms show that the model not only has stronger characterization and generalization ability, but also has better robustness compared with state-of-the-art methods.

Multifunctional 2D MoS2 Optoelectronic Artificial Synapse with Integrated Arithmetic and Reconfigurable Logic Operations for In‐Memory Neuromorphic Computing Applications

AbstractThe hardware implementation of advanced artificial intelligence (AI) technology based on complex deep learning and machine learning algorithms is constricted by the limitation of conventional Von–Neuman architecture. Emerging neuromorphic computing architecture based on the human brain with in‐memory computing capability could instigate unprecedented breakthroughs in AI technology. In this pursuit, 2D MoS2 optoelectronic artificial synapse imitating complex biological neuromorphic behavior such as short/long‐term memory, paired‐pulse facilitation, and long‐term depression‐potentiation is proposed and demonstrated. Furthermore, the broadband sensitivity of the device can be utilized to emulate Pavlov's classical conditioning for associative learning of the biological brain. More importantly, reconfigurable Boolean AND and OR logic gate operation is demonstrated within the same device by synergistically modulating the device conductance via the persistent photoconductivity and electrical gate stress. The linear response of the photocurrent to the optical stimulus can perform arithmetic operations such as counting, addition, and subtraction within a single device. This novel integration of memory, synaptic behavior, and processing within a single monolayer MoS2 device is believed to put forth a new horizon for the Non‐Von–Neuman type in‐memory computing architecture for highly advanced AI applications based on 2D materials.

Pathologic Image Classification of Flat Urothelial Lesions Using Pathologic Criteria-Based Deep Learning

Pathologic diagnosis of flat urothelial lesions is subject to high interobserver variability. We expected that deep learning could improve the accuracy and consistency of such pathologic diagnosis, although the learning process is a black box. We therefore propose a new approach for pathologic image classification incorporating the diagnostic process of the pathologist into a deep learning method. A total of 267 H&E-stained slides of normal urothelium and urothelial lesions from 127 cases were examined. Six independent convolutional neural networks were trained to classify pathologic images according to six pathologic criteria. We then used these networks in the main training for the final diagnosis. Compared with conventional manual analysis, our method significantly improved the classification accuracy of images of flat urothelial lesions. The automated classification showed almost perfect agreement (weighted κ = 0.98) with the consensus reading. In addition, our approach provides the advantages of reliable diagnosis corresponding to histologic interpretation. We used deep learning to establish an automated subtype classifier for flat urothelial lesions that successfully combines traditional morphologic approaches and complex deep learning to achieve a learning mechanism that seems plausible to the pathologist.

Obtaining genetics insights from deep learning via explainable artificial intelligence.

Artificial intelligence (AI) models based on deep learning now represent the state of the art for making functional predictions in genomics research. However, the underlying basis on which predictive models make such predictions is often unknown. For genomics researchers, this missing explanatory information would frequently be of greater value than the predictions themselves, as it can enable new insights into genetic processes. We review progress in the emerging area of explainable AI (xAI), a field with the potential to empower life science researchers to gain mechanistic insights into complex deep learning models. We discuss and categorize approaches for model interpretation, including an intuitive understanding of how each approach works and their underlying assumptions and limitations in the context of typical high-throughput biological datasets.

Advances in Cuffless Continuous Blood Pressure Monitoring Technology Based on PPG Signals.

Objective To review the progress of research on photoplethysmography- (PPG-) based cuffless continuous blood pressure monitoring technologies and prospect the challenges that need to be addressed in the future. Methods Using Web of Science and PubMed as search engines, the literature on cuffless continuous blood pressure studies using PPG signals in the recent five years were searched. Results Based on the retrieved literature, this paper describes the available open datasets, commonly used signal preprocessing methods, and model evaluation criteria. Early researches employed multisite PPG signals to calculate pulse wave velocity or time and predicted blood pressure by a simple linear equation. Later, extensive researches were dedicated to mine the features of PPG signals related to blood pressure and regressed blood pressure by machine learning models. Most recently, many researches have emerged to experiment with complex deep learning models for blood pressure prediction with the raw PPG signal as input. Conclusion This paper summarized the methods in the retrieved literature, provided insight into the artificial intelligence algorithms employed in the literature, and concluded with a discussion of the challenges and opportunities for the development of cuffless continuous blood pressure monitoring technologies.

Spatial and Temporal Normalization for Multi-Variate Time Series Prediction Using Machine Learning Algorithms

Multi-variable time series (MTS) information is a typical type of data inference in the real world. Every instance of MTS is produced via a hybrid dynamical scheme, the dynamics of which are often unknown. The hybrid species of this dynamical service are the outcome of high-frequency and low-frequency external impacts, as well as global and local spatial impacts. These influences impact MTS’s future growth; hence, they must be incorporated into time series forecasts. Two types of normalization modules, temporal and spatial normalization, are recommended to accomplish this. Each boosts the original data’s local and high-frequency processes distinctly. In addition, all components are easily incorporated into well-known deep learning techniques, such as Wavenet and Transformer. However, existing methodologies have inherent limitations when it comes to isolating the variables produced by each sort of influence from the real data. Consequently, the study encompasses conventional neural networks, such as the multi-layer perceptron (MLP), complex deep learning methods such as LSTM, two recurrent neural networks, support vector machines (SVM), and their application for regression, XGBoost, and others. Extensive experimental work on three datasets shows that the effectiveness of canonical frameworks could be greatly improved by adding more normalization components to how the MTS is used. This would make it as effective as the best MTS designs are currently available. Recurrent models, such as LSTM and RNN, attempt to recognize the temporal variability in the data; however, as a result, their effectiveness might soon decline. Last but not least, it is claimed that training a temporal framework that utilizes recurrence-based methods such as RNN and LSTM approaches is challenging and expensive, while the MLP network structure outperformed other models in terms of time series predictive performance.

Adaptively Equalized Bandwidth Optimization Model using SCADA-DWWAN based Neural Network

Artificial training and learning algorithms, enhanced with semi-supervised or self-supervised feature extraction capacities, employ adaptive decision optimization models. These are often favored over complex deep learning algorithms for achieving better controllability and ease of observation, lower complexity in simulating, building or designing and virtual prototyping of automatic network resource management (ANRM) standards. An Adaptive Linear Neuron type Artificial Neural Network (ADALINE-ANN) which is based on multi-tapered machine learning approach has been simulated in a virtual Supervisory Control and Data Acquisition (SCADA) framework integrated with a Distributed Control System (SCADA/DCS-Net). The system has been virtually simulated considering an adaptively equalized learning and decision approach which utilizes Markov Trained-Steepest Gradient Descent (HMM-SGD) based machine learning model employing Kalman optimization. Affinity clustering is employed for spectrum sensing by extracting the Constellation Nyquist Bands from an M-Quadrature Amplitude Modulation (QAM) orthogonal signal undergoing AWGN and Rayleigh fading as well as co-channel interference (CCI), and ensemble analysis using Channel State-Space plots are used for optimal spectrum allocation in an Adaptive Orthogonal Frequency Division Multiple Access (Adaptive-OFDMA) layout. It has been done by implementing an adaptively equalized Automatic Repeat Request (ARQ) pipelining model which utilizes minimum least square error (MLSE) minimization model. The objective is to improve the bandwidth allocation and usage ensuring most minimum spectrum wastage or loss. Successive Interference Cancellation (SIC) has been implemented to minimize static buffer and interference loss. Thus, spectrum loss due to latency and jitter which occurs from bandwidth congestion is minimized by improving the network resource tracking and allocation. It results in improved and stable bandwidth equalization.

Bayesian Active Learning for Scanning Probe Microscopy: From Gaussian Processes to Hypothesis Learning.

Recent progress in machine learning methods and the emerging availability of programmable interfaces for scanning probe microscopes (SPMs) have propelled automated and autonomous microscopies to the forefront of attention of the scientific community. However, enabling automated microscopy requires the development of task-specific machine learning methods, understanding the interplay between physics discovery and machine learning, and fully defined discovery workflows. This, in turn, requires balancing the physical intuition and prior knowledge of the domain scientist with rewards that define experimental goals and machine learning algorithms that can translate these to specific experimental protocols. Here, we discuss the basic principles of Bayesian active learning and illustrate its applications for SPM. We progress from the Gaussian process as a simple data-driven method and Bayesian inference for physical models as an extension of physics-based functional fits to more complex deep kernel learning methods, structured Gaussian processes, and hypothesis learning. These frameworks allow for the use of prior data, the discovery of specific functionalities as encoded in spectral data, and exploration of physical laws manifesting during the experiment. The discussed framework can be universally applied to all techniques combining imaging and spectroscopy, SPM methods, nanoindentation, electron microscopy and spectroscopy, and chemical imaging methods and can be particularly impactful for destructive or irreversible measurements.

Application of deep-learning based techniques for automatic metrology on scanning and transmission electron microscopy images

Scanning or transmission electron microscopy (SEM/TEM) are standard techniques used during Research and Development (R&D) phases to study the structure and morphology of microscopic materials. Variety in object shapes and sizes are observed in such images to ensure robust micro- and nanomaterials critical dimension analysis. This way, precision and accuracy can be guaranteed during materials manufacturing processes. Such diversity and complexity in the data make it challenging to automatically extract the desired measurements of these microscopic structures. Existing tools in metrology often require many manual interactions, therefore being slow and prone to user errors. Proposed semiautomatic and automatic tools in the state-of-the-art are also limited and not designed to handle large variations across the images. Thus, the application of advanced machine or deep learning techniques could bring great efficiency in SEM/TEM image analysis and measurements for microscopic scale R&D processes. In this paper, we demonstrate the feasibility of deep-learning based object detection and instance segmentation models to perform automatic and accurate metrology on microscopic images with high object variability. We also show that custom object detection models prepared using pretrained weights, finetuned on very limited custom data, can outperform detection models built using traditional methods. This is particularly useful in metrology for the semiconductor industry, where data scarcity is common. When the data are available, we observe that it can be useful to be able to generate a large number of quality annotations to use instance segmentation. This could allow the training of more complex deep learning models for particle recognition and analysis. Therefore, we propose a semiautomatic tool to help produce annotations and demonstrate its application in an instance segmentation task.

Consistency of Feature Importance Algorithms for Interpretable EEG Abnormality Detection.

Recent advances in machine learning show great potential for automatic detection of abnormalities in electroencephalography (EEG). While simple and interpretable models combined with expert-comprehensible input features offer full control of the decision making process, these methods commonly lag behind complex deep learning and feature extraction methods in terms of performance. Here we study a feasibility of a bridging solution, where deep learning is combined with interpretable input and an algorithm computing the importance of particular EEG features in the decision process. We built a convolutional neural network with multi-channel EEG frequency bands as input and investigated four different methods for feature importance attribution: Layer-wise Relevance Propagation (LRP), DeepLIFT, Integrated Gradients (IG) and Guided GradCAM. Our analysis showed consistency between the first three methods, and deviating attributions of the fourth method, suggesting the importance of using a package of methods together to ensure the robustness of medical interpretation.

Leveraging a Heterogeneous Ensemble Learning for Outcome-Based Predictive Monitoring Using Business Process Event Logs

Outcome-based predictive process monitoring concerns predicting the outcome of a running process case using historical events stored as so-called process event logs. This prediction problem has been approached using different learning models in the literature. Ensemble learners have been shown to be particularly effective in outcome-based business process predictive monitoring, even when compared with learners exploiting complex deep learning architectures. However, the ensemble learners that have been used in the literature rely on weak base learners, such as decision trees. In this article, an advanced stacking ensemble technique for outcome-based predictive monitoring is introduced. The proposed stacking ensemble employs strong learners as base classifiers, i.e., other ensembles. More specifically, we consider stacking of random forests, extreme gradient boosting machines, and gradient boosting machines to train a process outcome prediction model. We evaluate the proposed approach using publicly available event logs. The results show that the proposed model is a promising approach for the outcome-based prediction task. We extensively compare the performance differences among the proposed methods and the base strong learners, using also statistical tests to prove the generalizability of the results obtained.