Layers Of Deep Neural Network Research Articles

Towards the Importance of the Type of Deep Neural Network and Employment of Pre-trained Word Vectors for Toxicity Detection: An Experimental Study

As a natural consequence of offering many advantages to their users, social media platforms have become a part of daily lives. Recent studies emphasize the necessity of an automated way of detecting the offensive posts in social media since these ‘toxic’ posts have become pervasive. To this end, a novel toxic post detection approach based on Deep Neural Networks was proposed within this study. Given that several word embedding methods exist, we shed light on which word embedding method produces better results when employed with the five most common types of deep neural networks, namely, , , , , and a combination of and . To this end, the word vectors for the given comments were obtained through four different methods, namely, () , () , () , and () the layer of deep neural networks. Eventually, a total of twenty benchmark models were proposed and both trained and evaluated on a gold standard dataset which consists of tweets. According to the experimental result, the best , , was obtained on the proposed model without employing pre-trained word vectors which outperformed the state-of-the-art works and implies the effective embedding ability of s. Other key findings obtained through the conducted experiments are that the models, that constructed word embeddings through the layers, obtained higher s and converged much faster than the models that utilized pre-trained word vectors.

Towards Semantically-Rich Spatial Network Representation Learning via Automated Feature Topic Pairing.

Automated characterization of spatial data is a kind of critical geographical intelligence. As an emerging technique for characterization, spatial Representation Learning (SRL) uses deep neural networks (DNNs) to learn non-linear embedded features of spatial data for characterization. However, SRL extracts features by internal layers of DNNs, and thus suffers from lacking semantic labels. Texts of spatial entities, on the other hand, provide semantic understanding of latent feature labels, but is insensible to deep SRL models. How can we teach a SRL model to discover appropriate topic labels in texts and pair learned features with the labels? This paper formulates a new problem: feature-topic pairing, and proposes a novel Particle Swarm Optimization (PSO) based deep learning framework. Specifically, we formulate the feature-topic pairing problem into an automated alignment task between 1) a latent embedding feature space and 2) a textual semantic topic space. We decompose the alignment of the two spaces into: 1) point-wise alignment, denoting the correlation between a topic distribution and an embedding vector; 2) pair-wise alignment, denoting the consistency between a feature-feature similarity matrix and a topic-topic similarity matrix. We design a PSO based solver to simultaneously select an optimal set of topics and learn corresponding features based on the selected topics. We develop a closed loop algorithm to iterate between 1) minimizing losses of representation reconstruction and feature-topic alignment and 2) searching the best topics. Finally, we present extensive experiments to demonstrate the enhanced performance of our method.

Equality of opportunity in travel behavior prediction with deep neural networks and discrete choice models

Although researchers increasingly adopt machine learning to model travel behavior, they predominantly focus on prediction accuracy, ignoring the ethical challenges embedded in machine learning algorithms. This study introduces an important missing dimension - computational fairness - to travel behavior analysis. It highlights the accuracy-fairness tradeoff instead of the single dimensional focus on prediction accuracy in the contexts of deep neural network (DNN) and discrete choice models (DCM). We first operationalize computational fairness by equality of opportunity, then differentiate between the bias inherent in data and the bias introduced by modeling. The models inheriting the inherent biases can risk perpetuating the existing inequality in the data structure, and the biases in modeling can further exacerbate it. We then demonstrate the prediction disparities in travel behavior modeling using the 2017 National Household Travel Survey (NHTS) and the 2018–2019 My Daily Travel Survey in Chicago. Empirically, DNN and DCM reveal consistent prediction disparities across multiple social groups: both over-predict the false negative rate of frequent driving for the ethnic minorities, the low-income and the disabled populations, and falsely predict a higher travel burden of the socially disadvantaged groups and the rural populations than reality. Comparing DNN with DCM, we find that DNN can outperform DCM in prediction disparities because of DNN’s smaller misspecification error. To mitigate prediction disparities, this study introduces an absolute correlation regularization method, which is evaluated with synthetic and real-world data. The results demonstrate the prevalence of prediction disparities in travel behavior modeling, and the disparities still persist regarding a variety of model specifics such as the number of DNN layers, batch size and weight initialization. Since these prediction disparities can exacerbate social inequity if prediction results without fairness adjustment are used for transportation policy making, we advocate for careful consideration of the fairness problem in travel behavior modeling, and the use of bias mitigation algorithms for fair transport decisions.

A Signature of Orientation Invariance in Human fMRI & (Some) Deep Neural Networks

Deep neural network (DNN) models capture the representational geometry of neural responses to different objects (Yamins & Dicarlo, 2016), but do they show the same hallmark invariances as human vision? In the human visual system, orientation invariant representations (consistent responses to an item viewed at different angles) emerge abruptly between V3 and LOC (Morgan & Alvarez, 2014). Here, we ask where (if anywhere) orientation invariance emerges in DNNs, and whether that invariance mirrors the invariance we observe in humans. Using a large battery of DNNs (both trained and randomly initialized, from model zoos including Torchvision and the Taskonomy project), we extract the responses of each layer of each network to a series of 8 stimuli at 5 different degrees of rotation (0-45-90-135-180) – the same stimulus set used in the human fMRI. We then assess the representational similarity of each stimulus to the same stimulus at different rotations (“within-category” similarity) and to different stimuli at the same rotation (“across-category” similarity) in each layer of the network and each area of the brain. We compare the DNN responses directly to the fMRI responses with a second Pearson distance metric across the aggregated similarities (within- and across-category). In the end, this process produces a similarity score for each DNN layer to each area of the brain. Smoothing across the layer-wise similarity scores in each network with a generalized additive model (GAM), we show that in the majority of trained models we test, earlier and intermediate layers of the network are more similar to brain areas with little to no invariance (EVC / OPC) and later layers (usually following a fully connected layer) are more similar to brain areas with strong invariance (LOC / OTC). Overall, our results provide a preliminary signature of human brain-like orientation invariance in deep neural networks.

THINGSvision: A Python Toolbox for Streamlining the Extraction of Activations From Deep Neural Networks.

Over the past decade, deep neural network (DNN) models have received a lot of attention due to their near-human object classification performance and their excellent prediction of signals recorded from biological visual systems. To better understand the function of these networks and relate them to hypotheses about brain activity and behavior, researchers need to extract the activations to images across different DNN layers. The abundance of different DNN variants, however, can often be unwieldy, and the task of extracting DNN activations from different layers may be non-trivial and error-prone for someone without a strong computational background. Thus, researchers in the fields of cognitive science and computational neuroscience would benefit from a library or package that supports a user in the extraction task. THINGSvision is a new Python module that aims at closing this gap by providing a simple and unified tool for extracting layer activations for a wide range of pretrained and randomly-initialized neural network architectures, even for users with little to no programming experience. We demonstrate the general utility of THINGsvision by relating extracted DNN activations to a number of functional MRI and behavioral datasets using representational similarity analysis, which can be performed as an integral part of the toolbox. Together, THINGSvision enables researchers across diverse fields to extract features in a streamlined manner for their custom image dataset, thereby improving the ease of relating DNNs, brain activity, and behavior, and improving the reproducibility of findings in these research fields.

Exploring deep features and ECG attributes to detect cardiac rhythm classes

Arrhythmia is a condition characterized by perturbation of the regular rhythm of the heart. The development of computerized self-diagnostic systems for the detection of these arrhythmias is very popular, thanks to the machine learning models included in these systems, which eliminate the need for visual inspection of long electrocardiogram (ECG) recordings. In order to design a reliable, generalizable and highly accurate model, large number of subjects and arrhythmia classes are included in the training and testing phases of the model. In this study, an ECG dataset containing more than 10,000 subject records was used to train and diagnose arrhythmia. A deep neural network (DNN) model was used on the data set during the extraction of the features of the ECG inputs. Feature maps obtained from hierarchically placed layers in DNN were fed to various shallow classifiers. Principal component analysis (PCA) technique was used to reduce the high dimensions of feature maps. In addition to the morphological features obtained with DNN, various ECG features obtained from lead-II for rhythmic information are fused to increase the performance. Using the ECG features, an accuracy of 90.30% has been achieved. Using only deep features, this accuracy was increased to 97.26%. However, the accuracy was increased to 98.00% by fusing both deep and ECG-based features. Another important research subject of the study is the examination of the features obtained from DNN network both on a layer basis and at each training step. The findings show that the more abstract features obtained from the last layers of the DNN network provide high performance in shallow classifiers, and weight updates of DNN network also increases the performance of these classifiers. Hence, the study presents important findings on the fusion of deep features and shallow classifiers to improve the performance of the proposed system.

A Novel Automate Python Edge-to-Edge: From Automated Generation on Cloud to User Application Deployment on Edge of Deep Neural Networks for Low Power IoT Systems FPGA-Based Acceleration.

Deep Neural Networks (DNNs) deployment for IoT Edge applications requires strong skills in hardware and software. In this paper, a novel design framework fully automated for Edge applications is proposed to perform such a deployment on System-on-Chips. Based on a high-level Python interface that mimics the leading Deep Learning software frameworks, it offers an easy way to implement a hardware-accelerated DNN on an FPGA. To do this, our design methodology covers the three main phases: (a) customization: where the user specifies the optimizations needed on each DNN layer, (b) generation: the framework generates on the Cloud the necessary binaries for both FPGA and software parts, and (c) deployment: the SoC on the Edge receives the resulting files serving to program the FPGA and related Python libraries for user applications. Among the study cases, an optimized DNN for the MNIST database can speed up more than 60× a software version on the ZYNQ 7020 SoC and still consume less than . A comparison with the state-of-the-art frameworks demonstrates that our methodology offers the best trade-off between throughput, power consumption, and system cost.

In-circuit tuning of deep learning designs

This paper presents OTune, a novel overlay-based approach for rapid in-circuit debugging and tuning of Deep Neural Network (DNN) designs targeting Field-Programmable Gate Array (FPGA). We first propose overlay-based instruments that provide hardware profiling information to FPGA-based DNN developers for tuning and debugging their designs. Our instrumentation is optimized to take advantage of characteristics of the DNN application domain and traces useful information for in-circuit domain-specific development. Besides, a light-weight overlay-based DNN processing engine is implemented to support rapid word length tuning, which allows adjusting each DNN layer’s datapath without time-consuming FPGA compilation. Furthermore, our approach enables tuning of FPGA-based DNN designs for edge systems, which would benefit developing adaptive learning systems. Evaluation results show that OTune can tune a fixed-point design to the same accuracy as a floating-point one with less than 4% added FPGA area.

Brain hierarchy score: Which deep neural networks are hierarchically brain-like?

SummaryAchievement of human-level image recognition by deep neural networks (DNNs) has spurred interest in whether and how DNNs are brain-like. Both DNNs and the visual cortex perform hierarchical processing, and correspondence has been shown between hierarchical visual areas and DNN layers in representing visual features. Here, we propose the brain hierarchy (BH) score as a metric to quantify the degree of hierarchical correspondence based on neural decoding and encoding analyses where DNN unit activations and human brain activity are predicted from each other. We find that BH scores for 29 pre-trained DNNs with various architectures are negatively correlated with image recognition performance, thus indicating that recently developed high-performance DNNs are not necessarily brain-like. Experimental manipulations of DNN models suggest that single-path sequential feedforward architecture with broad spatial integration is critical to brain-like hierarchy. Our method may provide new ways to design DNNs in light of their representational homology to the brain.

DORY: Automatic End-to-End Deployment of Real-World DNNs on Low-Cost IoT MCUs

The deployment of Deep Neural Networks (DNNs) on end-nodes at the extreme edge of the Internet-of-Things is a critical enabler to support pervasive Deep Learning-enhanced applications. Low-Cost MCU-based end-nodes have limited on-chip memory and often replace caches with scratchpads, to reduce area overheads and increase energy efficiency -- requiring explicit DMA-based memory transfers between different levels of the memory hierarchy. Mapping modern DNNs on these systems requires aggressive topology-dependent tiling and double-buffering. In this work, we propose DORY (Deployment Oriented to memoRY) - an automatic tool to deploy DNNs on low cost MCUs with typically less than 1MB of on-chip SRAM memory. DORY abstracts tiling as a Constraint Programming (CP) problem: it maximizes L1 memory utilization under the topological constraints imposed by each DNN layer. Then, it generates ANSI C code to orchestrate off- and on-chip transfers and computation phases. Furthermore, to maximize speed, DORY augments the CP formulation with heuristics promoting performance-effective tile sizes. As a case study for DORY, we target GreenWaves Technologies GAP8, one of the most advanced parallel ultra-low power MCU-class devices on the market. On this device, DORY achieves up to 2.5x better MAC/cycle than the GreenWaves proprietary software solution and 18.1x better than the state-of-the-art result on an STM32-F746 MCU on single layers. Using our tool, GAP-8 can perform end-to-end inference of a 1.0-MobileNet-128 network consuming just 63 pJ/MAC on average @ 4.3 fps - 15.4x better than an STM32-F746. We release all our developments - the DORY framework, the optimized backend kernels, and the related heuristics - as open-source software.

Deep fair models for complex data: Graphs labeling and explainable face recognition

The central goal of Algorithmic Fairness is to develop AI-based systems which do not discriminate subgroups in the population with respect to one or multiple notions of inequity, knowing that data is often humanly biased. Researchers are racing to develop AI-based systems able to reach superior performance in terms of accuracy, increasing the risk of inheriting the human biases hidden in the data. An obvious tension exists between these two lines of research that are currently colliding due to increasing concerns regarding the widespread adoption of these systems and their ethical impact. The problem is even more challenging when the input data is complex (e.g. graphs, trees, or images) and deep uninterpretable models need to be employed to achieve satisfactory performance. In fact, it is required to develop a deep architecture to learn a data representation able, from one side, to be expressive enough to describe the data and lead to highly accurate models and, from the other side, to discard all the information which may lead to unfair behavior. In this work we measure fairness according to Demographic Parity, requiring the probability of the model decisions to be independent of the sensitive information. We investigate how to impose this constraint in the different layers of deep neural networks for complex data, with particular reference to deep networks for graph and face recognition. We present experiments on different real-world datasets, showing the effectiveness of our proposal both quantitatively by means of accuracy and fairness metrics and qualitatively by means of visual explanation.

Memristor crossbar architectures for implementing deep neural networks

The paper presents memristor crossbar architectures for implementing layers in deep neural networks, including the fully connected layer, the convolutional layer, and the pooling layer. The crossbars achieve positive and negative weight values and approximately realize various nonlinear activation functions. Then the layers constructed by the crossbars are adopted to build the memristor-based multi-layer neural network (MMNN) and the memristor-based convolutional neural network (MCNN). Two kinds of in-situ weight update schemes, which are the fixed-voltage update and the approximately linear update, respectively, are used to train the networks. Consider variations resulted from the inherent characteristics of memristors and the errors of programming voltages, the robustness of MMNN and MCNN to these variations is analyzed. The simulation results on standard datasets show that deep neural networks (DNNs) built by the memristor crossbars work satisfactorily in pattern recognition tasks and have certain robustness to memristor variations.

Identifying complex motifs in massive omics data with a variable-convolutional layer in deep neural network.

Motif identification is among the most common and essential computational tasks for bioinformatics and genomics. Here we proposed a novel convolutional layer for deep neural network, named variable convolutional (vConv) layer, for effective motif identification in high-throughput omics data by learning kernel length from data adaptively. Empirical evaluations on DNA-protein binding and DNase footprinting cases well demonstrated that vConv-based networks have superior performance to their convolutional counterparts regardless of model complexity. Meanwhile, vConv could be readily integrated into multi-layer neural networks as an 'in-place replacement' of canonical convolutional layer. All source codes are freely available on GitHub for academic usage.

Ensemble Based Neural Network for the Classification of MURA Dataset

The Musculoskeletal Radiographs (MURA) dataset, proposed by Stamford Machine Learning (ML) group, has 40,561 images of bone X-rays from 14,863 studies. The X-ray images belong to seven body areas of the upper extremity namely wrist, elbow, finger, humerus, forearm, hand, and shoulder. Radiologists have classified the data into two classes, namely normal and abnormal. Six board-certified Stanford radiologists labeled the data samples using most votes, which is considered the gold standard. The 169 layers deep model, introduced by the Stamford ML group, works well on a par with the gold standard except for the humerus radiographs, despite humerus data labeled with high accuracy. We propose to develop a comparatively shallower version of a neural network and a convolutional network with 10 hidden layers each in an Adaboost framework in the humerus data and the model performance is on par or sometimes superior to the Stamford ML group model. We evaluate the performance of our model using the validation error and Cohen’s kappa coefficients. We have shown that our modeling framework is much faster in terms of the model training time and as accurate compared to the 169 layers of deep neural network introduced by the Stamford ML group. Also, with increased resources, the performance of our model will increase.

BEAN: Interpretable and Efficient Learning With Biologically-Enhanced Artificial Neuronal Assembly Regularization.

Deep neural networks (DNNs) are known for extracting useful information from large amounts of data. However, the representations learned in DNNs are typically hard to interpret, especially in dense layers. One crucial issue of the classical DNN model such as multilayer perceptron (MLP) is that neurons in the same layer of DNNs are conditionally independent of each other, which makes co-training and emergence of higher modularity difficult. In contrast to DNNs, biological neurons in mammalian brains display substantial dependency patterns. Specifically, biological neural networks encode representations by so-called neuronal assemblies: groups of neurons interconnected by strong synaptic interactions and sharing joint semantic content. The resulting population coding is essential for human cognitive and mnemonic processes. Here, we propose a novel Biologically Enhanced Artificial Neuronal assembly (BEAN) regularization1 to model neuronal correlations and dependencies, inspired by cell assembly theory from neuroscience. Experimental results show that BEAN enables the formation of interpretable neuronal functional clusters and consequently promotes a sparse, memory/computation-efficient network without loss of model performance. Moreover, our few-shot learning experiments demonstrate that BEAN could also enhance the generalizability of the model when training samples are extremely limited.

A novel stochastic deep resilient network for effective speech recognition.

Speech recognition is a subjective occurrence. This work proposes a novel stochastic deep resilient network(SDRN) for speech recognition. It uses a deep neural network (DNN) for classification to predict the input speech signal. The hidden layers of DNN and its neurons are additionally optimized to reduce the computation time by using a neural-based opposition whale optimization algorithm (NOWOA). The novelty of the SDRN network is in using NOWOA to recognize large vocabulary isolated and continuous speech signals. The trained DNN features are then utilized for predicting isolated and continuous speech signals. The standard database is used for training and testing. The real-time data (recorded in ambient condition) for isolated words and continuous speech signals are additionally used for validation to increase the accuracy of the SDRN network. The proposed methodology unveils an accuracy of 99.6% and 98.1% for isolated words (standard and real-time) database and 98.7% for continuous speech signal (real-time). The obtained results exhibit the supremacy of SDRN over other techniques.

Toward an Automatic Quality Assessment of Voice-Based Telemedicine Consultations: A Deep Learning Approach.

Maintaining a high quality of conversation between doctors and patients is essential in telehealth services, where efficient and competent communication is important to promote patient health. Assessing the quality of medical conversations is often handled based on a human auditory-perceptual evaluation. Typically, trained experts are needed for such tasks, as they follow systematic evaluation criteria. However, the daily rapid increase of consultations makes the evaluation process inefficient and impractical. This paper investigates the automation of the quality assessment process of patient–doctor voice-based conversations in a telehealth service using a deep-learning-based classification model. For this, the data consist of audio recordings obtained from Altibbi. Altibbi is a digital health platform that provides telemedicine and telehealth services in the Middle East and North Africa (MENA). The objective is to assist Altibbi’s operations team in the evaluation of the provided consultations in an automated manner. The proposed model is developed using three sets of features: features extracted from the signal level, the transcript level, and the signal and transcript levels. At the signal level, various statistical and spectral information is calculated to characterize the spectral envelope of the speech recordings. At the transcript level, a pre-trained embedding model is utilized to encompass the semantic and contextual features of the textual information. Additionally, the hybrid of the signal and transcript levels is explored and analyzed. The designed classification model relies on stacked layers of deep neural networks and convolutional neural networks. Evaluation results show that the model achieved a higher level of precision when compared with the manual evaluation approach followed by Altibbi’s operations team.

Automatic detection of cyberbullying using multi-feature based artificial intelligence with deep decision tree classification

Recent studies have shown that cyberbullying is a rising youth epidemic. In this paper, we develop a novel automated classification model that identifies the cyberbullying texts without fitting them into large dimensional space. On the other hand, a classifier .cannot provide a limited convergent solution due to its overfitting problem. Considering such limitations, we developed a text classification engine that initially pre-processes the tweets, eliminates noise and other background information, extracts the selected features and classifies without data overfitting. The study develops a novel Deep Decision Tree classifier that utilizes the hidden layers of Deep Neural Network (DNN) as its tree node to process the input elements. The validation confirms the accuracy of classification using the novel Deep classifier with its improved text classification accuracy.

Representational Content of Oscillatory Brain Activity during Object Recognition: Contrasting Cortical and Deep Neural Network Hierarchies.

Numerous theories propose a key role for brain oscillations in visual perception. Most of these theories postulate that sensory information is encoded in specific oscillatory components (e.g., power or phase) of specific frequency bands. These theories are often tested with whole-brain recording methods of low spatial resolution (EEG or MEG), or depth recordings that provide a local, incomplete view of the brain. Opportunities to bridge the gap between local neural populations and whole-brain signals are rare. Here, using representational similarity analysis (RSA) in human participants we explore which MEG oscillatory components (power and phase, across various frequency bands) correspond to low or high-level visual object representations, using brain representations from fMRI, or layer-wise representations in seven recent deep neural networks (DNNs), as a template for low/high-level object representations. The results showed that around stimulus onset and offset, most transient oscillatory signals correlated with low-level brain patterns (V1). During stimulus presentation, sustained β (∼20 Hz) and γ (>60 Hz) power best correlated with V1, while oscillatory phase components correlated with IT representations. Surprisingly, this pattern of results did not always correspond to low-level or high-level DNN layer activity. In particular, sustained β band oscillatory power reflected high-level DNN layers, suggestive of a feed-back component. These results begin to bridge the gap between whole-brain oscillatory signals and object representations supported by local neuronal activations.

Dynamic resource allocation for jointing vehicle-edge deep neural network inference

The emergence of mobile edge computing provides an efficient and stable computing platform for intelligent applications of autonomous vehicles, and deep neural network (DNN) based tasks collaborative inference through joint device-edge is considered an effective way to reduce latency. However, the computing resources allocated to the vehicle are dynamic as the number of requesters changes due to the limitation of edge server resources, which causes the best partition point of the DNN is not fixed. In this paper, we consider a dynamic resource allocation scheme to select the best partition point of DNN inference tasks by vehicle-edge collaborative computing. Specifically, the latency constrained DNN tasks of vehicles are partially offloaded to edge at the granularity of DNN layers. Considering the heterogeneity of vehicular computing capabilities and multiple DNN inference tasks, we formulate an optimization problem for dynamic resource allocation and automatically select the best partition point to minimize the overall latency of all vehicles, which is NP-hard. Then we design a chemical reaction optimization based algorithm for low complexity to solve the problem. The results of extensive evaluations illustrate that our proposed scheme is superior to other baseline schemes in terms of overall latency, and with lower failure rate.