High-level Representation Research Articles

An optimized fuzzy deep learning model for data classification based on NSGA-II

As a powerful paradigm, deep learning (DL) models have been used in many applications for classification tasks in images, text, and audio. Through DL models, we can learn task-driven features from big data. However, DL models are fully deterministic and cannot handle uncertain and imprecise data. DL models are often sensitive to noise in data and do not operate well in areas where data are vague. Moreover, when there is a large feature set or high-dimensional data with irrelevant and redundant features, the DL models’ performance decreases in classification due to training with irrelevant features. To gain reliable results in such high-dimensional problems, DL models require a large amount of data which usually grows exponentially concerning the number of features. Data uncertainty problems and irrelevant features in DL models cause low performance in classification tasks. This paper proposes an optimized fuzzy deep learning (OFDL) model for data classification based on Non-Dominated Sorting Genetic Algorithm II (NSGA-II). OFDL utilizes optimization in the composition of DL and fuzzy learning via the NSGA-II in multi-modal learning. To achieve effective classification, OFDL first considers intelligent feature selection by finding the best trade-offs between two conflicting objective functions, minimizing the number of features, and maximizing the accuracy (maximizing weights of selected features). Next, to reach optimized backpropagation and fuzzy membership functions, OFDL utilizes Pareto optimal solutions for multi-objective optimization using NSGA-II based on their objective functions. Furthermore, the fusion layer in OFDL fused optimized views of DL and fuzzy learning that provides a high-level representation of inputs and optimum features for classification tasks where the data contain high uncertainties and noises. This functionality gives valuable attributes during classification since identifying and selecting appropriate features ensures prompt and correct class. Also, it provides deep insight into tackling the effect of ambiguous data and each feature’s uncertainty in the classification tasks. The examination of OFDL reveals good performance in terms of F-measure, accuracy, recall, precision, and True Positive Rate (TPR) compared to fuzzy classifiers. Furthermore, OFDL has higher accuracy in classification tasks than earlier fuzzy DNN models.

TEMPO: A transformer-based mutation prediction framework for SARS-CoV-2 evolution

The widespread of SARS-CoV-2 presents a significant threat to human society, as well as public health and economic development. Extensive efforts have been undertaken to battle against the pandemic, whereas effective approaches such as vaccination would be weakened by the continuous mutations, leading to considerable attention being attracted to the mutation prediction. However, most previous studies lack attention to phylogenetics. In this paper, we propose a novel and effective model TEMPO for predicting the mutation of SARS-CoV-2 evolution. Specifically, we design a phylogenetic tree-based sampling method to generate sequence evolution data. Then, a transformer-based model is presented for the site mutation prediction after learning the high-level representation of these sequence data. We conduct experiments to verify the effectiveness of TEMPO, leveraging a large-scale SARS-CoV- 2 dataset. Experimental results show that TEMPO is effective for mutation prediction of SARS- CoV-2 evolution and outperforms several state-of-the-art baseline methods. We further perform mutation prediction experiments of other infectious viruses, to explore the feasibility and robustness of TEMPO, and experimental results verify its superiority. The codes and datasets are freely available at https://github.com/ZJUDataIntelligence/TEMPO.

CRMSS: predicting circRNA-RBP binding sites based on multi-scale characterizing sequence and structure features.

Circular RNAs (circRNAs) are reverse-spliced and covalently closed RNAs. Their interactions with RNA-binding proteins (RBPs) have multiple effects on the progress of many diseases. Some computational methods are proposed to identify RBP binding sites on circRNAs but suffer from insufficient accuracy, robustness and explanation. In this study, we first take the characteristics of both RNA and RBP into consideration. We propose a method for discriminating circRNA-RBP binding sites based on multi-scale characterizing sequence and structure features, called CRMSS. For circRNAs, we use sequence ${k}\hbox{-}{mer}$ embedding and the forming probabilities of local secondary structures as features. For RBPs, we combine sequence and structure frequencies of RNA-binding domain regions to generate features. We capture binding patterns with multi-scale residual blocks. With BiLSTM and attention mechanism, we obtain the contextual information of high-level representation for circRNA-RBP binding. To validate the effectiveness of CRMSS, we compare its predictive performance with other methods on 37 RBPs. Taking the properties of both circRNAs and RBPs into account, CRMSS achieves superior performance over state-of-the-art methods. In the case study, our model provides reliable predictions and correctly identifies experimentally verified circRNA-RBP pairs. The code of CRMSS is freely available at https://github.com/BioinformaticsCSU/CRMSS.

LayerUMAP: A tool for visualizing and understanding deep learning models in biological sequence classification using UMAP.

Despite the impressive success of deep learning techniques in various types of classification and prediction tasks, interpreting these models and explaining their predictions are still major challenges. In this article, we present an easy-to-use command line tool capable of visualizing and analyzing alternative representations of biological observations learned by deep learning models. This new tool, namely, layerUMAP, integrates autoBioSeqpy software and the UMAP library to address learned high-level representations. An important advantage of the tool is that it provides an interactive option that enables users to visualize the outputs of hidden layers along the depth of the model. We use two different classes of examples to illustrate the potential power of layerUMAP, and the results demonstrate that layerUMAP can provide insightful visual feedback about models and further guide us to develop better models.

Joint alignment of the distribution in input and feature space for cross-domain aerial image semantic segmentation

Benefiting from the development of deep learning, researchers have made significant progress and achieved superior performance in the semantic segmentation of remote sensing (RS) data. However, when encountering an unseen scenario, the performance of a trained model deteriorates dramatically because of the domain shift. Unsupervised domain adaptation (UDA) provides an alternative to address the issue. Aligning the high-level representations via adversarial learning is a popular way, but it is difficult when there is a large gap in input space. With this consideration, we design a framework to jointly align the distribution in input and feature space. For input space alignment, we unify the resolution for the consistency of content and propose a lightweight module named Digital Number Transformer (DNT) to reduce the visual differences. For feature space alignment, we design a Multi-Scale Feature aggregation (MSF) module and introduce the Fine-Grained Discriminator (FGD) to conduct a category-level alignment at multiple layers so that features can be fully aligned and negative transfer can be reduced. We carried out experiments in diverse cross-domain scenarios, including the discrepancy in geographic position and the discrepancy in both geographic position and imaging mode. Comprehensive experiments demonstrate that our method outperforms other state-of-the-art methods in all scenarios.

A survey of field programmable gate array (FPGA)-based graph convolutional neural network accelerators: challenges and opportunities.

Graph convolutional networks (GCNs) based on convolutional operations have been developed recently to extract high-level representations from graph data. They have shown advantages in many critical applications, such as recommendation system, natural language processing, and prediction of chemical reactivity. The problem for the GCN is that its target applications generally pose stringent constraints on latency and energy efficiency. Several studies have demonstrated that field programmable gate array (FPGA)-based GCNs accelerators, which balance high performance and low power consumption, can continue to achieve orders-of-magnitude improvements in the inference of GCNs models. However, there still are many challenges in customizing FPGA-based accelerators for GCNs. It is necessary to sort out the current solutions to these challenges for further research. For this purpose, we first summarize the four challenges in FPGA-based GCNs accelerators. Then we introduce the process of the typical GNN algorithm and several examples of representative GCNs. Next, we review the FPGA-based GCNs accelerators in recent years and introduce their design details according to different challenges. Moreover, we compare the key metrics of these accelerators, including resource utilization, performance, and power consumption. Finally, we anticipate the future challenges and directions for FPGA-based GCNs accelerators: algorithm and hardware co-design, efficient task scheduling, higher generality, and faster development.

DEEP QUALITY-CONSTRAINED LSTM FOR TEXTUAL DATA ANALYSIS

The creation of language models is a vital object of natural language processing because it creates a possible distribution in a sequence of words and offers a unique representation of each occurrence of a word. Therefore, an intelligent language model can differentiate the subtle nuances of language and model its syntax and semantics. It also allows the acquisition of high-level representations. In addition, it serves as an initial model with helpful information, which can be transferred to various other word processing methods for a deeper and more complete understanding of the language. This work proposes an innovative Long Short-Term Memory (LSTM) neural network whose implementation is based on the use of an attention mechanism as a measure of alignment of the output with the input sequence and a corresponding coverage optimization mechanism. The optimization mechanism informs the model about the possible outcomes at each step which have already been produced in previous actions. The adaptation and evaluation of the system’s design parameters were tested in multidimensional and challenging data sets We were considering parametric and heuristic procedures for finding the optimal combination of hyper-parameters. Experimental results demonstrate the quality of the proposed system, revealing essential directions for the design of similar systems.

Neuron-level Interpretation of Deep NLP Models: A Survey

Abstract The proliferation of Deep Neural Networks in various domains has seen an increased need for interpretability of these models. Preliminary work done along this line, and papers that surveyed such, are focused on high-level representation analysis. However, a recent branch of work has concentrated on interpretability at a more granular level of analyzing neurons within these models. In this paper, we survey the work done on neuron analysis including: i) methods to discover and understand neurons in a network; ii) evaluation methods; iii) major findings including cross architectural comparisons that neuron analysis has unraveled; iv) applications of neuron probing such as: controlling the model, domain adaptation, and so forth; and v) a discussion on open issues and future research directions.

Modeling Cognitive Load as a Self-Supervised Brain Rate with Electroencephalography and Deep Learning.

The principal reason for measuring mental workload is to quantify the cognitive cost of performing tasks to predict human performance. Unfortunately, a method for assessing mental workload that has general applicability does not exist yet. This is due to the abundance of intuitions and several operational definitions from various fields that disagree about the sources or workload, its attributes, the mechanisms to aggregate these into a general model and their impact on human performance. This research built upon these issues and presents a novel method for mental workload modelling from EEG data employing deep learning. This method is self-supervised, employing a continuous brain rate, an index of cognitive activation, and does not require human declarative knowledge. The aim is to induce models automatically from data, supporting replicability, generalisability and applicability across fields and contexts. This specific method is a convolutional recurrent neural network trainable with spatially preserving spectral topographic head-maps from EEG data, aimed at fitting a novel brain rate variable. Findings demonstrate the capacity of the convolutional layers to learn meaningful high-level representations from EEG data since within-subject models had, on average, a test Mean Absolute Percentage Error of around 11%. The addition of a Long-Short Term Memory layer for handling sequences of high-level representations was not significant, although it did improve their accuracy. These findings point to the existence of quasi-stable blocks of automatically learnt high-level representations of cognitive activation because they can be induced through convolution and seem not to be dependent on each other over time, intuitively matching the non-stationary nature of brain responses. Additionally, across-subject models, induced with data from an increasing number of participants, thus trained with data containing more variability, obtained a similar accuracy to the within-subject models. This highlights the potential generalisability of the induced high-level representations across people, suggesting the existence of subject-independent cognitive activation patterns. This research contributes to the body of knowledge by providing scholars with a novel computational method for mental workload modelling that aims to be generally applicable and does not rely on ad hoc human crafted models.

Prediction of Click-through Rate of Marketing Advertisements Using Deep Learning

Aiming at the defect that the click-through rate of marketing advertisements cannot provide accurate prediction results for the company in time in the marketing strategy of Internet companies, this paper uses a deep learning algorithm to establish a prediction model for the click-through rate of marketing advertisements. The suggested model is called high-order cross-feature network (HCN). Furthermore, this paper also introduces the combination of feature vectors into the graph structure and as the nodes in the graph; therefore, the graph neural network (GNN) is used to obtain the high-level representation ability of structured data more fully. Through numerical simulations, we observed that HCN has the capability to provide Internet companies with more accurate advertising business information, user information, and advertising content. Moreover, HCN model is more reasonable to adjust the advertising strategy and can provide better user experience. The simulation outcomes indicate that the suggested HCN approach has noble adaptability and high correctness in forecasting the click-through rate of marketing advertisements. We observed that this improvement, in terms of predictions precisions and accuracies, can be as high as 17.52% higher than the deep neural network (DNN) method and 10.45% higher than the factorization network (FM) approach.

A Decoder-Free Variational Deep Embedding for Unsupervised Clustering.

In deep clustering frameworks, autoencoder (AE)- or variational AE-based clustering approaches are the most popular and competitive ones that encourage the model to obtain suitable representations and avoid the tendency for degenerate solutions simultaneously. However, for the clustering task, the decoder for reconstructing the original input is usually useless when the model is finished training. The encoder-decoder architecture limits the depth of the encoder so that the learning capacity is reduced severely. In this article, we propose a decoder-free variational deep embedding for unsupervised clustering (DFVC). It is well known that minimizing reconstruction error amounts to maximizing a lower bound on the mutual information (MI) between the input and its representation. That provides a theoretical guarantee for us to discard the bloated decoder. Inspired by contrastive self-supervised learning, we can directly calculate or estimate the MI of the continuous variables. Specifically, we investigate unsupervised representation learning by simultaneously considering the MI estimation of continuous representations and the MI computation of categorical representations. By introducing the data augmentation technique, we incorporate the original input, the augmented input, and their high-level representations into the MI estimation framework to learn more discriminative representations. Instead of matching to a simple standard normal distribution adversarially, we use end-to-end learning to constrain the latent space to be cluster-friendly by applying the Gaussian mixture distribution as the prior. Extensive experiments on challenging data sets show that our model achieves higher performance over a wide range of state-of-the-art clustering approaches.

Partial-order reduction for parity games and parameterised Boolean equation systems

In model checking, reduction techniques can be helpful tools to fight the state-space explosion problem. Partial-order reduction (POR) is a well-known example, and many POR variants have been developed over the years. However, none of these can be used in the context of model checking stutter-sensitive temporal properties. We propose POR techniques for parity games, a well-established formalism for solving a variety of decision problems, including model checking. As a result, we obtain the first POR method that is sound for the full modal upmu -calculus. We show how our technique can be applied to the fixed point logic called parameterised Boolean equation systems, which provides a high-level representation of parity games. Experiments with our implementation indicate that substantial reductions can be achieved.

Defending against adversarial attacks on Covid-19 classifier: A denoiser-based approach

Covid-19 has posed a serious threat to the existence of the human race. Early detection of the virus is vital to effectively containing the virus and treating the patients. Profound testing methods such as the Real-time reverse transcription-polymerase chain reaction (RT-PCR) test and the Rapid Antigen Test (RAT) are being used for detection, but they have their limitations. The need for early detection has led researchers to explore other testing techniques. Deep Neural Network (DNN) models have shown high potential in medical image classification and various models have been built by researchers which exhibit high accuracy for the task of Covid-19 detection using chest X-ray images. However, it is proven that DNNs are inherently susceptible to adversarial inputs, which can compromise the results of the models. In this paper, the adversarial robustness of such Covid-19 classifiers is evaluated by performing common adversarial attacks, which include the Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD). Using these attacks, it is found that the accuracy of the models for Covid-19 samples decreases drastically. In the medical domain, adversarial training is the most widely explored technique to defend against adversarial attacks. However, using this technique requires replacing the original model and retraining it by including adversarial samples. Another defensive technique, High-Level Representation Guided Denoiser (HGD), overcomes this limitation by employing an adversarial filter which is also transferable across models. Moreover, the HGD architecture, being suitable for high-resolution images, makes it a good candidate for medical image applications. In this paper, the HGD architecture has been evaluated as a potential defensive technique for the task of medical image analysis. Experiments carried out show an increased accuracy of up to 82% in the white box setting. However, in the black box setting, the defense completely fails to defend against adversarial samples.

An Artificial Intelligence-Based Approach to Social Data-Aware Optimization for Enterprise Management

Enterprise management has always been a hot issue in society. In today’s society, enterprises are no longer individuals cut off from society and no longer have profit as their sole purpose, but need to exist and develop in combination with social information. With the increasing competition among enterprises, the original enterprise management methods can no longer meet the needs of sustainable development of enterprises, nor can they effectively utilize social data. To better understand the daily emotions of corporate workers, we use a multimodal emotion recognition method in this paper. Multimodal emotion recognition refers to the recognition of human emotional states through different modal information such as speech, visual, and text related to human emotional expressions, which has important research significance in the fields of human-computer interaction, artificial intelligence, and emotional computing and has received much attention from researchers. Given the great success of deep learning methods developed in recent years for various tasks, various deep neural networks are now used to learn high-level representations of emotional features for multimodal emotion recognition. The analysis of employee sentiment is complemented by traditional management methods that make full use of social data. In this paper, based on the study of a single enterprise management model, the proposed model contains five substructure modules, starting with feature inputs, extracting features at different levels through three convolutional modules and outputting recognition results through a softmax classifier. The focus is on how to utilize social data, while combining deep learning with traditional enterprise management methods to fill the research gap in this area in academia.

Mixed Feature Prediction on Boundary Learning for Point Cloud Semantic Segmentation

Existing point cloud semantic segmentation approaches do not perform well on details, especially for the boundary regions. However, supervised-learning-based methods depend on costly artificial annotations for performance improvement. In this paper, we bridge this gap by designing a self-supervised pretext task applicable to point clouds. Our main innovation lies in the mixed feature prediction strategy during the pretraining stage, which facilitates point cloud feature learning with boundary-aware foundations. Meanwhile, a dynamic feature aggregation module is proposed to regulate the range of receptive field according to the neighboring pattern of each point. In this way, more spatial details are preserved for discriminative high-level representations. Extensive experiments across several point cloud segmentation datasets verify the superiority of our proposed method, including ShapeNet-part, ScanNet v2, and S3DIS. Furthermore, transfer learning on point cloud classification and object detection tasks demonstrates the generalization ability of our method.

STAR-Lite: A light-weight scalable self-taught learning framework for older adults’ activity recognition

Recognizing activities for older adults is challenging as we observe a variety of activity patterns caused due to aging (e.g., limited dexterity, limb control, slower response time) or/and underlying health conditions (e.g., dementia). However, existing literature with deep learning methods has successfully recognized activities when the dataset contains high-quality annotations and is captured in a controlled environment. On the contrary, data captured in a real-world environment, especially with older adults exhibiting memory-related symptoms, varying psychological and mental health status, reliance on caregivers to perform daily activities, and unavailability of domain-specific annotators, makes obtaining quality data with annotations challenging; leaving us with limited labeled data and abundant unlabeled data. In this paper, we hypothesize that projecting the labeled data representations comprising a specific set of activities onto a new representation space characterized by the unlabeled data comprising activities beyond the limited activities in the labeled dataset would help us rely less on the annotated data to improve activity detection performance. Motivated by this, we propose STAR-Lite, a self-taught learning framework that involves a pre-training framework to prepare the new representation space considering activities beyond the initial labels in the labeled dataset. STAR-Lite projects the labeled data representations on the new representation space characterized by unlabeled data labels and learns higher-level representations of the labeled dataset while optimizing inter- and intra- class distances without explicitly using a computation hungry similarity-based approach. We demonstrate that our proposed approach, STAR-Lite (a) improves activity recognition performance in a supervised setting and (b) is feasible for real-world deployment. To enhance the feasibility of deploying STAR-Lite on devices with limited memory resources, we explore model compression techniques such as pruning and quantization and propose a novel layer-wise pruning-rate optimization technique that effectively compresses the network while preserving the model performance. The evaluation was performed using the Alzheimer’s Activity Recognition dataset (AAR) captured from 25 individuals living in a retirement community center with IRB approval (#Y18NR12035) using an in-house SenseBox infrastructure while concurrently assessing the clinical evaluation of the participants for dementia, and independent living. Our extensive evaluation reveals that STAR-Lite can detect activities with an F1-score of 85.12% despite 62% reduction in model size and 5% improvement of execution time on a resource constrained device.

From Spatial to Spectral Domain, a New Perspective for Detecting Adversarial Examples

Deep neural networks (DNNs) have been closely related to the Pandora’s box from the moment of its birth. Although it achieves a high accuracy significantly in real-world tasks (e.g., object detecting and speech recognition), it still retains fatal vulnerabilities and flaws. Malicious attackers can manipulate DNN model misclassification just by adding tiny perturbations to the original image. These crafted samples are also called adversarial examples. One of the effective defense methods is to detect them before feeding them into the model. In this paper, we delve into the representation of adversarial examples in the original spatial and spectral domains. By qualitative and quantitative analysis, it is confirmed that the high-level representation and high-frequency components of abnormal samples contain richer discriminative information. To further explore the influence mechanism between the two factors, we perform an ablation study and the results show a win-win effect. Utilizing the finding, a detecting method (HLFD) is proposed based on extracting high-level representation and high-frequency components. Compared with other state-of-the-art detection methods, we achieve a better detection performance in most scenarios via a series of experiments conducted on MNIST, CIFAR-10, CIFAR-100, SVHN, and Tiny-ImageNet. In particular, we improve detection rates by a large margin on DeepFool and CW attacks.

EBSNN: Extended Byte Segment Neural Network for Network Traffic Classification

Network traffic classification is important to intrusion detection and network management. Most of existing methods are based on machine learning techniques and rely on the features extracted manually from flows or packets. However, with the rapid growth of network applications, it is difficult for these approaches to handle new complex applications. In this article, we design a novel neural network, the Extended Byte Segment Neural Network (EBSNN), to classify netwrk traffic. EBSNN first divides a packet into header segments and payload segments, which are then fed into encoders composed of the recurrent neural networks with the attention mechanism. Based on the outputs, another encoder learns the high-level representation of the whole packet. In particular, side-channel features are learned from header segments to improve the performance. Finally, the label of the packet is obtained by the softmax function. Furthermore, EBSNN can classify network flows by examining the first few packets. Thorough experiments on the real-world datasets show that EBSNN achieves better performance than the state-of-the-art methods in both the application identification task and the website identification task.

High-level visual search in children with autism.

Visual search has been classified as easy feature search, with rapid target detection and little set size dependence, versus slower difficult search with focused attention, with set size–dependent speed. Reverse hierarchy theory attributes these classes to rapid high cortical-level vision at a glance versus low-level vision with scrutiny, attributing easy search to high-level representations. Accordingly, faces “pop out” of heterogeneous object photographs. Individuals with autism have difficulties recognizing faces, and we now asked if this disability disturbs their search for faces. We compare search times and set size slopes for children with autism spectrum disorders (ASDs) and those with neurotypical development (NT) when searching for faces. Human face targets were found rapidly, with shallow set size slopes. The between-group difference between slopes (18.8 vs. 11.3 ms/item) is significant, suggesting that faces may not “pop out” as easily, but in our view does not warrant classifying ASD face search as categorically different from that of NT children. We also tested search for different target categories, dog and lion faces, and nonface basic categories, cars and houses. The ASD group was generally a bit slower than the NT group, and their slopes were somewhat steeper. Nevertheless, the overall dependencies on target category were similar: human face search fastest, nonface categories slowest, and dog and lion faces in between. We conclude that autism may spare vision at a glance, including face detection, despite its reported effects on face recognition, which may require vision with scrutiny. This dichotomy is consistent with the two perceptual modes suggested by reverse hierarchy theory.

Bayesian Integration in a Spiking Neural System for Sensorimotor Control.

The brain continuously estimates the state of body and environment, with specific regions that are thought to act as Bayesian estimator, optimally integrating noisy and delayed sensory feedback with sensory predictions generated by the cerebellum. In control theory, Bayesian estimators are usually implemented using high-level representations. In this work, we designed a new spike-based computational model of a Bayesian estimator. The state estimator receives spiking activity from two neural populations encoding the sensory feedback and the cerebellar prediction, and it continuously computes the spike variability within each population as a reliability index of the signal these populations encode. The state estimator output encodes the current state estimate. We simulated a reaching task at different stages of cerebellar learning. The activity of the sensory feedback neurons encoded a noisy version of the trajectory after actual movement, with an almost constant intrapopulation spiking variability. Conversely, the activity of the cerebellar output neurons depended on the phase of the learning process. Before learning, they fired at their baseline not encoding any relevant information, and the variability was set to be higher than that of the sensory feedback (more reliable, albeit delayed). When learning was complete, their activity encoded the trajectory before the actual execution, providing an accurate sensory prediction; in this case, the variability was set to be lower than that of the sensory feedback. The state estimator model optimally integrated the neural activities of the afferent populations, so that the output state estimate was primarily driven by sensory feedback in prelearning and by the cerebellar prediction in postlearning. It was able to deal even with more complex scenarios, for example, by shifting the dominant source during the movement execution if information availability suddenly changed. The proposed tool will be a critical block within integrated spiking, brain-inspired control systems for simulations of sensorimotor tasks.