Data Representation Learning Research Articles

Semi-Supervised Learning With Heterogeneous Distribution Consistency for Visible Infrared Person Re-Identification.

Visible infrared person re-identification (VI-ReID) exposes considerable challenges because of the modality gaps between the person images captured by daytime visible cameras and nighttime infrared cameras. Several fully-supervised VI-ReID methods have improved the performance with extensive labeled heterogeneous images. However, the identity of the person is difficult to obtain in real-world situations, especially at night. Limited known identities and large modality discrepancies impede the effectiveness of the model to a great extent. In this paper, we propose a novel Semi-Supervised Learning framework with Heterogeneous Distribution Consistency (HDC-SSL) for VI-ReID. Specifically, through investigating the confidence distribution of heterogeneous images, we introduce a Gaussian Mixture Model-based Pseudo Labeling (GMM-PL) method, which adaptively adjusts different thresholds for each modality to label the identity. Moreover, to facilitate the representation learning of unutilized data whose prediction is lower than the threshold, Modality Consistency Regularization (MCR) is proposed to ensure the prediction consistency of the cross-modality pedestrian images and handle the modality variance. Extensive experiments with different label settings on two VI-ReID datasets demonstrate the effectiveness of our method. Particularly, HDC-SSL achieves competitive performance with state-of-the-art fully-supervised VI-ReID methods on RegDB dataset with only 1 visible label and 1 infrared label per class.

Multivariate Time-Series Representation Learning via Hierarchical Correlation Pooling Boosted Graph Neural Network

Representation learning is vital for the performance of Multivariate Time Series (MTS) related tasks. Given high-dimensional MTS data, researchers generally rely on deep learning (DL) models to learn representative features. Among them, the methods that can capture the spatial-temporal dependencies within MTS data generally achieve better performance. However, they ignored hierarchical relations and the dynamic property within MTS data, hindering their performance. To address these problems, we propose a Hierarchical Correlation Pooling boosted graph neural network (HierCorrPool) for MTS data representation learning. First, we propose a novel correlation pooling scheme to learn and capture hierarchical correlations between sensors. Meanwhile, a new assignment matrix is designed to ensure the effective learning of hierarchical correlations by adaptively combining both sensor features and correlations. Second, we learn sequential graphs to represent the dynamic property within MTS data, so that this property can be captured for learning decent representations. We conducted extensive experiments to test our model on various MTS tasks, including remaining useful life prediction, human activity recognition, and sleep stage classification. Experimental results have shown the effectiveness of our proposed model.

Unsupervised Human Activity Recognition Learning for Disassembly Tasks

Large volumes of used electronics are often collected in remanufacturing plants, which requires disassembly before harvesting parts for reuse. Disassembly is mainly conducted manually with low productivity. Recently, human-robot collaboration is considered as a solution. For robots to assist effectively, they should observe work environments and recognize human actions accurately. Rich activity video recording and supervised learning can be used to extract insights; however, supervised learning does not allow robots to self-accomplish the learning process. This study proposes an unsupervised learning framework for achieving video-based human activity recognition. The framework consists of two main elements: a variational autoencoder-based architecture for unlabeled data representation learning, and a hidden Markov model for activity state division. The complete explicit activity classification is validated against ground truth labels; here, we use a case study of disassembling a hard disk drive. The framework shows an average recognition accuracy of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$91.52\%$</tex-math></inline-formula> , higher than competing methods.

Anomaly Detection in Cloud Computing using Knowledge Graph Embedding and Machine Learning Mechanisms

The orchestration of cloud computing infrastructures is challenging, considering the number, heterogeneity and dynamicity of the involved resources, along with the highly distributed nature of the applications that use them for computation and storage. Evidently, the volume of relevant monitoring data can be significant, and the ability to collect, analyze, and act on this data in real time is critical for the infrastructure’s efficient use. In this study, we introduce a novel methodology that adeptly manages the diverse, dynamic, and voluminous nature of cloud resources and the applications that they support. We use knowledge graphs to represent computing and storage resources and illustrate the relationships between them and the applications that utilize them. We then train GraphSAGE to acquire vector-based representations of the infrastructures’ properties, while preserving the structural properties of the graph. These are efficiently provided as input to two unsupervised machine learning algorithms, namely CBLOF and Isolation Forest, for the detection of storage and computing overusage events, where CBLOF demonstrates better performance across all our evaluation metrics. Following the detection of such events, we have also developed appropriate re-optimization mechanisms that ensure the performance of the served applications. Evaluated in a simulated environment, our methods demonstrate a significant advancement in anomaly detection and infrastructure optimization. The results underscore the potential of this closed-loop operation in dynamically adapting to the evolving demands of cloud infrastructures. By integrating data representation and machine learning methods with proactive management strategies, this research contributes substantially to the field of cloud computing, offering a scalable, intelligent solution for modern cloud infrastructures.

LW-CNN-based extraction with optimized encoder-decoder model for detection of diabetic retinopathy

&lt;p&gt;In the field of computer vision, automatic diabetic retinopathy (D.R.) screening is a well-established topic of study. It’s tough since the retinal vessels are hardly distinguishable from the backdrop in the fundus picture, and the structure is complicated. To learn data representations at numerous levels of abstraction, deep learning (DL) allows for the development of computational representations with several processing layers. Small, inconspicuous lesions generated by the disorder are hard to detect since they are tucked away beneath the eye’s structure. In this research, a lightweight convolutional neural network (LW-CNN) was used to extract structures from images of blood vessels, and different preprocessing methods were employed. The features are extracted, and then D.R. is classified using the suggested learning technique, which includes an encoder, dense branch. Effective categorization relies on the usage of multi-scale information collected from various nodes in the network. Grasshopper’s optimisation algorithm (GHOA) is used to fine-tune the recommended classifier’s hyper-parameters. The DIARETDB1 benchmark dataset is assessed using 80% training data and 20% testing data to get a diagnosis of the disease’s severity. The proposed model improved D.R. image classification with accuracy of 0.992 for DIARETDB1 database and 0.981 for APTOS 2019 blindness detection dataset. The state-of-the-art models for D.R. dataset images only achieved less accuracy and precision as compared with the proposed model.&lt;/p&gt;

Criticality Analysis: Bio-Inspired Nonlinear Data Representation.

The representation of arbitrary data in a biological system is one of the most elusive elements of biological information processing. The often logarithmic nature of information in amplitude and frequency presented to biosystems prevents simple encapsulation of the information contained in the input. Criticality Analysis (CA) is a bio-inspired method of information representation within a controlled Self-Organised Critical system that allows scale-free representation. This is based on the concept of a reservoir of dynamic behaviour in which self-similar data will create dynamic nonlinear representations. This unique projection of data preserves the similarity of data within a multidimensional neighbourhood. The input can be reduced dimensionally to a projection output that retains the features of the overall data, yet has a much simpler dynamic response. The method depends only on the Rate Control of Chaos applied to the underlying controlled models, which allows the encoding of arbitrary data and promises optimal encoding of data given biologically relevant networks of oscillators. The CA method allows for a biologically relevant encoding mechanism of arbitrary input to biosystems, creating a suitable model for information processing in varying complexity of organisms and scale-free data representation for machine learning.

Semisupervised Manifold Regularization via a Subnetwork-Based Representation Learning Model.

Semisupervised classification with a few labeled training samples is a challenging task in the area of data mining. Moore-Penrose inverse (MPI)-based manifold regularization (MR) is a widely used technique in tackling semisupervised classification. However, most of the existing MPI-based MR algorithms can only generate loosely connected feature encoding, which is generally less effective in data representation and feature learning. To alleviate this deficiency, we introduce a new semisupervised multilayer subnet neural network called SS-MSNN. The key contributions of this article are as follows: 1) a novel MPI-based MR model using the subnetwork structure is introduced. The subnet model is utilized to enrich the latent space representations iteratively; 2) a one-step training process to learn the discriminative encoding is proposed. The proposed SS-MSNN learns parameters by directly optimizing the entire network, accepting input from one end, and producing output at the other end; and 3) a new semisupervised dataset called HFSWR-RDE is built for this research. Experimental results on multiple domains show that the SS-MSNN achieves promising performance over the other semisupervised learning algorithms, demonstrating fast inference speed and better generalization ability.

Deep Neural Decision Forest (DNDF): A Novel Approach for Enhancing Intrusion Detection Systems in Network Traffic Analysis.

Intrusion detection systems, also known as IDSs, are widely regarded as one of the most essential components of an organization's network security. This is because IDSs serve as the organization's first line of defense against several cyberattacks and are accountable for accurately detecting any possible network intrusions. Several implementations of IDSs accomplish the detection of potential threats throughout flow-based network traffic analysis. Traditional IDSs frequently struggle to provide accurate real-time intrusion detection while keeping up with the changing landscape of threat. Innovative methods used to improve IDSs' performance in network traffic analysis are urgently needed to overcome these drawbacks. In this study, we introduced a model called a deep neural decision forest (DNDF), which allows the enhancement of classification trees with the power of deep networks to learn data representations. We essentially utilized the CICIDS 2017 dataset for network traffic analysis and extended our experiments to evaluate the DNDF model's performance on two additional datasets: CICIDS 2018 and a custom network traffic dataset. Our findings showed that DNDF, a combination of deep neural networks and decision forests, outperformed reference approaches with a remarkable precision of 99.96% by using the CICIDS 2017 dataset while creating latent representations in deep layers. This success can be attributed to improved feature representation, model optimization, and resilience to noisy and unbalanced input data, emphasizing DNDF's capabilities in intrusion detection and network security solutions.

Data representation learning via dictionary learning and self-representation

Data representation learning via dictionary learning and self-representation

Res-TranBiLSTM: An intelligent approach for intrusion detection in the Internet of Things

The Internet of Things (IoT), as the information carrier of the Internet and telecommunications networks, is a new network technology comprising physical entities embedded with electronic components, software and sensors, and characterized by strong complexity and openness. With the massive amount of data, the occurrence of network intrusion is also increasingly frequent, involving industrial control systems, IoT devices, mobile security, cloud services, and telecommunications services. With the diversification and intelligence of cyberattack behaviors, traditional intrusion detection systems (IDSs) face problems—such as insufficient feature extraction and inaccurate model classification—when faced with high-dimensional features and nonlinear massive data. Due to their powerful data representation learning ability, deep learning methods save substantial time in processing high-dimensional and complex intrusion data. On this basis, we propose an intrusion detection model using ResNet, Transformer and BiLSTM (Res-TranBiLSTM) that takes into account both the spatial and temporal features of network traffic. We use the Synthetic Minor Overriding Technique (SMOTE) – Edited Nearest Neighbor (ENN) method to alleviate the degree of data imbalance. In addition, we respectively establish a spatial feature extraction model based on ResNet and a temporal feature extraction model based on Transformer and BiLSTM to extract spatial features and temporal features parallelly. Finally, spatiotemporal features are included to achieve attack detection and classification. Further, simulation experiments are conducted using the public data sets NSL-KDD and CIC-IDS2017. The experiments are implemented using Python programming language and Pytorch framework. The results reveal that the performance of our proposed model is better than that of other models, with accuracy reaching 90.99%, 99.15% and 99.56%, on NSL-KDD dataset, CIC-IDS2017 dataset and MQTTset dataset, respectively. It increased the detection accuracy by about 1%-10% on NSL-KDD dataset and about 0.2%-10% on CIC-IDS2017 dataset, and about 1%-10% on MQTTset dataset. These results demonstrate that this method is effective in constructing and optimizing large-scale IDS in the IoT environment.

Membership Mappings for Practical Secure Distributed Deep Learning

<underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Problem</u> : We consider the problem of privacy-preserving distributed deep learning where data privacy is protected by fully homomorphic encryption. <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Aim</u> : The aim is to develop a method for practical and scalable distributed deep learning with fully homomorphic encrypted data. The method must address the issue arising from the large computational cost associated with fully homomorphic encrypted data to offer a practical and scalable solution. <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Methods</u> : An approach that leverages fuzzy-based membership-mappings for data representation learning is considered for distributed deep learning with fully homomorphic encrypted data. The method introduces globally convergent and robust variational membership-mappings to build local deep models. The local models are combined in a robust and flexible manner by means of fuzzy attributes to build a global model such that the global model can be homomorphically evaluated in an efficient manner. <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Results</u> : The membership-mappings based privacy-preserving distributed deep learning method is accurate, practical, and scalable. This is verified through numerous experiments which include demonstrations using MNIST and Freiburg Groceries datasets, and a biomedical application related to the detection of mental stress on individuals. <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Conclusion</u> : The study develops globally convergent and robust variational membership-mappings for their application to accurate, practical, and scalable privacy-preserving distributed deep learning.

Dynamic Weighted Multitask Learning and Contrastive Learning for Multimodal Sentiment Analysis

Multimodal sentiment analysis (MSA) has attracted more and more attention in recent years. This paper focuses on the representation learning of multimodal data to reach higher prediction results. We propose a model to assist in learning modality representations with multitask learning and contrastive learning. In addition, our approach obtains dynamic weights by considering the homoscedastic uncertainty of each task in multitask learning. Specially, we design two groups of subtasks, which predict the sentiment polarity of unimodal and bimodal representations, to assist in learning representation through a hard parameter-sharing mechanism in the upstream neural network. A loss weight is learned according to the homoscedastic uncertainty of each task. Moreover, a training strategy based on contrastive learning is designed to balance the inconsistency between training and inference caused by the randomness of the dropout layer. This method minimizes the MSE between two submodels. Experimental results on the MOSI and MOSEI datasets show our method achieves better performance than the current state-of-the-art methods by comprehensively considering the intramodality and intermodality interaction information.

Two-Stream Graph Convolutional Network-Incorporated Latent Feature Analysis

Historical Quality-of-Service (QoS) data describing existing user-service invocations are vital to understanding user behaviors and cloud service conditions. Collaborative Filtering (CF) models based on Matrix Factorization (MF) have proven to be highly efficient in performing representation learning in QoS data. However, its performance is hindered by its linear inherence and implicit encoding of collaborative QoS signal. To address this critical issue, we present a novel approach in this paper, dubbed as <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <u>T</u></b> wo-stream <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <u>G</u></b> raph convolutional network-incorporated <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <u>L</u></b> atent <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <u>F</u></b> eature <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <u>A</u></b> nalysis (TGLFA). The proposed TGLFA is significantly different from previous approaches to representation learning in QoS data in the following three aspects. First, it constructs a multilayer fully-connected network to capture the attribute characteristics representing the nonlinear latent features of service. Second, TGLFA constructs a biparty graph to represent the user-service interactions, where the light graph convolutional network is adopted to acquire the high-order connectivity in QoS data. Last, Aiming to improve computational efficiency, the proposed approach considers the mechanism for data density-oriented modeling when building the input and output layers. Detailed experimental results on eight large-scale cases constructed on two real QoS datasets demonstrate that the proposed TGLFA significantly outperforms its state-of-the-art peers in both estimation accuracy for missing QoS data and computational efficiency. The notable results show that TGLFA is a novel and effective approach to QoS data representation learning.

Improving Uncertainty Quantification of Deep Classifiers via Neighborhood Conformal Prediction: Novel Algorithm and Theoretical Analysis

Safe deployment of deep neural networks in high-stake real-world applications require theoretically sound uncertainty quantification. Conformal prediction (CP) is a principled framework for uncertainty quantification of deep models in the form of prediction set for classification tasks with a user-specified coverage (i.e., true class label is contained with high probability). This paper proposes a novel algorithm referred to as Neighborhood Conformal Prediction (NCP) to improve the efficiency of uncertainty quantification from CP for deep classifiers (i.e., reduce prediction set size). The key idea behind NCP is to use the learned representation of the neural network to identify k nearest-neighbor calibration examples for a given testing input and assign them importance weights proportional to their distance to create adaptive prediction sets. We theoretically show that if the learned data representation of the neural network satisfies some mild conditions, NCP will produce smaller prediction sets than traditional CP algorithms. Our comprehensive experiments on CIFAR-10, CIFAR-100, and ImageNet datasets using diverse deep neural networks strongly demonstrate that NCP leads to significant reduction in prediction set size over prior CP methods.

Hierarchical block aggregation network for long-tailed visual recognition

It is usually supposed that training database is manually balanced in traditional visual recognition tasks. However, in nature, data tends to follow long-tailed distributions. In recent years, many plug-and-play methods based on data augmentation or representation learning have been proposed to tackle the long-tailed visual recognition task. Although these methods are effective, we find that when different plug-and-play methods are applied to the same long-tail recognition model, they sometimes fail to promote each other. The reason for this phenomenon may lie in the fact that the overall performance of the model is constrained by the insufficient capability of a traditional feature extractor. Motivated by this fact, we first propose Hierarchical Block Aggregation Network (HBAN), a network structure with stronger feature extraction capability. Then, we design a Quantity-Aware Balanced (QAB) loss and a decoupled training paradigm to optimize HBAN. The effectiveness of HBAN is demonstrated by extensive experiments. In particular, HBAN achieves significant improvements over our baseline on three benchmark datasets, and outperforms the state-of-the-art methods on CIFAR 100-LT.

Valve Stiction Detection Using Multitimescale Feature Consistent Constraint for Time-Series Data

Using neural networks to build a reliable fault detection model is an attractive topic in industrial processes but remains challenging due to the lack of labeled data. We propose a feature learning approach for industrial time-series data based on self-supervised contrastive learning to tackle this challenge. The proposed approach consists of two components: data transformation and representation learning. The data transformation converts the raw time-series to temporal distance matrices capable of storing temporal and spatial information. The representation learning component uses a convolution-based encoder to encode the temporal distance matrices to embedding representations. The encoder is trained using a new constraint called multitimescale feature consistent constraint. Finally, a fault detection framework for the valve stiction detection task is developed based on the feature learning method. The proposed framework is evaluated not only on an industrial benchmark dataset but also on a hardware experimental system and real industrial environments.

Unsupervised representation learning of spontaneous MEG data with nonlinear ICA

Resting-state magnetoencephalography (MEG) data show complex but structured spatiotemporal patterns. However, the neurophysiological basis of these signal patterns is not fully known and the underlying signal sources are mixed in MEG measurements. Here, we developed a method based on the nonlinear independent component analysis (ICA), a generative model trainable with unsupervised learning, to learn representations from resting-state MEG data. After being trained with a large dataset from the Cam-CAN repository, the model has learned to represent and generate patterns of spontaneous cortical activity using latent nonlinear components, which reflects principal cortical patterns with specific spectral modes. When applied to the downstream classification task of audio-visual MEG, the nonlinear ICA model achieves competitive performance with deep neural networks despite limited access to labels. We further validate the generalizability of the model across different datasets by applying it to an independent neurofeedback dataset for decoding the subject's attentional states, providing a real-time feature extraction and decoding mindfulness and thought-inducing tasks with an accuracy of around 70% at the individual level, which is much higher than obtained by linear ICA or other baseline methods. Our results demonstrate that nonlinear ICA is a valuable addition to existing tools, particularly suited for unsupervised representation learning of spontaneous MEG activity which can then be applied to specific goals or tasks when labelled data are scarce.

Few-shot weakly-supervised cybersecurity anomaly detection

With increased reliance on Internet based technologies, cyberattacks compromising users’ sensitive data are becoming more prevalent. The scale and frequency of these attacks are escalating rapidly, affecting systems and devices connected to the Internet. The traditional defense mechanisms may not be sufficiently equipped to handle the complex and ever-changing new threats. The significant breakthroughs in the machine learning methods including deep learning, had attracted interests from the cybersecurity research community for further enhancements in the existing anomaly detection methods. Unfortunately, collecting labelled anomaly data for all new evolving and sophisticated attacks is not practical. Training and tuning the machine learning model for anomaly detection using only a handful of labelled data samples is a pragmatic approach. Therefore, few-shot weakly supervised anomaly detection is an encouraging research direction. In this paper, we propose an enhancement to an existing few-shot weakly-supervised deep learning anomaly detection framework. This framework incorporates data augmentation, representation learning and ordinal regression. We then evaluated and showed the performance of our implemented framework on three benchmark datasets: NSL-KDD, CIC-IDS2018, and TON_IoT.

Stretching Deep Architectures: A Deep Learning Method without Back-Propagation Optimization

In recent years, researchers have proposed many deep learning algorithms for data representation learning. However, most deep networks require extensive training data and a lot of training time to obtain good results. In this paper, we propose a novel deep learning method based on stretching deep architectures that are composed of stacked feature learning models. Hence, the method is called “stretching deep architectures” (SDA). In the feedforward propagation of SDA, feature learning models are firstly stacked and learned layer by layer, and then the stretching technique is applied to map the last layer of the features to a high-dimensional space. Since the feature learning models are optimized effectively, and the stretching technique can be easily calculated, the training of SDA is very fast. More importantly, the learning of SDA does not need back-propagation optimization, which is quite different from most of the existing deep learning models. We have tested SDA in visual texture perception, handwritten text recognition, and natural image classification applications. Extensive experiments demonstrate the advantages of SDA over traditional feature learning models and related deep learning models.

A Self-Supervised Fault Detection for UAV Based on Unbalanced Flight Data Representation Learning and Wavelet Analysis

This paper aims to build a Self-supervised Fault Detection Model for UAVs combined with an Auto-Encoder. With the development of data science, it is imperative to detect UAV faults and improve their safety. Many factors affect the fault of a UAV, such as the voltage of the generator, angle of attack, and position of the rudder surface. A UAV is a typical complex system, and its flight data are typical high-dimensional large sample data sets. In practical applications such as UAV fault detection, the fault data only appear in a small part of the data sets. In this study, representation learning is used to extract the normal features of the flight data and reduce the dimensions of the data. The normal data are used for the training of the Auto-Encoder, and the reconstruction loss is used as the criterion for fault detection. An Improved Auto-Encoder suitable for UAV Flight Data Sets is proposed in this paper. In the Auto-Encoder, we use wavelet analysis to extract the low-frequency signals with different frequencies from the flight data. The Auto-Encoder is used for the feature extraction and reconstruction of the low-frequency signals with different frequencies. To improve the effectiveness of the fault localization at inference, we develop a new fault factor location model, which is based on the reconstruction loss of the Auto-Encoder and edge detection operator. The UAV Flight Data Sets are used for hard-landing detection, and an average accuracy of 91.01% is obtained. Compared with other models, the results suggest that the developed Self-supervised Fault Detection Model for UAVs has better accuracy. Concluding this study, an explanation is provided concerning the proposed model’s good results.