Unsupervised Learning Scenarios Research Articles

Implicit sensing self-supervised learning based on graph multi-pretext tasks for traffic flow prediction

AbstractIn recent years, spatio-temporal graph neural networks (GNNs) have successfully been used to improve traffic prediction by modeling intricate spatio-temporal dependencies in irregular traffic networks. However, these approaches may not capture the intrinsic properties of traffic data and can suffer from overfitting due to their local nature. This paper introduces the Implicit Sensing Self-Supervised learning model (ISSS), which leverages a multi-pretext task framework for traffic flow prediction. By transforming data into an alternative feature space, ISSS effectively captures both specific and general representations through self-supervised tasks, including contrastive learning and spatial jigsaw puzzles. This enhancement promotes a deeper understanding of traffic features, improved regularization, and more accurate representations. Comparative experiments on six datasets demonstrate the effectiveness of ISSS in learning general and discriminative features in both supervised and unsupervised modes. ISSS outperforms existing models, demonstrating its capabilities in improving traffic flow predictions while addressing challenges associated with local operations and overfitting. Comprehensive evaluations across various traffic prediction datasets, have established the validity of the proposed approach. Unsupervised learning scenarios have shown the improvements in RMSE for the METR-LA and PEMSBAY datasets of 0.39 and 0.35 for location-dependent and location-independent tasks, respectively. In supervised learning scenarios, for the same datasets, the improvements were 1.16 for location-dependent tasks and 0.55 for location-independent tasks.

Generative artificial intelligence: a systematic review and applications

AbstractIn recent years, the study of artificial intelligence (AI) has undergone a paradigm shift. This has been propelled by the groundbreaking capabilities of generative models both in supervised and unsupervised learning scenarios. Generative AI has shown state-of-the-art performance in solving perplexing real-world conundrums in fields such as image translation, medical diagnostics, textual imagery fusion, natural language processing, and beyond. This paper documents the systematic review and analysis of recent advancements and techniques in Generative AI with a detailed discussion of their applications including application-specific models. Indeed, the major impact that generative AI has made to date, has been in language generation with the development of large language models, in the field of image translation and several other interdisciplinary applications of generative AI. Moreover, the primary contribution of this paper lies in its coherent synthesis of the latest advancements in these areas, seamlessly weaving together contemporary breakthroughs in the field. Particularly, how it shares an exploration of the future trajectory for generative AI. In conclusion, the paper ends with a discussion of Responsible AI principles, and the necessary ethical considerations for the sustainability and growth of these generative models.

BisSiam: Bispectrum Siamese Network Based Contrastive Learning for UAV Anomaly Detection

In recent years, a surging number of unmanned aerial vehicles (UAVs) are pervasively utilized in many areas. However, the increasing number of UAVs may cause privacy and security issues such as voyeurism and espionage. It is critical for individuals or organizations to manage their behaviors and proactively prevent the misbehaved invasion of unauthorized UAVs through effective anomaly detection. The UAV anomaly detection framework needs to cope with complex signals in noisy-prone environments and to function with very limited labeled samples. This paper proposes BISSIAM, a novel framework that is capable of identifying UAV presence, types, and operation modes. BISSIAM converts UAV signals to bispectrum as the input and exploits a siamese network-based contrastive learning model to learn the vector encoding. A sampling mechanism is proposed for optimizing the sample size involved in the model training whilst ensuring the model accuracy without compromising the training efficiency. Finally, we present a similarity-based fingerprint matching mechanism for detecting unseen UAVs without the need of retraining the whole model. Experimental results show that our approach outperforms other baselines and can reach 92.85% accuracy of UAV type detection in unsupervised learning scenarios, and 91.4% accuracy for detecting the UAV type of the out-of-sample UAVs.

Explainable AI approach with original vegetation data classifies spatio-temporal nitrogen in flows from ungauged catchments to the Great Barrier Reef

Transfer of processed data and parameters to ungauged catchments from the most similar gauged counterpart is a common technique in water quality modelling. But catchment similarities for Dissolved Inorganic Nitrogen (DIN) are ill posed, which affects the predictive capability of models reliant on such methods for simulating DIN. Spatial data proxies to classify catchments for most similar DIN responses are a demonstrated solution, yet their applicability to ungauged catchments is unexplored. We adopted a neural network pattern recognition model (ANN-PR) and explainable artificial intelligence approach (SHAP-XAI) to match all ungauged catchments that flow to the Great Barrier Reef to gauged ones based on proxy spatial data. Catchment match suitability was verified using a neural network water quality (ANN-WQ) simulator trained on gauged catchment datasets, tested by simulating DIN for matched catchments in unsupervised learning scenarios. We show that discriminating training data to DIN regime benefits ANN-WQ simulation performance in unsupervised scenarios ( p< 0.05). This phenomenon demonstrates that proxy spatial data is a useful tool to classify catchments with similar DIN regimes. Catchments lacking similarity with gauged ones are identified as priority monitoring areas to gain observed data for all DIN regimes in catchments that flow to the Great Barrier Reef, Australia.

Theoretical Aspects in Penalty Hyperparameters Optimization

Learning processes play an important role in enhancing understanding and analyzing real phenomena. Most of these methodologies revolve around solving penalized optimization problems. A significant challenge arises in the choice of the penalty hyperparameter, which is typically user-specified or determined through Grid search approaches. There is a lack of automated tuning procedures for the estimation of these hyperparameters, particularly in unsupervised learning scenarios. In this paper, we focus on the unsupervised context and propose a bi-level strategy to address the issue of tuning the penalty hyperparameter. We establish suitable conditions for the existence of a minimizer in an infinite-dimensional Hilbert space, along with presenting some theoretical considerations. These results can be applied in situations where obtaining an exact minimizer is unfeasible. Working on the estimation of the hyperparameter with the gradient-based method, we also introduce a modified version of Ekeland’s principle as a stopping criterion for these methods. Our approach distinguishes from conventional techniques by reducing reliance on random or black-box strategies, resulting in stronger mathematical generalization.

Perturbation bounds for (nearly) orthogonally decomposable tensors with statistical applications

Abstract We develop deterministic perturbation bounds for singular values and vectors of orthogonally decomposable tensors, in a spirit similar to classical results for matrices such as those due to Weyl, Davis, Kahan and Wedin. Our bounds demonstrate intriguing differences between matrices and higher order tensors. Most notably, they indicate that for higher order tensors perturbation affects each essential singular value/vector in isolation, and its effect on an essential singular vector does not depend on the multiplicity of its corresponding singular value or its distance from other singular values. Our results can be readily applied and provide a unified treatment to many different problems involving higher order orthogonally decomposable tensors. In particular, we illustrate the implications of our bounds through connected yet seemingly different high-dimensional data analysis tasks: the unsupervised learning scenario of tensor SVD and the supervised task of tensor regression, leading to new insights in both of these settings.

Learning unsupervised node representation from multi-view network

This paper studies the problem of learning node representations for networks with multiple views, which aims to infer robust node representations by simultaneously considering multiple views during the representations learning process. We propose an effective method for this task, named as Multi-View Representation Learning (MVRL). The new method extends the matrix factorization model for node representation learning of multi-view network in unsupervised representation learning scenario, which simultaneously learns a set of view weights to identify the quality of each view, and the network representations as matrix factorization of the weighted combination of multiple views. An efficient optimization method with linear complexity is proposed to solve the new model, and a simple yet efficient method is proposed for fast updating of the new node’s vector representation without updating the whole nodes’ representation vectors. We have evaluated the performance of our proposed approach on five real-world multi-view network datasets. Experimental results on the node classification task demonstrated the superior performance and efficiency of our proposed method.

On the uniform concentration bounds and large sample properties of clustering with Bregman divergences

Clustering with Bregman divergence has been used in literature to unify centroid‐based parametric clustering approaches and to allow the detection of nonspherical clusters within the data. Although empirically useful, the large sample theoretical aspects of Bregman clustering techniques remain largely unexplored. In this paper, we attempt to bridge the gap between the theory and practice of centroid‐based Bregman hard clustering by providing uniform deviation bounds on the clustering objective. Our theoretical analysis relies on the famous Vapnik–Chervonenkis (VC) theory, which, although has been extensively used in a supervised learning context, remains largely unexplored in finding empirical risks for unsupervised learning scenarios. As opposed to most of the theoretical works on clustering, our framework allows the number of features (p) to vary with the number of observations (n). The strong consistency of the sample cluster centroids, under standard assumptions in the literature, also follows as a corollary under this general framework. Furthermore, we show that the rate of convergence is at most of the order , under the standard regularity conditions.

Distribution preserving learning for unsupervised feature selection

Abstract Selection of most relevant features from high-dimensional data is difficult especially in unsupervised learning scenario, this is because there is an absence of class labels that would guide the search for relevant features. In this work, we propose a distribution preserving feature selection (DPFS) method for unsupervised feature selection. Specifically, we select those features such that the distribution of the data can be preserved. Theoretical analysis show that our proposed DPFS method share some excellent properties of kernel method. Moreover, traditional “wrapper” and “filter” feature selection methods often involve an exhaustive search optimization, feature selection problem is treated as variable of optimization problem in our proposed method, the optimization is tractable. Extensive experimental results over various real-life data sets have demonstrated the effectiveness of the proposed algorithm.

Feature Selection With $\ell_{2,1-2}$ Regularization.

Feature selection aims to select a subset of features from high-dimensional data according to a predefined selecting criterion. Sparse learning has been proven to be a powerful technique in feature selection. Sparse regularizer, as a key component of sparse learning, has been studied for several years. Although convex regularizers have been used in many works, there are some cases where nonconvex regularizers outperform convex regularizers. To make the process of selecting relevant features more effective, we propose a novel nonconvex sparse metric on matrices as the sparsity regularization in this paper. The new nonconvex regularizer could be written as the difference of the $\ell _{2,1}$ norm and the Frobenius ( $\ell _{2,2}$ ) norm, which is named the $\ell _{2,1-2}$ . To find the solution of the resulting nonconvex formula, we design an iterative algorithm in the framework of ConCave-Convex Procedure (CCCP) and prove its strong global convergence. An adopted alternating direction method of multipliers is embedded to solve the sequence of convex subproblems in CCCP efficiently. Using the scaled cluster indictors of data points as pseudolabels, we also apply $\ell _{2,1-2}$ to the unsupervised case. To the best of our knowledge, it is the first work considering nonconvex regularization for matrices in the unsupervised learning scenario. Numerical experiments are performed on real-world data sets to demonstrate the effectiveness of the proposed method.

LLE Score: A New Filter-Based Unsupervised Feature Selection Method Based on Nonlinear Manifold Embedding and Its Application to Image Recognition.

The task of feature selection is to find the most representative features from the original high-dimensional data. Because of the absence of the information of class labels, selecting the appropriate features in unsupervised learning scenarios is much harder than that in supervised scenarios. In this paper, we investigate the potential of locally linear embedding (LLE), which is a popular manifold learning method, in feature selection task. It is straightforward to apply the idea of LLE to the graph-preserving feature selection framework. However, we find that this straightforward application suffers from some problems. For example, it fails when the elements in the feature are all equal; it does not enjoy the property of scaling invariance and cannot capture the change of the graph efficiently. To solve these problems, we propose a new filter-based feature selection method based on LLE in this paper, which is named as LLE score. The proposed criterion measures the difference between the local structure of each feature and that of the original data. Our experiments of classification task on two face image data sets, an object image data set, and a handwriting digits data set show that LLE score outperforms state-of-the-art methods, including data variance, Laplacian score, and sparsity score.

A Novel Unsupervised Adaptive Learning Method for Long-Term Electromyography (EMG) Pattern Recognition.

Performance degradation will be caused by a variety of interfering factors for pattern recognition-based myoelectric control methods in the long term. This paper proposes an adaptive learning method with low computational cost to mitigate the effect in unsupervised adaptive learning scenarios. We presents a particle adaptive classifier (PAC), by constructing a particle adaptive learning strategy and universal incremental least square support vector classifier (LS-SVC). We compared PAC performance with incremental support vector classifier (ISVC) and non-adapting SVC (NSVC) in a long-term pattern recognition task in both unsupervised and supervised adaptive learning scenarios. Retraining time cost and recognition accuracy were compared by validating the classification performance on both simulated and realistic long-term EMG data. The classification results of realistic long-term EMG data showed that the PAC significantly decreased the performance degradation in unsupervised adaptive learning scenarios compared with NSVC (9.03% ± 2.23%, p < 0.05) and ISVC (13.38% ± 2.62%, p = 0.001), and reduced the retraining time cost compared with ISVC (2 ms per updating cycle vs. 50 ms per updating cycle).

Feature selection with effective distance

As more features are introduced in pattern recognition and machine learning applications, feature selection remains a critically important task to find the most compact representation of data, especially in unsupervised learning scenarios without enough class labels. Although there are a number of unsupervised feature selection methods in the literature, most of existing methods only focus on using conventional distances (e.g., Euclidean distance) to measure the similarity between two samples, which could not capture the dynamic structure of data due to the static characteristics of conventional distances. To reflect the dynamic structure of data, in this paper, we propose a set of effective distance-based feature selection methods, where a probabilistically motivated effective distance is used to measure the similarity of samples. Specifically, we first develop a sparse representation-based algorithm to compute the effective distance. Then, we propose three new filter-type unsupervised feature selection methods using effective distance, including an effective distance-based Laplacian Score (EDLS), and two effective distance-based Sparsity Scores (i.e., EDSS-1, and EDSS-2). Experimental results of clustering and classification tasks on a series of benchmark data sets show that our effective distance-based feature selection methods can achieve better performance than conventional methods using Euclidean distance.

Sparsity preserving score for feature selection

Compared with supervised feature selection, selecting features in unsupervised learning scenarios is a much harder problem due to the lack of label information. In this paper, we propose sparsity preserving score (SPS) for unsupervised feature selection based on recent advances in sparse representation technique. SPS evaluates the importance of a feature by its power of sparse reconstructive relationship preserving. Specially, SPS selects features that minimize reconstruction residual based on sparse representation in the space of selected features. SPS aims to jointly select features by transforming data from a high-dimensional space of original features to a low-dimensional space of selected features through a special binary feature selection matrix. When the sparse representation is fixed, our searching strategy is an essentially discrete optimization and our theoretical analysis guarantees our objective function can be easily solved with a closed-form solution. The experimental results on two face data sets demonstrate the effectiveness and efficiency of our algorithm.

Robust Model-Based Learning via Spatial-EM Algorithm

This paper presents a new robust EM algorithm for the finite mixture learning procedures. The proposed Spatial-EM algorithm utilizes median-based location and rank-based scatter estimators to replace sample mean and sample covariance matrix in each M step, hence enhancing stability and robustness of the algorithm. It is robust to outliers and initial values. Compared with many robust mixture learning methods, the Spatial-EM has the advantages of simplicity in implementation and statistical efficiency. We apply Spatial-EM to supervised and unsupervised learning scenarios. More specifically, robust clustering and outlier detection methods based on Spatial-EM have been proposed. We apply the outlier detection to taxonomic research on fish species novelty discovery. Two real datasets are used for clustering analysis. Compared with the regular EM and many other existing methods such as K-median, X-EM and SVM, our method demonstrates superior performance and high robustness.

Unsupervised feature selection via maximum projection and minimum redundancy

Dimensionality reduction is an important and challenging task in machine learning and data mining. It can facilitate data clustering, classification and information retrieval. As an efficient technique for dimensionality reduction, feature selection is about finding a small feature subset preserving the most relevant information. In this paper, we propose a new criterion, called maximum projection and minimum redundancy feature selection, to address unsupervised learning scenarios. First, the feature selection is formalized with the use of the projection matrices and then characterized equivalently as a matrix factorization problem. Second, an iterative update algorithm and a greedy algorithm are proposed to tackle this problem. Third, kernel techniques are considered and the corresponding algorithm is also put forward. Finally, the proposed algorithms are compared with four state-of-the-art feature selection methods. Experimental results reported for six publicly datasets demonstrate the superiority of the proposed algorithms.

Neuronal Assembly Dynamics in Supervised and Unsupervised Learning Scenarios

The dynamic formation of groups of neurons--neuronal assemblies--is believed to mediate cognitive phenomena at many levels, but their detailed operation and mechanisms of interaction are still to be uncovered. One hypothesis suggests that synchronized oscillations underpin their formation and functioning, with a focus on the temporal structure of neuronal signals. In this context, we investigate neuronal assembly dynamics in two complementary scenarios: the first, a supervised spike pattern classification task, in which noisy variations of a collection of spikes have to be correctly labeled; the second, an unsupervised, minimally cognitive evolutionary robotics tasks, in which an evolved agent has to cope with multiple, possibly conflicting, objectives. In both cases, the more traditional dynamical analysis of the system's variables is paired with information-theoretic techniques in order to get a broader picture of the ongoing interactions with and within the network. The neural network model is inspired by the Kuramoto model of coupled phase oscillators and allows one to fine-tune the network synchronization dynamics and assembly configuration. The experiments explore the computational power, redundancy, and generalization capability of neuronal circuits, demonstrating that performance depends nonlinearly on the number of assemblies and neurons in the network and showing that the framework can be exploited to generate minimally cognitive behaviors, with dynamic assembly formation accounting for varying degrees of stimuli modulation of the sensorimotor interactions.

A Variance Minimization Criterion to Feature Selection Using Laplacian Regularization

In many information processing tasks, one is often confronted with very high-dimensional data. Feature selection techniques are designed to find the meaningful feature subset of the original features which can facilitate clustering, classification, and retrieval. In this paper, we consider the feature selection problem in unsupervised learning scenarios, which is particularly difficult due to the absence of class labels that would guide the search for relevant information. Based on Laplacian regularized least squares, which finds a smooth function on the data manifold and minimizes the empirical loss, we propose two novel feature selection algorithms which aim to minimize the expected prediction error of the regularized regression model. Specifically, we select those features such that the size of the parameter covariance matrix of the regularized regression model is minimized. Motivated from experimental design, we use trace and determinant operators to measure the size of the covariance matrix. Efficient computational schemes are also introduced to solve the corresponding optimization problems. Extensive experimental results over various real-life data sets have demonstrated the superiority of the proposed algorithms.