Unsupervised Tasks Research Articles

Deep semi-supervised clustering for multi-variate time-series

Huge amount of data are nowadays produced by a large and disparate family of sensors, which typically measure multiple variables over time. Such rich information can be profitably organized as multivariate time-series. Collect enough labelled samples to set up supervised analysis for such kind of data is challenging while a reasonable assumption is to dispose of a limited background knowledge that can be injected in the analysis process. In this context, semi-supervised clustering methods represent a well suited tool to get the most out of such reduced amount of knowledge. With the aim to deal with multivariate time-series analysis under a limited background knowledge setting, we propose a semi-supervised (constrained) deep embedding time-series clustering framework that exploits knowledge supervision modeled as Must- and Cannot-link constraints. More in detail, our proposal, named conDetSEC (constrained Deep embedding time SEries Clustering), is based on Gated Recurrent Units (GRUs) with the aim to explicitly manage the temporal dimension associated to multi-variate time series data. conDetSEC implements a procedure in which an embedding generation step is combined with a clustering refinement step. Both steps exploit the small amount of available knowledge provided by Must- and Cannot-link constraints. More specifically, during the data embedding generation the constraints are used by jointly optimizing the network parameters via both unsupervised and semi-supervised tasks, while at the refinement step they are used in conjunction with the goal to stretch the embedding manifold towards the clustering centroids to recover a more clear cluster structure. Experimental evaluation on real-world benchmarks coming from diverse domains has highlighted the effectiveness of our proposal in comparison with state-of-the-art unsupervised and semi-supervised time-series clustering methods.

AAGAN: Accuracy-Aware Generative Adversarial Network for Supervised Tasks

Owing to their great success in unsupervised tasks, generative adversarial networks (GANs) are widely adopted for supervised (conditional) image-generation tasks such as in-painting. Unfortunately, designing an objective function using GANs is not trivial. For supervised tasks, the generator trained only with GAN loss does not yield the corresponding target for the given input because GAN is trained to match the data distribution, not to find the exact answer. Therefore, the loss function of a generator is often formulated by linearly combining supervised loss and GAN loss as similar to multi-task learning, expecting that each loss’s weakness is complemented. Contrary to expectation, both losses cause a conflict in practice due to different optimum of each loss, yielding low objective and subjective image qualities. To address this problem, we empirically investigated the conflict caused by using a conventional GAN with pixel-wise losses; we then propose a novel (relativistic) accuracy-aware discriminator. Based on the proposed discriminator, we developed an accuracy-aware GAN (AAGAN) and proved its optimality under an ideal assumption. We then propose a relativistic accuracy-aware GAN (RAAGAN) by considering practical assumptions. Experimental results on supervised tasks demonstrated that the proposed schemes alleviated the competition between losses and outperformed conventional GANs in terms of both objective and subjective qualities.

RemixIT: Continual Self-Training of Speech Enhancement Models via Bootstrapped Remixing

We present <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">RemixIT</i> , a simple yet effective self-supervised method for training speech enhancement without the need of a single isolated in-domain speech nor a noise waveform. Our approach overcomes limitations of previous methods which make them dependent on clean in-domain target signals and thus, sensitive to any domain mismatch between train and test samples. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">RemixIT</i> is based on a continuous self-training scheme in which a pre-trained teacher model on out-of-domain data infers estimated pseudo-target signals for in-domain mixtures. Then, by permuting the estimated clean and noise signals and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">remixing</i> them together, we generate a new set of bootstrapped mixtures and corresponding pseudo-targets which are used to train the student network. Vice-versa, the teacher periodically refines its estimates using the updated parameters of the latest student models. Experimental results on multiple speech enhancement datasets and tasks not only show the superiority of our method over prior approaches but also showcase that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">RemixIT</i> can be combined with any separation model as well as be applied towards any semi-supervised and unsupervised domain adaptation task. Our analysis, paired with empirical evidence, sheds light on the inside functioning of our self-training scheme wherein the student model keeps obtaining better performance while observing severely degraded pseudo-targets.

Unsupervised Domain Adaptation through high-order tensor matching with discriminative manifold learning

Unsupervised Domain Adaptation (UDA) seeks to exploit the source domain knowledge to address similar but unsupervised target domain tasks. To perform UDA, most of the existing works typically learn domain-invariant representations or align the distributions across source and target domains through low-order statistical matching, which fail to explore more discriminative and high-order adaptation information. In this work, we proposed a kind of UDA, namely UDA through high-order tensor matching with discriminative manifold learning (UDA-HOTDML), by exploring the discriminative manifold structure jointly with the high-order tensor to more desirably match cross-domain alignment. In UDA-HOTDML, multiple aspects of domain knowledge are exploited to benefit more generalizable UDA by jointly modeling the discriminative structures of target domain, consistence as well as high-order tensor statistical characteristics. Finally, our evaluation results show that the proposed UDA-HOTDML method outperforms many related UDA methods.

Research Review for Broad Learning System: Algorithms, Theory, and Applications.

In recent years, the appearance of the broad learning system (BLS) is poised to revolutionize conventional artificial intelligence methods. It represents a step toward building more efficient and effective machine-learning methods that can be extended to a broader range of necessary research fields. In this survey, we provide a comprehensive overview of the BLS in data mining and neural networks for the first time, focusing on summarizing various BLS methods from the aspects of its algorithms, theories, applications, and future open research questions. First, we introduce the basic pattern of BLS manifestation, the universal approximation capability, and essence from the theoretical perspective. Furthermore, we focus on BLS's various improvements based on the current state of the theoretical research, which further improves its flexibility, stability, and accuracy under general or specific conditions, including classification, regression, semisupervised, and unsupervised tasks. Due to its remarkable efficiency, impressive generalization performance, and easy extendibility, BLS has been applied in different domains. Next, we illustrate BLS's practical advances, such as computer vision, biomedical engineering, control, and natural language processing. Finally, the future open research problems and promising directions for BLSs are pointed out.

Unsupervised SMT: an analysis of Indic languages and a low resource language

ABSTRACT The rapid globalisation in language technology and the Internet’s fast expansion have brought nations and their cultures close together, and the demand for inter-language interactions has risen enormously. However, in many low-resource languages (LRL) pairings and areas, Machine Translation (MT) is still not viable because of a lack of parallel data. The challenge of MT is still unsolved. Recent studies employing monolingual datasets have shown excellent outcomes in Phrase-based Statistical MT (PBSMT) and Neural MT (NMT) systems. However, earlier researchers have demonstrated that unsupervised Statistical MT surpasses unsupervised NMT, especially for different language pairings. The study unveils the compendium of ten unsupervised SMT systems translation tasks utilizing a monolingual dataset from the Dravidian and Indo-Aryan language families; and a low-resource endangered language. The machine-translated experimental outcomes examined the system using different tokenizers and investigated them for various language pairs using different evaluation metrics for various iterations. The statistical significance of test results has been computed for each evaluation metric to check the true system quality of the translation tasks.

Stochastic gate-based autoencoder for unsupervised hyperspectral band selection

Due to its strong feature representation ability, the deep learning (DL)-based method is preferable for the unsupervised band selection task of hyperspectral image (HSI). However, the current DL-based UBS methods have not further investigated the nonlinear relationship between spectral bands, a more robust DL model with effective loss function is desired. To solve the above problem, a novel stochastic gate-based autoencoder (SGAE) has been proposed for the UBS task. With the proposed stochastic gate layer, the desired band subset with learnable parameters can be directly obtained. For obtaining better UBS results, a nonlinear regularization term is added with the loss function to supervise the training process of SGAE. Furthermore, an early stopping criteria with a regularization term-based threshold is developed. Experimental results on four publicly available remote sensing datasets prove the effectiveness of our SGAE.

An unsupervised domain adaptation brain CT segmentation method across image modalities and diseases

Computed tomography (CT) is the primary diagnostic tool for brain diseases. To determine the appropriate treatment plan, it is necessary to ascertain the patient’s bleeding volume. Automatic segmentation algorithms for hemorrhagic lesions can significantly improve efficiency and avoid treatment delays. However, for deep supervised learning algorithms, a large amount of labeled training data is usually required, making them difficult to apply clinically. In this study, we propose an unsupervised domain adaptation method that is an unsupervised domain adaptation segmentation model that can be trained across modalities and diseases. We call it AMD-DAS for brain CT hemorrhage segmentation tasks. This circumvents the heavy data labeling task by converting the source domain data (MRI with glioma) to our task’s required data (CT with Intraparenchymal hemorrhage (IPH)). Our model implements a two-stage domain adaptation process to achieve this objective. In the first stage, we train a pseudo-CT image synthesis network using the CycleGAN architecture through a matching mechanism and domain adaptation approach. In the second stage, we use the model trained in the first stage to synthesize the pseudo-CT images. We use the pseudo-CT with source domain labels and real CT images to train a domain-adaptation segmentation model. Our method exhibits a better performance than the basic one-stage domain adaptation segmentation method (+11.55 Dice score) and achieves an 86.93 Dice score in the IPH unsupervised segmentation task. Our model can be trained without using a ground-truth label, therefore increasing its application potential. Our implementation is publicly available at https://github.com/GuanghuiFU/AMD-DAS-Brain-CT-Segmentation.

Semantic driven attention network with attribute learning for unsupervised person re-identification

Unsupervised domain adaptation (UDA) person re-identification (re-ID) aims to transfer knowledge from a labeled source domain to guide the task proposed on the unlabeled target domain, in which people share different identifications and cross multiple camera views within two different domains. Consequently, traditional UDAre-ID techniques generally suffer due to the negative transfer caused by the inevitable noise generated by variant backgrounds, while the foregrounds also lack sufficient reliable identification knowledge to guarantee the qualified cross-domain re-ID. To remedy the raised negative transfer caused by variant backgrounds, we propose a novel body structure estimation(BSE) mechanism enforced semantic driven attention network (SDA), which enables the designed model with semantic effectiveness to distinguish the foreground and background. In searching for the reliable feature representations as in the foreground areas, we propose a novel label refinery mechanism to dynamically optimize the traditional attribute learning techniques for the strengthened personal attribute features and thus resulting the qualified UDA-re-ID. Extensive experiments demonstrate the effectiveness of our method in solving unsupervised domain adaptation person re-ID task on three large-scale datasets including Market-1501, DukeMTMC-reID and MSMT17.

Unsupervised MRI Reconstruction via Zero-Shot Learned Adversarial Transformers.

Supervised reconstruction models are characteristically trained on matched pairs of undersampled and fully-sampled data to capture an MRI prior, along with supervision regarding the imaging operator to enforce data consistency. To reduce supervision requirements, the recent deep image prior framework instead conjoins untrained MRI priors with the imaging operator during inference. Yet, canonical convolutional architectures are suboptimal in capturing long-range relationships, and priors based on randomly initialized networks may yield suboptimal performance. To address these limitations, here we introduce a novel unsupervised MRI reconstruction method based on zero-Shot Learned Adversarial TransformERs (SLATER). SLATER embodies a deep adversarial network with cross-attention transformers to map noise and latent variables onto coil-combined MR images. During pre-training, this unconditional network learns a high-quality MRI prior in an unsupervised generative modeling task. During inference, a zero-shot reconstruction is then performed by incorporating the imaging operator and optimizing the prior to maximize consistency to undersampled data. Comprehensive experiments on brain MRI datasets clearly demonstrate the superior performance of SLATER against state-of-the-art unsupervised methods.

Adversarial Examples Can Be Effective Data Augmentation for Unsupervised Machine Learning

Adversarial examples causing evasive predictions are widely used to evaluate and improve the robustness of machine learning models. However, current studies focus on supervised learning tasks, relying on the ground truth data label, a targeted objective, or supervision from a trained classifier. In this paper, we propose a framework of generating adversarial examples for unsupervised models and demonstrate novel applications to data augmentation. Our framework exploits a mutual information neural estimator as an information theoretic similarity measure to generate adversarial examples without supervision. We propose a new MinMax algorithm with provable convergence guarantees for the efficient generation of unsupervised adversarial examples. Our framework can also be extended to supervised adversarial examples. When using unsupervised adversarial examples as a simple plugin data augmentation tool for model retraining, significant improvements are consistently observed across different unsupervised tasks and datasets, including data reconstruction, representation learning, and contrastive learning. Our results show novel methods and considerable advantages in studying and improving unsupervised machine learning via adversarial examples.

Multi-Centroid Representation Network for Domain Adaptive Person Re-ID

Recently, many approaches tackle the Unsupervised Domain Adaptive person re-identification (UDA re-ID) problem through pseudo-label-based contrastive learning. During training, a uni-centroid representation is obtained by simply averaging all the instance features from a cluster with the same pseudo label. However, a cluster may contain images with different identities (label noises) due to the imperfect clustering results, which makes the uni-centroid representation inappropriate. In this paper, we present a novel Multi-Centroid Memory (MCM) to adaptively capture different identity information within the cluster. MCM can effectively alleviate the issue of label noises by selecting proper positive/negative centroids for the query image. Moreover, we further propose two strategies to improve the contrastive learning process. First, we present a Domain-Specific Contrastive Learning (DSCL) mechanism to fully explore intra-domain information by comparing samples only from the same domain. Second, we propose Second-Order Nearest Interpolation (SONI) to obtain abundant and informative negative samples. We integrate MCM, DSCL, and SONI into a unified framework named Multi-Centroid Representation Network (MCRN). Extensive experiments demonstrate the superiority of MCRN over state-of-the-art approaches on multiple UDA re-ID tasks and fully unsupervised re-ID tasks.

Technical Perspective of TURL

Several efforts aim at representing tabular data with neural models for supporting target applications at the intersection of natural language processing (NLP) and databases (DB) [1-3]. The goal is to extend to structured data the recent neural architectures, which achieve state of the art results in NLP applications. Language models (LMs) are usually pre-trained with unsupervised tasks on a large text corpus. The output LM is then fine-tuned on a variety of downstream tasks with a small set of specific examples. This process has many advantages, because the LM contains information about textual structure and content, which are used by the target application without manually defining features.

Emergence of associative learning in a neuromorphic inference network

Objective. In the theoretical framework of predictive coding and active inference, the brain can be viewed as instantiating a rich generative model of the world that predicts incoming sensory data while continuously updating its parameters via minimization of prediction errors. While this theory has been successfully applied to cognitive processes—by modelling the activity of functional neural networks at a mesoscopic scale—the validity of the approach when modelling neurons as an ensemble of inferring agents, in a biologically plausible architecture, remained to be explored. Approach. We modelled a simplified cerebellar circuit with individual neurons acting as Bayesian agents to simulate the classical delayed eyeblink conditioning protocol. Neurons and synapses adjusted their activity to minimize their prediction error, which was used as the network cost function. This cerebellar network was then implemented in hardware by replicating digital neuronal elements via a low-power microcontroller. Main results. Persistent changes of synaptic strength—that mirrored neurophysiological observations—emerged via local (neurocentric) prediction error minimization, leading to the expression of associative learning. The same paradigm was effectively emulated in low-power hardware showing remarkably efficient performance compared to conventional neuromorphic architectures. Significance. These findings show that: (a) an ensemble of free energy minimizing neurons—organized in a biological plausible architecture—can recapitulate functional self-organization observed in nature, such as associative plasticity, and (b) a neuromorphic network of inference units can learn unsupervised tasks without embedding predefined learning rules in the circuit, thus providing a potential avenue to a novel form of brain-inspired artificial intelligence.

WCATN: Unsupervised deep learning to classify weather conditions from outdoor images

Weather classification from single images is significant for many outdoor computer vision applications, while it has not been thoroughly studied. Existing methods view it as a supervised task under the guidance of weather labels. Though these methods have made great success, they are not applicable to real-world applications because of their reliance on massive human-annotated weather images. In this paper, we take the initial attempt to treat weather classification as an unsupervised task, i.e. to classify weather conditions from unlabeled outdoor images. Specifically, we propose a three-step unsupervised approach to automatically group images into weather clusters. In the first step, i.e. weather feature learning, semantically meaningful weather features are learned by employing a self-supervised task. Furthermore, we also introduce online triplet mining into the self-supervised task to extract discriminative weather features. In the second step, i.e. weather clustering, we design a learnable clustering method via mining the nearest neighbors and enforcing consistency between each sample and its nearest neighbors. To alleviate the uncertainty of weather clustering due to noisy nearest neighbors, we propose a self-labeling method as the third step to employ the already well-classified samples for fine-tuning. Experimental results on two publicly available weather image datasets demonstrate that the proposed approach achieves promising performance.

Unsupervised domain adaptation with Joint Adversarial Variational AutoEncoder

Unsupervised domain adaptation techniques increase the classification performance of tasks from the target domain by utilizing the information in a related source domain. Since the target labeled samples are unavailable, matching similar samples across different domains effectively becomes increasingly hard, which retards the progress in this field. In this paper, we propose a Joint Adversarial Variational AutoEncoder (JVA2E) for unsupervised domain adaptation tasks. JVA2E chooses variational autoencoder as the basic framework to improve the generative ability. Both the marginal and conditional distributions are considered for joint distribution adaptation. The Wasserstein distance is chosen for improving the final performance. Multiple unique classifiers are carefully designed for generating pseudo labels which are utilized to increase intra-class similarity as well as narrow conditional distribution. Experiments are conducted on three publicly available datasets and the final results are compared with some state-of-the-art techniques. It illustrates that our proposed method yields better performances for most tasks against previous domain adaptation methods.

Virtual exam for Parkinson’s disease enables frequent and reliable remote measurements of motor function

Sensor-based remote monitoring could help better track Parkinson’s disease (PD) progression, and measure patients’ response to putative disease-modifying therapeutic interventions. To be useful, the remotely-collected measurements should be valid, reliable, and sensitive to change, and people with PD must engage with the technology. We developed a smartwatch-based active assessment that enables unsupervised measurement of motor signs of PD. Participants with early-stage PD (N = 388, 64% men, average age 63) wore a smartwatch for a median of 390 days. Participants performed unsupervised motor tasks both in-clinic (once) and remotely (twice weekly for one year). Dropout rate was 5.4%. Median wear-time was 21.1 h/day, and 59% of per-protocol remote assessments were completed. Analytical validation was established for in-clinic measurements, which showed moderate-to-strong correlations with consensus MDS-UPDRS Part III ratings for rest tremor (⍴ = 0.70), bradykinesia (⍴ = −0.62), and gait (⍴ = −0.46). Test-retest reliability of remote measurements, aggregated monthly, was good-to-excellent (ICC = 0.75–0.96). Remote measurements were sensitive to the known effects of dopaminergic medication (on vs off Cohen’s d = 0.19–0.54). Of note, in-clinic assessments often did not reflect the patients’ typical status at home. This demonstrates the feasibility of smartwatch-based unsupervised active tests, and establishes the analytical validity of associated digital measurements. Weekly measurements provide a real-life distribution of disease severity, as it fluctuates longitudinally. Sensitivity to medication-induced change and improved reliability imply that these methods could help reduce sample sizes needed to demonstrate a response to therapeutic interventions or disease progression.

DEDGCN: Dual Evolving Dynamic Graph Convolutional Network

With the wide application of graph data in many fields, the research of graph representation learning technology has become the focus of scholars’ attention. Especially, dynamic graph representation learning is an important part of solving the problem of change graph in reality. On the one hand, most dynamic graph representation methods focus either on graph structure changes or node embedding changes, ignoring the internal relationship. On the other hand, most dynamic graph neural networks require learn node embeddings from specific tasks, resulting in poor universality of node embeddings and cannot be used in unsupervised tasks. Hence, Dual Evolving Dynamic Graph Convolutional Network (DEDGCN) was proposed to solve the above problems. DEDGCN uses the recurrent neural network to push the evolvement of GCN and nodes, from which it can extract the structural features of dynamic graph and learns the stability features of nodes, respectively, forming an adaptive dynamic graph convolution network. DEDGCN can be classified as unsupervised graph convolutional network. Thus, it is capable of training the unlabeled dynamic graph, it has more extensive application scenarios, and the calculated node embedding has strong generality. We evaluate our proposed method on experimental data in three tasks which are node classification, edge classification, and link prediction. In the classification task, facing the graph with large scale, complex connection relationship, and uncertain change rule, the F1 value of node classification task obtained by DEDGCN reaches 77%, and the F1 value of edge classification task reaches more than 90%. The results show that DEDGCN is effective in capturing graph features, and the effect of DEDGCN is much higher than other baseline methods, which proves the importance of capturing node stability features in dynamic graph representation learning. At the same time, the ability of DEDGCN in unsupervised tasks is further verified by using clustering and anomaly detection tasks, which proves that DEDGCN learning network embedding is widely used.

An Unsupervised Feature Extraction Approach Based on Self-Expression.

Feature extraction algorithms lack good interpretability during the projection learning. To solve this problem, an unsupervised feature extraction algorithm, that is, block diagonal projection (BDP), based on self-expression is proposed. Specifically, if the original data are projected into a low-dimensional subspace by a feature extraction algorithm, although the data may be more compact, the new features obtained may not be as explanatory as the original sample features. Therefore, by imposing L2,1 norm constraint on the projection matrix, the projection matrix can be of row sparsity. On one hand, discriminative features can be selected to make the projection matrix to be more interpretable. On the other hand, irrelevant or redundant features can be suppressed. The proposed model integrates feature extraction and selection into one framework. In addition, since self-expression can well excavate the correlation between samples or sample features, the unsupervised feature extraction task can be better guided using this property between them. At the same time, the block diagonal representation regular term is introduced to directly pursue the block diagonal representation. Thus, the accuracy of pattern recognition tasks such as clustering and classification can be improved. Finally, the effectiveness of BDP in linear dimensionality reduction and classification is proved on various reference datasets. The experimental results show that this algorithm is superior to previous feature extraction counterparts.

Unsupervised authorship attribution using feature selection and weighted cosine similarity

This paper presents a computational model for the unsupervised authorship attribution task based on a traditional machine learning scheme. An improvement over the state of the art is achieved by comparing different feature selection methods on the PAN17 author clustering dataset. To achieve this improvement, specific pre-processing and features extraction methods were proposed, such as a method to separate tokens by type to assign them to only one category. Similarly, special characters are used as part of the punctuation marks to improve the result obtained when applying typed character n-grams. The Weighted cosine similarity measure is applied to improve the B3 F-score by reducing the vector values where attributes are exclusive. This measure is used to define distances between documents, which later are occupied by the clustering algorithm to perform authorship attribution.