Unsupervised Mode Research Articles

The Challenges of Machine Learning: A Critical Review

The concept of learning has multiple interpretations, ranging from acquiring knowledge or skills to constructing meaning and social development. Machine Learning (ML) is considered a branch of Artificial Intelligence (AI) and develops algorithms that can learn from data and generalize their judgment to new observations by exploiting primarily statistical methods. The new millennium has seen the proliferation of Artificial Neural Networks (ANNs), a formalism able to reach extraordinary achievements in complex problems such as computer vision and natural language recognition. In particular, designers claim that this formalism has a strong resemblance to the way the biological neurons operate. This work argues that although ML has a mathematical/statistical foundation, it cannot be strictly regarded as a science, at least from a methodological perspective. The main reason is that ML algorithms have notable prediction power although they cannot necessarily provide a causal explanation about the achieved predictions. For example, an ANN could be trained on a large dataset of consumer financial information to predict creditworthiness. The model takes into account various factors like income, credit history, debt, spending patterns, and more. It then outputs a credit score or a decision on credit approval. However, the complex and multi-layered nature of the neural network makes it almost impossible to understand which specific factors or combinations of factors the model is using to arrive at its decision. This lack of transparency can be problematic, especially if the model denies credit and the applicant wants to know the specific reasons for the denial. The model’s “black box” nature means it cannot provide a clear explanation or breakdown of how it weighed the various factors in its decision-making process. Secondly, this work rejects the belief that a machine can simply learn from data, either in supervised or unsupervised mode, just by applying statistical methods. The process of learning is much more complex, as it requires the full comprehension of a learned ability or skill. In this sense, further ML advancements, such as reinforcement learning and imitation learning denote encouraging similarities to similar cognitive skills used in human learning.

Semi-supervised generative adversarial networks for improved colorectal polyp classification using histopathological images

Early and accurate detection of dysplasia in colorectal polyps can improve prognosis and increase survival chances. Recently, automated learning-based approaches using histopathological images have been adopted for improved classification of polyps. The supervised learning approaches do not provide a reliable classification performance due to limited annotated samples. But, in unsupervised learning, some hidden features are extracted from the unlabeled data which may not be effective in discriminating the complex patterns of the dataset. A generative adversarial network (GAN) is proposed in this work based on a semi-supervised framework for colorectal polyp classification using histopathological images. Our framework learns the discriminating features in an adversarial manner from the limited labeled and huge unlabeled data. In the supervised mode, the discriminator of the proposed model is trained to classify the real histopathological images, whereas, in the unsupervised mode, it tries to discriminate between real and fake images, similar to the classical GAN network. By training in unsupervised mode, the discriminator can identify and extract the subtle features from unlabeled images, to develop a generalized robust model. Our technique yielded classification accuracies of 87.50% and 76.25% using 25% and 50% majority voting schemes, respectively, on the UniToPatho dataset.

M6A regulator-mediated methylation modification patterns and tumor microenvironment immune infiltration with prognostic analysis in esophageal cancer

Esophageal cancer is a highly malignant disease with poor prognosis. Despite recent advances in the study of esophageal cancer, there has been only limited improvement in the treatment and prognosis. N6-methyladenosine (m6A), a type of RNA modification, has been extensively investigated and is involved in many biological behaviors, including tumorigenesis and progression. Thus, more research on m6A modification may increase our understanding of esophageal cancer pathogenesis and provide potential targets. In our study, we integrated the public data of esophageal cancer from The Cancer Genome Atlas (TCGA) and Gene-Expression Omnibus (GEO) databases. Unsupervised clustering analysis was used to classify patients into different groups. Gene set variation analysis (GSVA) was performed in a nonparametric and unsupervised mode. We evaluated immune cell infiltration by single sample gene set enrichment analysis (ssGSEA). Differentially expressed genes (DEGs) among m6A clusters were identified using Empirical Bayesian approach. Both multivariate and univariate Cox regression models were used for prognostic analysis. We provided an overview of gene variation and expression of 23 m6A regulators in esophageal cancer, as well as their effects on survival. Based on the overall expression level of m6A regulators, patients were classified into three m6A clusters (A-C) with different immune cell infiltration abundance, gene expression signatures and prognosis. Among m6A clusters, we identified 206 DEGs, according to which patients were classified into 4 gene clusters (A-D). Quantitative m6A score was calculated for each patient based on those DEGs with significant impact on survival. The infiltration of all types of immune cells except type 2 T helper (Th2) cells were negatively correlated with m6A score. M6Acluster C exhibited the lowest m6A score, the most abundant immune cell infiltration, and the worst prognosis, suggesting an immune excluded phenotype. Consistently, gene cluster D with the lowest m6A score showed the worst prognosis. In short, patients with esophageal cancer showed different m6A modification patterns. Quantitative scoring indicated that patients with the lowest m6A score exhibited the most abundant immune cell infiltration and the poorest prognosis. This m6A scoring system is promising to assess m6A modification pattern, characterize immune infiltration and guide personalized treatment and prognostic prediction.

GAMNet: Global attention via multi‐scale context for depth estimation algorithm and application

AbstractDeep neural networks significantly enhance the accuracy of the stereo‐based disparity estimation. Some current methods suffer from inefficient use of the global context information, which will lead to the loss of structural details in ill‐posed areas. To this end, a novel stereo network GAMNet is designed, composed of three core components (GDA, MPF, DCA) for estimating the depth prediction in challenging real‐world environments. First, a lightweight attention module is presented, integrating the global semantic cues for every feature position across the channel and spatial dimensions. Next, the MPF module is constructed to fuse the diverse semantic and contextual information from different levels of the feature pyramid. Finally, cost volume is aggregated with a stacked encoder‐decoder composed of the DCA module and 3D convolutions, filtering the transmission of matching clues and capturing the rich global contexts. Substantial experiments conducted on KITTI 2012, KITTI 2015, SceneFlow, and Middlebury‐v3 datasets manifest that GAMNet surpasses preceding methods with contour‐preserving disparity predictions. In addition, the first 3D scene reconstruction linear evaluation strategy on spatial grasping points for the end‐to‐end stereo networks in an unsupervised mode is proposed, and it is deployed on the designed robot vision‐guided system. In application experiments, the method can produce densely high‐precision 3D reconstructions to implement the grasping task in complex real‐world scenes, and achieve excellent robust performance with competitive inference efficiency.

Constraint-Based Adversarial Networks for Unsupervised Abstract Text Summarization

Abstract text summarization is a classic sequence-to-sequence natural language generation task. In order to improve the quality of unsupervised abstract text summarization in unsupervised mode, we propose two constraints for training text summarization model, embedding space constraint and information ratio constraint. We construct a generative adversarial network with two discriminators based on these two constraints (TC-SUM-GAN). We use unsupervised and supervised methods to train the model in the experiment. Experimental results show that the ROUGE-1 value of the unsupervised TC-SUM-GAN increases by [Formula: see text] points compared with the basic model and at least 1.96 points compared with other comparative models. The ROUGE scores of the supervised TC-SUM-GAN are also improved. TC-SUM-GAN achieves very competitive results for the metrics of ROUGE-1 and ROUGE-2. In addition, the abstracts generated by our model are closer to those generated manually.

A deep generative framework with embedded vector arithmetic and classifier for sample generation, label transfer, and clustering of single-cell data

Multiple-source single-cell datasets have accumulated quickly and need computational methods to integrate and decompose into meaningful components. Here, we present inClust (integrated clustering), a flexible deep generative framework that enables embedding auxiliary information, latent space vector arithmetic, and clustering. All functional parts are relatively modular, independent in implementation but interrelated at runtime, resulting in an all-in general framework that could work in supervised, semi-supervised, or unsupervised mode. We show that inClust is superior to most data integration methods in benchmark datasets. Then, we demonstrate the capability of inClust in the tasks of conditional out-of-distribution generation in supervised mode, label transfer in semi-supervised mode, and spatial domain identification in unsupervised mode. In these examples, inClust could accurately express the effect of each covariate, distinguish the query-specific cell types, or segment spatial domains. The results support that inClust is an excellent general framework for multiple-task harmonization and data decomposition.

Decoding the physiological response of plants to stress using deep learning for forecasting crop loss due to abiotic, biotic, and climatic variables

This paper presents a simple method for detecting both biotic and abiotic stress in plants. Stress levels are measured based on the increase in nutrient uptake by plants as a mechanism of self-defense when under stress. A continuous electrical resistance measurement was used to estimate the rate of change of nutrients in agarose as the growth medium for Cicer arietinum (Chickpea) seeds. To determine the concentration of charge carriers in the growth medium, Drude’s model was used. For identifying anomalies and forecasting plant stress, two experiments were conducted and outliers were found in electrical resistance and relative changes in carrier concentration. Anomaly in the first iteration was detected by applying k-Nearest Neighbour, One Class Support Vector Machine and Local Outlier Factor in unsupervised mode on electrical resistance data. In the second iteration, the neural network-based Long Short Term Memory method was used on the relative change in the carrier concentration data. As a result of the change in resistance of growth media during stress, nutrient concentrations shifted by 35%, as previously reported. Farmers who cater to small communities around them and are most affected by local and global stress factors can use this method of forecasting.

Role of length-scale in machine learning based image analysis of ductile fracture surfaces

Recent advancements in machine learning (ML) techniques have opened up new opportunities for using image analysis to solve materials science and engineering problems. In this work, we have used an ML-based workflow to classify the fracture surfaces of dual-phase steels subjected to different stress states. This task is not straightforward, as the ductile fracture surfaces of many metallic materials exhibit similar features, such as dimples. The ML-based workflow uses a pre-trained convolution neural network in unsupervised mode to extract image features, which are then reduced in dimensionality using principal component analysis. Next, images are clustered and classified using K-Means and K-Nearest Neighbors algorithms, respectively. Our results show that the accuracy of the ML-based technique is sensitive to the length-scales of the fracture surface images, and the critical length-scale corresponding to the maximum accuracy depends on the typological categories being classified. A physical interpretation of the critical length-scales associated with the fracture surface images is provided through quantitative fracture surface roughness analysis. Our work demonstrates the potential of using unsupervised ML-based techniques for fractography of ductile materials, especially for typological classification. More importantly, it emphasizes the importance of length-scales in image analysis in materials science and engineering.

Deep image and feature prior algorithm based on U-ConformerNet structure

PurposeThe reconstruction performance of the deep image prior (DIP) approach is limited by the conventional convolutional layer structure and it is difficult to enhance its potential. In order to improve the quality of image reconstruction and suppress artifacts, we propose a DIP algorithm with better performance, and verify its superiority in the latest case. MethodsWe construct a new U-ConformerNet structure as the DIP algorithm's network, replacing the traditional convolutional layer-based U-net structure, and introduce the 'lpips' deep network based feature distance regularization method. Our algorithm can switch between supervised and unsupervised modes at will to meet different needs. ResultsThe reconstruction was performed on the low dose CT dataset (LoDoPaB). Our algorithm attained a PSNR of more than 35 dB under unsupervised conditions, and the PSNR under the supervised condition is greater than 36 dB. Both of which are better than the performance of the DIP-TV. Furthermore, the accuracy of this method is positively connected with the quality of the a priori image with the help of deep networks. In terms of noise eradication and artifact suppression, the DIP algorithm with U-ConformerNet structure outperforms the standard DIP method based on convolutional structure. ConclusionsIt is known by experimental verification that, in unsupervised mode, the algorithm improves the output PSNR by at least 2–3 dB when compared to the DIP-TV algorithm (proposed in 2020). In supervised mode, our algorithm approaches that of the state-of-the-art end-to-end deep learning algorithms.

Network intrusion detection system for DDoS attacks in ICS using deep autoencoders

Anomaly detection in industrial control and cyber-physical systems has gained much attention over the past years due to the increasing modernisation and exposure of industrial environments. Current dangers to the connected industry include the theft of industrial intellectual property, denial of service, or the compromise of cloud components; all of which might result in a cyber-attack across the operational network. However, most scientific work employs device logs, which necessitate substantial understanding and preprocessing before they can be used in anomaly detection. In this paper, we propose a network intrusion detection system (NIDS) architecture based on a deep autoencoder trained on network flow data, which has the advantage of not requiring prior knowledge of the network topology or its underlying architecture. Experimental results show that the proposed model can detect anomalies, caused by distributed denial of service attacks, providing a high detection rate and low false alarms, outperforming the state-of-the-art and a baseline model in an unsupervised learning environment. Furthermore, the deep autoencoder model can detect abnormal behaviour in legitimate devices after an attack. We also demonstrate the suitability of the proposed NIDS in a real industrial plant from the alimentary sector, analysing the false positive rate and the viability of the data generation, filtering and preprocessing procedure for a near real time scenario. The suggested NIDS architecture is a low-cost solution that uses only fifteen network-based features, requires minimal processing, operates in unsupervised mode, and is straightforward to deploy in real-world scenarios.

MESAC: Learning to Remove Mismatches via Maximizing the Expected Score of Sample Consensuses

Most learning-based methods require labelling the training data, which is time-consuming and gives rise to wrong labels. To address the labelling issues thoroughly, we propose an unsupervised learning framework to remove mismatches by maximizing the expected score of sample consensuses (MESAC). The proposed MESAC can train various permutation invariant networks (PINs) based on training data with no labels, and has three distinct merits: 1) the framework can train various PINs in an unsupervised mode such that these are immune to wrong labels; 2) the gradients of the expected score are explicitly calculated by a revised score-function estimator, which can avoid gradient explosion; 3) the distribution of matching probabilities is learned from the PIN and precisely modelled by a categorical distribution, which can decrease the sampling times and improve the computational efficiency accordingly. Experiments of testing datasets disclose that mean recall is increased by at most 77% when pure PINs are embedded in MESAC, and mean precision is also improved by 16%. Applications in pose recovery indicate that the success rates of MESAC-integrated PINs outperform the compared methods when training with neither matching labels nor ground truth epipolar geometry (EG) constraints, showing the great potential of MESAC in mismatch removal.

Semi-supervised underwater acoustic source localization based on residual convolutional autoencoder

Passive localization of underwater targets was a thorny problem in underwater acoustics. For traditional model-driven passive localization methods, the main challenges are the inevitable environmental mismatch and the presence of interference and noise everywhere. In recent years, data-driven machine learning approaches have opened up new possibilities for passive localization of underwater acoustics. However, the acquisition and processing of underwater acoustics data are more restricted than other scenarios, and the lack of data is one of the most enormous difficulties in the application of machine learning to underwater acoustics. To take full advantage of the relatively easy accessed unlabeled data, this paper proposes a framework for underwater acoustic source localization based on a two-step semi-supervised learning classification model. The first step is trained in unsupervised mode with the whole available dataset (labeled and unlabeled dataset), and it consists of a convolutional autoencoder (CAE) for feature extraction and self-attention (RA) mechanism for picking more useful features by applying constraints on the CAE. The second step is trained in supervised mode with the labeled dataset, and it consists of a multilayer perceptron connected to an encoder from the first step and is used to perform the source location task. The proposed framework is validated on uniform vertical line array data of SWellEx-96 event S5. Compared with the supervised model and the model without the RA, the proposed framework maintains good localization performance with the reduced labeled dataset, and the proposed framework is more robust when the training dataset and the test dataset of the second step are distributed differently, which is called “data mismatch.”

Data-driven seismic prestack velocity inversion via combining residual network with convolutional autoencoder

Seismic full-waveform inversion (FWI) is a powerful imaging approach in exploration geophysics, to estimate high-resolution subsurface geophysical properties via fitting synthetic data to observed seismic records. Nevertheless, there exist several intractable issues, e.g., solution non-uniqueness, cycle-skipping phenomenon, weak robustness and high computational burden. Recently, more attention has been paid to data-driven FWI techniques based on deep neural networks. However, their success greatly depends on an ocean of training instances, which is not available in oil exploration conveniently. To cope with this situation, an effective strategy is to reduce the dimensionality of input and output spaces for given training instances. By exploiting 2D velocity profile nonlinear compressibility, we train a residual convolutional autoencoder under an unsupervised mode using velocity data, where its encoder acts as a velocity dimensionality reduction operator to transform 2D velocity profiles into low-dimensional 1D pseudo-velocity curves. Then, a residual network is trained to build a nonlinear mapping from multi-shot seismic records to 1D pseudo-velocity curves. In the inference stage, the trained decoder converts those predicted 1D curves back to 2D velocity profiles. Therefore, we propose a novel two-stage data-driven seismic inversion technique combining a residual network with a residual convolutional autoencoder. The detailed experiments on four synthetic data sets (salt body, layered, faulted and salt dome models) illustrate that our proposed method with three different residual network architectures outperforms other approaches including FCNVMB, VelocityGAN and pure residual network, according to four objective evaluation metrics and subjective visualization.

Enhancer-associated regulatory network and gene signature based on transcriptome and methylation data to predict the survival of patients with lung adenocarcinoma.

Accumulating evidence has proved that aberrant methylation of enhancers plays regulatory roles in gene expression for various cancers including lung adenocarcinoma (LUAD). In this study, the transcriptome and methylation data of The Cancer Genome Atlas (TCGA)-LUAD cohort were comprehensively analyzed with a five-step Enhancer Linking by Methylation/Expression Relationships (ELMER) process. Step 1: 131,371 distal (2 kb upstream from the transcription start site) probes were obtained. Step 2: 10,665 distal hypomethylated probes were identified in an unsupervised mode with the get.diff.meth function. Step 3: 699 probe-gene pairs with negative correlations were screened using the get.pair function in an unsupervised mode. Step 4: After mapping with probes, 768 motifs were obtained and 24 of them were enriched. Step 5: 127 transcription factors (TFs) with differential expressions and negative correlations with methylation levels were screened, which were corresponding to 21 motifs. After the ELMER process, a prognostic “TFs-motifs-genes” regulatory network was constructed. The Least absolute shrinkage and selection operator (LASSO) and Stepwise regression analyses were further applied to identify variables in the TCGA-LUAD cohort and an eight-gene signature was constructed for calculating the risk score. The risk score was verified in two independent validation cohorts. The area under curve values of receiver operating characteristic curves predicting 1-, 3-, and 5-years survival ranged from 0.633 to 0.764. With the increase of the risk scores, both the survival statuses and clinical traits showed a worse tendency. There were significant differences in the degrees of immune cell infiltration, TMB values, and TIDE scores between the high-risk and low-risk groups. Finally, a better-performing prognostic nomogram was integrated with the risk score and other clinical traits. In short, this multi-omics analysis demonstrated the application of ELMER in analyzing enhancer-associated regulatory network in LUAD, which provided promising strategies for epigenetic therapy and prognostic biomarkers.

Cybercrime: Identification and Prediction Using Machine Learning Techniques.

In the world of cyber age, cybercrime is spreading its root extensively. Supervised classification methods such as the support vector machine (SVM) and K-nearest neighbor (KNN) models are employed for the classification of cybercrime data. Likewise, the unsupervised mode of classification involves the techniques of K-means clustering, Gaussian mixture model, and cluster quasi-random via fuzzy C-means clustering and fuzzy clustering. Neural networks are employed for determining synthetic identity theft. The formation of clusters takes place using these clustering techniques, which fetches crime data from the overall data. Cybercrime detection employs dataset that is fetched from CBS open data StatLine. The attributes utilized are concerning the crime victims through personal characteristics with total user identity being 1000. For analyzing the performance, different training and testing data undergo variation. Eventually using the best technique, the criminal is identified and the Gaussian mixture model in the unsupervised method reveals enhanced performance using the detection method. 76.56% percentage of accuracy is achieved in detecting the criminal. The accuracy achieved in case of classification via SVM classifier is 89% in the supervised method. Performance metrics for several attributes are being computed in terms of true positive (TP), false positive (FP), true negative (TN), false negative (FN), false alarm rate (FAR), detection rate (DR), accuracy (ACC), recall, precision, specificity, sensitivity, and Fowlkes–Mallows scores. The expectation-maximization (EM) algorithm is employed for assessing the performance of the Gaussian mixture model.

Computer-Algebra-Software-Assisted Calculus Instruction, Not Calculus for Dummies: Bespoke Applications Necessitate Theory

Today, calculus frequently is taught with artificial intelligence in the form of computer algebra systems. Although these software packages may reduce tedium associated with the mechanics of calculus, they may be less effective if not supplemented by the accompanying teaching of calculus theory. This paper presents two examples from spatial statistics in which computer software in an unsupervised auto-execution mode fails, or can fail, to yield correct calculus results. Accordingly, it emphasizes the need to teach calculus theory when using software packages such as Mathematica and Maple.

Classification of Pointillist paintings using colour and texture features

Fine art paintings classification based on artistic style is a field of growing interest. Pointillist style is one of the most easily recognized painting styles by humans, due to its characteristic tiny detached paintbrushes of pure colour. In this paper automatic discrimination of artworks belonging to the style of Pointillism is investigated. The opposite styles considered are Cubism, Purism, Naïve art and Impressionism. Several colour and texture features are considered and a feature selection procedure is employed to reveal the most relevant ones to pointillist movement. Binary classification is performed, both in supervised and unsupervised mode, to assess the features’ discriminative ability. A small number of selected features is shown, by simulations results, to be quite powerful predictors resulting in a classification accuracy of 94% for a SVM classifier, 93.5% for a KNN classifier and 87% for a k-means classifier.

Granulated RCNN and Multi-Class Deep SORT for Multi-Object Detection and Tracking

In this article, two new models, namely granulated RCNN (G-RCNN) and multi-class deep SORT (MCD-SORT), for object detection and tracking, respectively from videos are developed. Object detection has two stages: object localization (region of interest RoI) and classification. G-RCNN is an improved version of the well-known Fast RCNN and Faster RCNN for extracting RoIs by incorporating the unique concept of granulation in a deep convolutional neural network. Granulation with spatio-temporal information enables more accurate extraction of RoIs (object regions) in unsupervised mode. Compared to Fast and Faster RCNNs, G-RCNN uses (i) granules (clusters) formed over the pooling feature map, instead of its all feature values, in defining RoIs, (ii) only the positive RoIs during training, instead of the whole RoI-map, (iii) videos directly as input, rather than static images, and (iv) only the objects in RoIs, instead of the entire feature map, for performing object classification. All these lead to the improvement in real-time detection speed and accuracy. MCD-SORT is an advanced form of the popular Deep SORT. In MCD-SORT, the searching for association of objects with trajectories is restricted only within the same categories. This increases the performance in multi-class tracking. These characteristic features have been demonstrated over 37 videos containing single-class, two-class, and multi-class objects. Superiority of the models over several state-of-the-art methodologies is also established extensively, both qualitatively and quantitatively.

Video Summarization Through Reinforcement Learning With a 3D Spatio-Temporal U-Net.

Intelligent video summarization algorithms allow to quickly convey the most relevant information in videos through the identification of the most essential and explanatory content while removing redundant video frames. In this paper, we introduce the 3DST-UNet-RL framework for video summarization. A 3D spatio-temporal U-Net is used to efficiently encode spatio-temporal information of the input videos for downstream reinforcement learning (RL). An RL agent learns from spatio-temporal latent scores and predicts actions for keeping or rejecting a video frame in a video summary. We investigate if real/inflated 3D spatio-temporal CNN features are better suited to learn representations from videos than commonly used 2D image features. Our framework can operate in both, a fully unsupervised mode and a supervised training mode. We analyse the impact of prescribed summary lengths and show experimental evidence for the effectiveness of 3DST-UNet-RL on two commonly used general video summarization benchmarks. We also applied our method on a medical video summarization task. The proposed video summarization method has the potential to save storage costs of ultrasound screening videos as well as to increase efficiency when browsing patient video data during retrospective analysis or audit without loosing essential information.

Constrained plasticity reserve as a natural way to control frequency and weights in spiking neural networks

Biological neurons have adaptive nature and perform complex computations involving the filtering of redundant information. However, most common neural cell models, including biologically plausible, such as Hodgkin–Huxley or Izhikevich, do not possess predictive dynamics on a single-cell level. Moreover, the modern rules of synaptic plasticity or interconnections weights adaptation also do not provide grounding for the ability of neurons to adapt to the ever-changing input signal intensity. While natural neuron synaptic growth is precisely controlled and restricted by protein supply and recycling, weight correction rules such as widely used STDP are efficiently unlimited in change rate and scale. The present article introduces new mechanics of interconnection between neuron firing rate homeostasis and weight change through STDP growth bounded by abstract protein reserve, controlled by the intracellular optimization algorithm. We show how these cellular dynamics help neurons filter out the intense noise signals to help neurons keep a stable firing rate. We also examine that such filtering does not affect the ability of neurons to recognize the correlated inputs in unsupervised mode. Such an approach might be used in the machine learning domain to improve the robustness of AI systems.