Semi-supervised Learning Techniques Research Articles

Bot-MGAT: A Transfer Learning Model Based on a Multi-View Graph Attention Network to Detect Social Bots

Twitter, as a popular social network, has been targeted by different bot attacks. Detecting social bots is a challenging task, due to their evolving capacity to avoid detection. Extensive research efforts have proposed different techniques and approaches to solving this problem. Due to the scarcity of recently updated labeled data, the performance of detection systems degrades when exposed to a new dataset. Therefore, semi-supervised learning (SSL) techniques can improve performance, using both labeled and unlabeled examples. In this paper, we propose a framework based on the multi-view graph attention mechanism using a transfer learning (TL) approach, to predict social bots. We called the framework ‘Bot-MGAT’, which stands for bot multi-view graph attention network. The framework used both labeled and unlabeled data. We used profile features to reduce the overheads of the feature engineering. We executed our experiments on a recent benchmark dataset that included representative samples of social bots with graph structural information and profile features only. We applied cross-validation to avoid uncertainty in the model’s performance. Bot-MGAT was evaluated using graph SSL techniques: single graph attention networks (GAT), graph convolutional networks (GCN), and relational graph convolutional networks (RGCN). We compared Bot-MGAT to related work in the field of bot detection. The results of Bot-MGAT with TL outperformed, with an accuracy score of 97.8%, an F1 score of 0.9842, and an MCC score of 0.9481.

Remote sensing image fusion on 3D scenarios: A review of applications for agriculture and forestry

Three-dimensional (3D) image mapping of real-world scenarios has a great potential to provide the user with a more accurate scene understanding. This will enable, among others, unsupervised automatic sampling of meaningful material classes from the target area for adaptive semi-supervised deep learning techniques. This path is already being taken by the recent and fast-developing research in computational fields, however, some issues related to computationally expensive processes in the integration of multi-source sensing data remain. Recent studies focused on Earth observation and characterization are enhanced by the proliferation of Unmanned Aerial Vehicles (UAV) and sensors able to capture massive datasets with a high spatial resolution. In this scope, many approaches have been presented for 3D modeling, remote sensing, image processing and mapping, and multi-source data fusion. This survey aims to present a summary of previous work according to the most relevant contributions for the reconstruction and analysis of 3D models of real scenarios using multispectral, thermal and hyperspectral imagery. Surveyed applications are focused on agriculture and forestry since these fields concentrate most applications and are widely studied. Many challenges are currently being overcome by recent methods based on the reconstruction of multi-sensorial 3D scenarios. In parallel, the processing of large image datasets has recently been accelerated by General-Purpose Graphics Processing Unit (GPGPU) approaches that are also summarized in this work. Finally, as a conclusion, some open issues and future research directions are presented.

Semi-Supervised Learning for Automatic Atrial Fibrillation Detection in 24-Hour Holter Monitoring.

Paroxysmal atrial fibrillation (AF) is generally diagnosed by long-term dynamic electrocardiogram (ECG) monitoring. Identifying AF episodes from long-term ECG data can place a heavy burden on clinicians. Many machine-learning-based automatic AF detection methods have been proposed to solve this issue. However, these methods require numerous annotated data to train the model, and the annotation of AF in long-term ECG is extremely time-consuming. Reducing the demand for labeled data can effectively improve the clinical practicability of automatic AF detection methods. In this study, we developed a novel semi-supervised learning method that generated modified low-entropy labels of unlabeled samples for training a deep learning model to automatically detect paroxysmal AF in 24 h Holter monitoring data. Our method employed a 1D CNN-LSTM neural network with RR intervals as input and used few labeled training data with numerous unlabeled data for training the neural network. This method was evaluated using a 24 h Holter monitoring dataset collected from 1000 paroxysmal AF patients. Using labeled samples from only 10 patients for model training, our method achieved a sensitivity of 97.8%, specificity of 97.9%, and accuracy of 97.9% in five-fold cross-validation. Compared to the supervised learning method with complete labeled samples, the detection accuracy of our method was only 0.5% lower, while the workload of data annotation was significantly reduced by more than 98%. In general, this is the first study to apply semi-supervised learning techniques for automatic AF detection using ECG. Our method can effectively reduce the demand for AF data annotations and can improve the clinical practicability of automatic AF detection.

SelfMatch: Robust semisupervised time‐series classification with self‐distillation

Over the years, a number of semisupervised deep-learning algorithms have been proposed for time-series classification (TSC). In semisupervised deep learning, from the point of view of representation hierarchy, semantic information extracted from lower levels is the basis of that extracted from higher levels. The authors wonder if high-level semantic information extracted is also helpful for capturing low-level semantic information. This paper studies this problem and proposes a robust semisupervised model with self-distillation (SD) that simplifies existing semisupervised learning (SSL) techniques for TSC, called SelfMatch. SelfMatch hybridizes supervised learning, unsupervised learning, and SD. In unsupervised learning, SelfMatch applies pseudolabeling to feature extraction on labeled data. A weakly augmented sequence is used as a target to guide the prediction of a Timecut-augmented version of the same sequence. SD promotes the knowledge flow from higher to lower levels, guiding the extraction of low-level semantic information. This paper designs a feature extractor for TSC, called ResNet–LSTMaN, responsible for feature and relation extraction. The experimental results show that SelfMatch achieves excellent SSL performance on 35 widely adopted UCR2018 data sets, compared with a number of state-of-the-art semisupervised and supervised algorithms.

Automated Geological Features Detection in 3D Seismic Data Using Semi-Supervised Learning

A geological interpretation plays an important role to gain information about the structural and stratigraphic of hydrocarbon reservoirs. However, this is a time-consuming task due to the complexity and size of seismic data. We propose a semi-supervised learning technique to automatically and accurately delineate the geological features from 3D seismic data. To generate labeling data for training the supervised Convolutional Neural Network (CNN) model, we propose an efficient workflow based on unsupervised learning. This workflow utilized seismic attributes and KernelPCA to enhance the visualization of geological targets and clustering the features into binary classes using K-means approach. With this workflow, we are able to develop a data-driven model and reduce human subjectivity. We applied this technique in two cases with different geological settings. The synthetic data and the real seismic investigation from the A Field in the Malay Basin. From this application, we demonstrate that our CNN-based model is highly accurate and consistent with the previous manual interpretation in both cases. In addition to qualitatively evaluating the interpretations, we further extract the predicted result into a 3D geobody. This result could help the interpreter focus on tasks requiring human expertise and aid the model’s prediction in the next studies.

Building machine learning assisted phase diagrams: Three chemically relevant examples

In this work, we present a systematic procedure to build phase diagrams for chemically relevant properties by the use of a semi-supervised machine learning technique called uncertainty sampling. Concretely, we focus on ground state spin multiplicity and chemical bonding properties. As a first step, we have obtained single-eutectic-point-containing solid–liquid systems that have been suitable for contrasting the validity of this approach. Once this was settled, on the one hand, we built magnetic phase diagrams for several Hooke atoms containing a few electrons (4 and 6) trapped in spheroidal harmonic potentials. Changing the parameters of the confinement potential, such as curvature and anisotropy, and interelectronic interaction strength, we have been able to obtain and rationalize magnetic phase transitions flipping the ground state spin multiplicity from singlet (nonmagnetic) to triplet (magnetic) states. On the other hand, Bader’s analysis is performed upon helium dimers confined by spherical harmonic potentials. Covalency is studied using descriptors as the sign for Δρ(rC) and H(rC), and the dependency on the degrees of freedom of the system is studied, i.e., potential curvature ω2 and interatomic distance R. As a result, we have observed that there may exist a covalent bond between He atoms for short enough distances and strong enough confinement. This machine learning procedure could, in principle, be applied to the study of other chemically relevant properties involving phase diagrams, saving a lot of computational resources.

A Semisupervised Learning Framework for Recognition and Classification of Defects in Transient Thermography Detection

The defect classification task is of great benefit to evaluating the safety performance of equipment and providing useful feedback information for discovering production process problems. In this article, we present a semisupervised learning (SSL) framework for transient thermography detection to employ the temporal and spatial information encoded into the three-dimensional transient thermal tensor data and provide pixel-level classification results for defect types. The time- and frequency-domain physical models for the transient thermal evolution of different kinds of defects are established to illustrate the theoretical foundation of defects classification based on transient thermography. The semisupervised multiclass Laplacian support vector machine is proposed to enable involving the abundant unlabeled data for enhancing learning performance in practical industrial applications where labeled samples are insufficient and labeling work is costly and laborious. A case study on silicone insulating materials with various types of artificial simulated internal defects validates the stronger generalized ability of the proposed method. This work, for the first time, proposes an SSL framework in transient thermography-based defect detection studies. It is believed that our proposed method is quite inspired for introducing SSL techniques to transient thermography for preferable performance in practical industrial applications.

ColloSSL

A major bottleneck in training robust Human-Activity Recognition models (HAR) is the need for large-scale labeled sensor datasets. Because labeling large amounts of sensor data is an expensive task, unsupervised and semi-supervised learning techniques have emerged that can learn good features from the data without requiring any labels. In this paper, we extend this line of research and present a novel technique called Collaborative Self-Supervised Learning (ColloSSL) which leverages unlabeled data collected from multiple devices worn by a user to learn high-quality features of the data. A key insight that underpins the design of ColloSSL is that unlabeled sensor datasets simultaneously captured by multiple devices can be viewed as natural transformations of each other, and leveraged to generate a supervisory signal for representation learning. We present three technical innovations to extend conventional self-supervised learning algorithms to a multi-device setting: a Device Selection approach which selects positive and negative devices to enable contrastive learning, a Contrastive Sampling algorithm which samples positive and negative examples in a multi-device setting, and a loss function called Multi-view Contrastive Loss which extends standard contrastive loss to a multi-device setting. Our experimental results on three multi-device datasets show that ColloSSL outperforms both fully-supervised and semi-supervised learning techniques in majority of the experiment settings, resulting in an absolute increase of upto 7.9% in F1 score compared to the best performing baselines. We also show that ColloSSL outperforms the fully-supervised methods in a low-data regime, by just using one-tenth of the available labeled data in the best case.

Bi-level artificial intelligence model for risk classification of acute respiratory diseases based on Chinese clinical data.

Objective: The high incidence of respiratory diseases has dramatically increased the medical burden under the COVID-19 pandemic in the year 2020. It is of considerable significance to utilize a new generation of information technology to improve the artificial intelligence level of respiratory disease diagnosis. Methods: Based on the semi-structured data of Chinese Electronic Medical Records (CEMRs) from the China Hospital Pharmacovigilance System, this paper proposed a bi-level artificial intelligence model for the risk classification of acute respiratory diseases. It includes two levels. The first level is a dedicated design of the “BiLSTM+Dilated Convolution+3D Attention+CRF” deep learning model that is used for Chinese Clinical Named Entity Recognition (CCNER) to extract valuable information from the unstructured data in the CEMRs. Incorporating the transfer learning and semi-supervised learning technique into the proposed deep learning model achieves higher accuracy and efficiency in the CCNER task than the popular “Bert+BiLSTM+CRF” approach. Combining the extracted entity data with other structured data in the CEMRs, the second level is a customized XGBoost to realize the risk classification of acute respiratory diseases. Results: The empirical study shows that the proposed model could provide practical technical support for improving diagnostic accuracy. Conclusion: Our study provides a proof-of-concept for implementing a hybrid artificial intelligence-based system as a tool to aid clinicians in tackling CEMR data and enhancing the diagnostic evaluation under diagnostic uncertainty.

Semi-supervised learning-assisted imaging method for electrical capacitance tomography

Despite the enormous potential of the electrical capacitance tomography technology for the process industry, one of the main technological gaps and obstacles that must be overcome is the low accuracy reconstruction. To address this technical challenge, the data-dependent prior is introduced in this study, which is combined with the electrical measurement mechanism and the domain knowledge to reshape the tomographic imaging problem into a new optimization problem. The introduction of the data-dependent prior not only bridges the physical model and the advanced semi-supervised learning technique, but also improves the adaptability of the reconstruction model. A new numerical scheme that integrates the merits of the half-quadratic splitting method and the forward-backward splitting technique with a speedup strategy is proposed to solve the challenging imaging model, leading to the reduced computational burden. A new robust sparse semi-supervised extreme learning machine method that leverages both labeled and unlabeled samples and effectively handles high-dimensional image data is developed to predict the data-dependent prior, and the training is recast into a new optimization problem that is solved by a new numerical method. The novel imaging method achieves the reconstruction paradigm shift and acts as a catalyst for improving imaging quality. The evaluation of a series of reconstruction cases validates that the novel method shows satisfactory performances, outperforming both popular and classical imaging methods in terms of robustness and reconstruction accuracy.

Analysis of Neural Machine Translation KANGRI Language by Unsupervised and Semi Supervised Methods

ABSTRACT It is very challenging to work with low resource languages pairs as monolingual and parallel dataset do not exist or exist in a very small amount. Furthermore, there is a lack of digitization of the available written resources. This work provides a comparison and analysis of the neural machine translation system for low resource definitely endangered, Kangri (ISO 639-3xnr) language using unsupervised and semi supervised methods. For this a shared encoder with back translation machine translation system for both unsupervised and semi-supervised learning techniques and a language model with denoising autoencoder that uses fully unsupervised learning technique has been used. Kangri which is an Indo-Aryan language has Devanagari () script same as Hindi. The translation task is further complicated by the fact that Kangri is a morphologically rich language, and it does not have well defined linguistic rules. To remove out of vocabulary problem we have used different technique and in finally, we have provided the comparison of results by taking the different evaluation metrics which shows that semi supervised translation with semi supervised cross lingual word embedding has highest score as compared to other translation models.

Making the most of small Software Engineering datasets with modern machine learning

This paper provides a starting point for Software Engineering (SE) researchers and practitioners faced with the problem of training machine learning models on small datasets. Due to the high costs associated with labeling data, in Software Engineering, there exist many small (< 5,000 samples) and medium-sized (<100,000 samples) datasets. While deep learning has set the state of the art in many machine learning tasks, it is only recently that it has proven effective on small-sized datasets, primarily thanks to pre-training, a semi-supervised learning technique that leverages abundant unlabelled data alongside scarce labelled data. In this work, we evaluate pre-trained Transformer models on a selection of 13 smaller datasets from the SE literature, covering both, source code and natural language. Our results suggest that pre-trained Transformers are competitive and in some cases superior to previous models, especially for tasks involving natural language; whereas for source code tasks, in particular for very small datasets, traditional machine learning methods often has the edge. In addition, we experiment with several techniques that ought to aid training on small datasets, including active learning, data augmentation, soft labels, self-training and intermediate-task fine-tuning, and issue recommendations on when they are effective. We also release all the data, scripts, and most importantly pre-trained models for the community to reuse on their own datasets.

Semi-Supervised Learning for Joint SAR and Multispectral Land Cover Classification

Semi-supervised learning techniques are gaining popularity due to their capability of building models that are effective, even when scarce amounts of labeled data are available. In this paper, we present a framework and specific tasks for self-supervised pretraining of \textit{multichannel} models, such as the fusion of multispectral and synthetic aperture radar images. We show that the proposed self-supervised approach is highly effective at learning features that correlate with the labels for land cover classification. This is enabled by an explicit design of pretraining tasks which promotes bridging the gaps between sensing modalities and exploiting the spectral characteristics of the input. In a semi-supervised setting, when limited labels are available, using the proposed self-supervised pretraining, followed by supervised finetuning for land cover classification with SAR and multispectral data, outperforms conventional approaches such as purely supervised learning, initialization from training on ImageNet and other recent self-supervised approaches.

Adversarial Neural Network Classifiers for COVID-19 Diagnosis in Ultrasound Images

The novel Coronavirus disease 2019 (COVID-19) pandemic has begun in China and is still affecting thousands of patient lives worldwide daily. Although Chest X-ray and Computed Tomography are the gold standard medical imaging modalities for diagnosing potentially infected COVID-19 cases, applying Ultrasound (US) imaging technique to accomplish this crucial diagnosing task has attracted many physicians recently. In this article, we propose two modified deep learning classifiers to identify COVID-19 and pneumonia diseases in US images, based on generative adversarial neural networks (GANs). The proposed image classifiers are a semi-supervised GAN and a modified GAN with auxiliary classifier. Each one includes a modified discriminator to identify the class of the US image using semi-supervised learning technique, keeping its main function of defining the “realness” of tested images. Extensive tests have been successfully conducted on public dataset of US images acquired with a convex US probe. This study demonstrated the feasibility of using chest US images with two GAN classifiers as a new radiological tool for clinical check of COVID-19 patients. The results of our proposed GAN models showed that high accuracy values above 91.0% were obtained under different sizes of limited training data, outperforming other deep learning-based methods, such as transfer learning models in the recent studies. Consequently, the clinical implementation of our computer-aided diagnosis of US-COVID-19 is the future work of this study.

Detecting Low-Credibility Medical Websites Through Semi-Supervised Learning Techniques

Detecting Low-Credibility Medical Websites Through Semi-Supervised Learning Techniques

Advance Genome Disorder Prediction Model Empowered With Deep Learning

A major and essential issue in biomedical research is to predict genome disorder. Genome disorders cause multivariate diseases like cancer, dementia, diabetes, cystic fibrosis, leigh syndrome, etc. which are causes of high mortality rates around the world. In past, theoretical and explanatory-based approaches were introduced to predict genome disorder. With the development of technology, genetic data were improved to cover almost genome and protein then machine and deep learning-based approaches were introduced to predict genome disorder. Parallel machine and deep learning approaches were introduced. In past, many types of research were conducted on genome disorder prediction using supervised, unsupervised, and semi-supervised learning techniques, most of the approaches using binary problem prediction using genetic sequence data. The prediction results of these approaches were uncertain because of their lower accuracy rate and binary class prediction techniques using genome sequence data but not genome disorder patients&#x2019; data with his/her history. Most of the techniques used Ribonucleic acid (RNA) gene sequence and were not often capable of handling bid data effectively. Consequently, in this study, the AlexNet as an effective convolutional neural network architecture proposed to develop an advance genome disorder prediction model (AGDPM) for predicting genome multi classes disorder using a large amount of data. AGDPM tested and compare with the pre-trained AlexNet neural network model and AGDPM gives the best results with 89.89&#x0025; &#x0026; 81.25&#x0025; accuracy of training and testing respectively. So, the advance genome disorder prediction model shows the ability to efficiently predict genome disorder and can process a large amount of patients&#x2019; genome disorder data with a multi-class prediction method. AGDPM has proved that it is capable to predict single gene inheritance disorder, mitochondrial gene inheritance disorder, and multifactorial gene inheritance disorder with respect to various statistical performance parameters. So, with the help of AGDPM biomedical research will be improved in terms to predict genetic disorders and put control on high mortality rates.

Threshold Filtering Semi-Supervised Learning Method for SAR Target Recognition

The semi-supervised deep learning technology driven by a small part of labeled data and a large amount of unlabeled data has achieved excellent performance in the field of image processing. However, the existing semi-supervised learning techniques are all carried out under the assumption that the labeled data and the unlabeled data are in the same distribution, and its performance is mainly due to the two being in the same distribution state. When there is out-of-class data in unlabeled data, its performance will be affected. In practical applications, it is difficult to ensure that unlabeled data does not contain out-of-category data, especially in the field of Synthetic Aperture Radar (SAR) image recognition. In order to solve the problem that the unlabeled data contains out-of-class data which affects the performance of the model, this paper proposes a semi-supervised learning method of threshold filtering. In the training process, through the two selections of data by the model, unlabeled data outside the category is filtered out to optimize the performance of the model. Experiments were conducted on the Moving and Stationary Target Acquisition and Recognition (MSTAR) dataset, and compared with existing several state-of-the-art semi-supervised classification approaches, the superiority of our method was confirmed, especially when the unlabeled data contained a large amount of out-of-category data.

Lifting the limitations of Gaussian mixture regression through coupling with principal component analysis and deep autoencoding

The mathematical modeling of correlations between target properties and their factors of influence, particularly that allowing inverse analysis, is an essential part of molecular, material, and process designs. In contrast to approaches employing pseudo-inverse analysis, Gaussian mixture regression (GMR), which assumes that the relationships between variables can be represented as a mixture of Gaussian distributions, allows for direct inverse analysis. However, as this model optimizes the means and variance–covariance matrices of all variables, parameter estimation becomes increasingly difficult with the increasing number of variables. Herein, this drawback is addressed by the transformation of explanatory variables X into latent variables Z before GMR modeling. As the inverse (Z to X) transformation is necessary for direct inverse analysis, principal component analysis (PCA) and deep autoencoding (DAE) are employed as dimensionality reduction methods. After X is transformed to Z with the help of PCA or DAE, a GMR model is constructed with Z and objective variables Y, and the proposed method is therefore denoted as PCA-GMR or DAE-GMR, respectively. As Z values can be predicted by inputting Y values into the GMR model and can be transformed to X values, direct inverse analysis is also possible. Given that unlabeled data can be employed to construct PCA and DAE models, the proposed methods can also be used as semi-supervised learning techniques. The predictive abilities of PCA-GMR and DAE-GMR are verified using molecular, material, and spectral datasets and surpass that of traditional GMR on all datasets, with the maximum reduction of prediction errors equaling 63%.

Semisupervised Deep Learning Techniques for Predicting Acute Respiratory Distress Syndrome From Time-Series Clinical Data: Model Development and Validation Study.

BackgroundA high number of patients who are hospitalized with COVID-19 develop acute respiratory distress syndrome (ARDS).ObjectiveIn response to the need for clinical decision support tools to help manage the next pandemic during the early stages (ie, when limited labeled data are present), we developed machine learning algorithms that use semisupervised learning (SSL) techniques to predict ARDS development in general and COVID-19 populations based on limited labeled data.MethodsSSL techniques were applied to 29,127 encounters with patients who were admitted to 7 US hospitals from May 1, 2019, to May 1, 2021. A recurrent neural network that used a time series of electronic health record data was applied to data that were collected when a patient’s peripheral oxygen saturation level fell below the normal range (<97%) to predict the subsequent development of ARDS during the remaining duration of patients’ hospital stay. Model performance was assessed with the area under the receiver operating characteristic curve and area under the precision recall curve of an external hold-out test set.ResultsFor the whole data set, the median time between the first peripheral oxygen saturation measurement of <97% and subsequent respiratory failure was 21 hours. The area under the receiver operating characteristic curve for predicting subsequent ARDS development was 0.73 when the model was trained on a labeled data set of 6930 patients, 0.78 when the model was trained on the labeled data set that had been augmented with the unlabeled data set of 16,173 patients by using SSL techniques, and 0.84 when the model was trained on the entire training set of 23,103 labeled patients.ConclusionsIn the context of using time-series inpatient data and a careful model training design, unlabeled data can be used to improve the performance of machine learning models when labeled data for predicting ARDS development are scarce or expensive.

Multi-Modal Pain Intensity Assessment Based on Physiological Signals: A Deep Learning Perspective.

Traditional pain assessment approaches ranging from self-reporting methods, to observational scales, rely on the ability of an individual to accurately assess and successfully report observed or experienced pain episodes. Automatic pain assessment tools are therefore more than desirable in cases where this specific ability is negatively affected by various psycho-physiological dispositions, as well as distinct physical traits such as in the case of professional athletes, who usually have a higher pain tolerance as regular individuals. Hence, several approaches have been proposed during the past decades for the implementation of an autonomous and effective pain assessment system. These approaches range from more conventional supervised and semi-supervised learning techniques applied on a set of carefully hand-designed feature representations, to deep neural networks applied on preprocessed signals. Some of the most prominent advantages of deep neural networks are the ability to automatically learn relevant features, as well as the inherent adaptability of trained deep neural networks to related inference tasks. Yet, some significant drawbacks such as requiring large amounts of data to train deep models and over-fitting remain. Both of these problems are especially relevant in pain intensity assessment, where labeled data is scarce and generalization is of utmost importance. In the following work we address these shortcomings by introducing several novel multi-modal deep learning approaches (characterized by specific supervised, as well as self-supervised learning techniques) for the assessment of pain intensity based on measurable bio-physiological data. While the proposed supervised deep learning approach is able to attain state-of-the-art inference performances, our self-supervised approach is able to significantly improve the data efficiency of the proposed architecture by automatically generating physiological data and simultaneously performing a fine-tuning of the architecture, which has been previously trained on a significantly smaller amount of data.