Unlabeled Dataset Research Articles

Active learning for cross-sentence n-ary relation extraction

N-ary relation extraction models are required to be trained on large amount of high-quality data, but it is challenging to obtain such data in effect; thus, models are forced to rely on a limited amount of low-quality labeled data. This paper proposes an active learning method for cross-sentence n-ary relation extraction, addressing the following question: “How can we train an n-ary relation extraction model incrementally without a large amount of high-quality data?” To answer this research question, we introduce a schema-aware sampling strategy that selects informative samples in unlabeled dataset that will be used for model training. This method exploits structural relatedness between a relation and its entities to generate the context embeddings of inferred relations. Using the similarity between the clusters of context embeddings and target samples, we detect a set of informative samples in unlabeled dataset. Moreover, the paper proposes a balanced incremental learning method updating the extraction model without bias with only a small computational cost at each training iteration. Experimental results on benchmark datasets are demonstrated to confirm the validity and effectiveness of the proposed method.

Generalizable and efficient cross‐domain person re‐identification model using deep metric learning

AbstractMost of the successful person re‐ID models conduct supervised training and need a large number of training data. These models fail to generalise well on unseen unlabelled testing sets. The authors aim to learn a generalisable person re‐identification model. The model uses one labelled source dataset and one unlabelled target dataset during training and generalises well on the target testing set. To this end, after a feature extraction by the ResNext‐50 network, the authors optimise the model by three loss functions. (a) One loss function is designed to learn the features of the target domain by tuning the distances between target images. Therefore, the trained model will be more robust to overcome the intra‐domain variations in the target domain and generalises well on the target testing set. (b) One triplet loss is used which considers both source and target domains and makes the model learn the inter‐domain variations between source and target domain as well as the variations in the target domain. (c) Also, one loss function is for supervised learning on the labelled source domain. Extensive experiments on Market1501 and DukeMTMC re‐ID show that the model achieves a very competitive performance compared with state‐of‐the‐art models and also it requires an acceptable amount of GPU RAM compared to other successful models.

A semi supervised approach to Arabic aspect category detection using Bert and teacher-student model.

Aspect-based sentiment analysis tasks are well researched in English. However, we find such research lacking in the context of the Arabic language, especially with reference to aspect category detection. Most of this research is focusing on supervised machine learning methods that require the use of large, labeled datasets. Therefore, the aim of this research is to implement a semi-supervised self-training approach which utilizes a noisy student framework to enhance the capability of a deep learning model, AraBERT v02. The objective is to perform aspect category detection on both the SemEval 2016 hotel review dataset and the Hotel Arabic-Reviews Dataset (HARD) 2016. The four-step framework firstly entails developing a teacher model that is trained on the aspect categories of the SemEval 2016 labeled dataset. Secondly, it generates pseudo labels for the unlabeled HARD dataset based on the teacher model. Thirdly, it creates a noisy student model that is trained on the combined datasets (∼1 million sentences). The aim is to minimize the combined cross entropy loss. Fourthly, an ensembling of both teacher and student models is carried out to enhance the performance of AraBERT. Findings indicate that the ensembled teacher-student model demonstrates a 0.3% improvement in its micro F1 over the initial noisy student implementation, both in predicting the Aspect Categories in the combined datasets. However, it has achieved a 1% increase over the micro F1 of the teacher model. These results outperform both baselines and other deep learning models discussed in the related literature.

Analysis of RNA-Seq data using self-supervised learning for vital status prediction of colorectal cancer patients

BackgroundRNA sequencing (RNA-Seq) is a technique that utilises the capabilities of next-generation sequencing to study a cellular transcriptome i.e., to determine the amount of RNA at a given time for a given biological sample. The advancement of RNA-Seq technology has resulted in a large volume of gene expression data for analysis.ResultsOur computational model (built on top of TabNet) is first pretrained on an unlabelled dataset of multiple types of adenomas and adenocarcinomas and later fine-tuned on the labelled dataset, showing promising results in the context of the estimation of the vital status of colorectal cancer patients. We achieve a final cross-validated (ROC-AUC) Score of 0.88 by using multiple modalities of data.ConclusionThe results of this study demonstrate that self-supervised learning methods pretrained on a vast corpus of unlabelled data outperform traditional supervised learning methods such as XGBoost, Neural Networks, and Decision Trees that have been prevalent in the tabular domain. The results of this study are further boosted by the inclusion of multiple modalities of data pertaining to the patients in question. We find that genes such as RBM3, GSPT1, MAD2L1, and others important to the computation model’s prediction task obtained through model interpretability corroborate with pathological evidence in current literature.

MTR-SDL: a soft computing based multi-tier rank model for shoulder X-ray classification.

Deep neural networks (DNN) effectiveness are contingent upon access to quality-labelled training datasets since label mistakes (label noise) in training datasets may significantly impair the accuracy of models trained on clean test data. The primary impediments to developing and using DNN models in the healthcare sector include the lack of sufficient label data. Labeling data by a domain expert are a costly and time-consuming task. To overcome this limitation, the proposed Multi-Tier Rank-based Semi-supervised deep learning (MTR-SDL) for Shoulder X-Ray Classification uses the small labelled dataset to generate a labelled dataset from unable dataset to obtain performance equivalent to approaches trained on the enormous dataset. The motivation behind the suggested model MTR-SDL approach is analogous to how physicians deal with unknown or suspicious patients in everyday life. Practitioners handle these questionable circumstances with the support of professional colleagues. Before initiating treatment, some patients consult with a range of skilled doctors. Patients are treated according to the most suitable professional diagnosis (vote count). In this article, we have proposed a new ensemble learning technique called "Rank based Ensemble Selection with machine learning models" (MTR-SDL) approach. In this technique, multiple machine learning models are trained on a labeled dataset, and their accuracy is ranked. A dynamic ensemble voting approach is then used to tag samples for each base model in the ensemble. The combination of these tags is used to generate a final tag for an unlabeled dataset. Our suggested MTR-SDL model has attained the best accuracy and specificity, sensitivity, precision, Matthew's correlation coefficient, false discovery rate, false positive rate, f1 score, negative predictive value, and false negative rate negative 92.776%, 97.376%, 86.932%, 96.192%, 85.644%, 3.808%, 2.624%, 91.072%, 90.85%, and 13.068% for unseen dataset, respectively. This approach has the potential to improve the performance of ensemble models by leveraging the strengths of multiple base models and selecting the most informative samples for each model. This study results in an improved Semi-supervised deep learning model that is more effective and precise.

Parameter-Efficient Methods for Metastases Detection fromClinical Notes

Understanding the progression of cancer is crucial for defining treatments for patients. The objective of this study is to automate the detection of metastatic liver disease from free-style computed tomography (CT) radiology reports. Our research demonstrates that transferring knowledge using three approaches can improve model performance. First, we utilize generic language models (LMs), pretrained in a self-supervised manner. Second, we use a semi-supervised approach to train our model by automatically annotating a large unlabeled dataset; this approach substantially enhances the model’s performance. Finally, we transfer knowledge from related tasks by designing a multi-task transfer learning methodology. We leverage the recent advancement of parameter-efficient LM adaptation strategies to improve performance and training efficiency. Our dataset consists of CT reports collected at Memorial Sloan Kettering Cancer Center (MSKCC) over the course of 12 years. 2,641 reports were manually annotated by domain experts; among them, 841 reports have been annotated for the presence of liver metastases. Our best model achieved an F1-score of 73.8%, a precision of 84%, and a recall of 65.8%.

An Adversarial Attack Analysis on Malicious Advertisement URL Detection Framework

Malicious advertisement URLs pose a security risk since they are the source of cyber-attacks, and the need to address this issue is growing in both industry and academia. Several attempts have been made in recent years for malicious URL detection using machine learning (ML). The most widely used techniques extract linguistic features of URL string to extract features like bag-of-words (BoW) before applying ML model. Existing malicious URL detection techniques require effective manual feature engineering that can handle unseen features and generalise to test data. In this study, we extract a novel set of lexical and web-scrapped features and employ ML techniques for fraudulent advertisement URL detection. The combination set of six different kinds of features precisely overcomes the obfuscation in fraudulent URL classification. Based on distinct statistical properties, we use twelve differently formatted datasets for detection, prediction and classification task. We extend our prediction analysis for mismatched and unlabelled datasets. For this framework, we analyze the performance of four ML techniques: Random Forest, Gradient Boost, XGBoost and AdaBoost in the detection part. With our proposed method, we achieve a false negative rate up to 0.0037 while maintaining high detection accuracy of 99.63%. Moreover, we employ an unsupervised learning technique for data clustering using the K-Means algorithm for the visual analysis. This paper analyses the vulnerability of decision tree-based models using the limited knowledge attack scenario. We considered the exploratory attack during the test phase and implemented Zeroth Order Optimization adversarial attack on the detection models.

Generalizable FGFR prediction across tumor types using self supervised learning.

e15057 Background: Deep Learning models have demonstrated the ability to detect tumors, classify disease state, or infer genetic biomarkers from digital whole slide images (WSI) of Hematoxylin and Eosin (H&E)-stained tissue. Such models can be deployed to improve clinical practice or efficiently recruit for clinical trials. However, the data to train these models is often very limited and image capture protocols vary between labs. We propose to use Self Supervised Learning (SSL) on large unlabeled histopathology data sets to improve generalizability and predictive performance of deep learning models on unseen data. Methods: We pre-trained a Convolutional Neural Network (CNN) using SSL on 25,120 unlabeled, digital WSIs from various sources (multiple scanners, hospital systems, labs, diseases, tissue sites). We then fine-tuned the pre-trained CNN to infer the presence or absence of select FGFR alterations in Muscle Invasive Bladder Cancer (MIBC) from the digital WSI. We applied this model (FGFRSSL) to WSIs of biopsies from other data sources and tumor types and compared with a model (FGFRTRAD) that was trained exclusively on MIBC samples without self-supervised learning. Results: The FGFRSSL model achieved an Area Under ROC Curve (AUC = 0.80) on MIBC WSIs and maintained performance on images from an independent lab (AUC=0.82). We further show that the model generalizes to Non-Muscle Invasive Bladder Cancer (NMIBC) samples with (AUC = 0.76) and Pan-Tumor WSIs (AUC: 0.83). The FGFRTRAD model trained without SSL-based pre-training achieved an Area Under ROC Curve (AUC: 0.76) on MIBC WSIs was less generalizable, showing lower performance for independent data (AUC = 0.72), NMIBC samples (AUC = 0.72), and Pan-Tumor samples (AUC = 0.64). Conclusions: We leveraged SSL pretraining to improve the reliability and generalizability of AI-based models across multiple data sources. We also demonstrate our model’s ability to infer FGFR status across multiple solid tumor types after training on only MIBC samples, suggesting the cell morphology conferred by FGFR alteration in cancer may be shared across diverse tumor types. These models represent a means for efficiently screening patients for actionable clinical biomarkers in a robust manner to guide clinical decisions and inform drug development efforts. [Table: see text]

QBox: Partial Transfer Learning With Active Querying for Object Detection.

Object detection requires plentiful data annotated with bounding boxes for model training. However, in many applications, it is difficult or even impossible to acquire a large set of labeled examples for the target task due to the privacy concern or lack of reliable annotators. On the other hand, due to the high-quality image search engines, such as Flickr and Google, it is relatively easy to obtain resource-rich unlabeled datasets, whose categories are a superset of those of target data. In this article, to improve the target model with cost-effective supervision from source data, we propose a partial transfer learning approach QBox to actively query labels for bounding boxes of source images. Specifically, we design two criteria, i.e., informativeness and transferability, to measure the potential utility of a bounding box for improving the target model. Based on these criteria, QBox actively queries the labels of the most useful boxes from the source domain and, thus, requires fewer training examples to save the labeling cost. Furthermore, the proposed query strategy allows annotators to simply labeling a specific region, instead of the whole image, and, thus, significantly reduces the labeling difficulty. Extensive experiments are performed on various partial transfer benchmarks and a real COVID-19 detection task. The results validate that QBox improves the detection accuracy with lower labeling cost compared to state-of-the-art query strategies for object detection.

Dual Instance-Consistent Network for Cross-Domain Object Detection.

Cross-domain object detection aims to transfer knowledge from a labeled dataset to an unlabeled dataset. Most existing methods apply a unified embedding model to generate the tightly coupled source and target descriptions for domain alignment, leading to the destroyed feature distribution of the target domain because the embedding model is mainly controlled by the source domain. To reduce the representation bias of the target domain, we apply two independent networks to extract two types of discriminative descriptions with mutual consistency, i.e., a novel Dual Instance-Consistent Network (DICN) is proposed for cross-domain object detection. Especially, Dual Instance-Consistent Module containing the instance mutual consistency between Primary Network and Auxiliary Network is applied to align two domains, where Primary and Auxiliary Networks are used to obtain the source-specific and target-specific information, respectively. The instance mutual consistency consists of two terms: feature consistency and detection consistency, which is applied to align the instance feature and the output of detection head, respectively. With the instance mutual consistency, optimizing the Primary (Auxiliary) Network only with source (target) images by fixing the Auxiliary (Primary) Network can generate the source(target)-specific description. Extensive experiments on several benchmarks demonstrate the effectiveness of the proposed DICN, e.g., obtaining mAP of 44.10% for Cityscapes → Foggy Cityscapes, AP on car of 76.50% for Cityscapes → KITTI, MR -2 of 8.87%, 12.66%, 22.27%, and 42.06% for COCOPersons → Caltech, CityPersons → Caltech, COCOPersons → CityPersons, and Caltech → CityPersons, respectively.

0950 Sound-based Sleep Staging at Home Using Smartphone via Deep Learning

Abstract Introduction Daily sleep tracking at home is growing in demand as more and more people are aware of the significance of sleep. The objective of this study is to propose a sound-based sleep staging model based on deep learning that works well in home environments with recorded audio data from general smartphones. Methods Three different audio datasets were used. A labeled hospital dataset (PSG and audio, N=812) and an unlabeled home dataset (audio only, N=829) were used for training. A limited number of labeled sound data from home (PSG and audio, N=45) were used for testing. Our proposed HomeSleepNet has three components: (1) supervised learning using the labeled hospital data that trains the model to make correct predictions in hospital environments; (2) unsupervised domain adaptation (UDA), which used both the labeled hospital data and unlabeled home data, and transferred the sleep staging power from hospital domain to home domain by adversarial training; (3) unsupervised data augmentation for consistency training (UDC), which augmented hospital data by adding home noise and trained the model to make consistent predictions on original and augmented data. After all training, HomeSleepNet is expected to make robust sleep staging despite the home noise presence. Results HomeSleepNet achieved 76.2% accuracy on the sleep staging task in home environments for the 3-stage classification case (Wake, NREM, REM). Specifically, it correctly predicted 63.4% of wake, 83.6% of NREM sleep, and 64.9% of REM sleep. The contributions of UDA and UDC were demonstrated by the following results. The accuracy of the model trained without both was 69.2%. Either addition of UDA or UDC training to the model improved the performance, with increased accuracy of 69.3% for UDA and 73.5% for UDC. As expected, using both UDA and UDC (i.e., HomeSleepNet) achieved the best performance, with a 7% increase in accuracy compared to the model trained without both components. Conclusion To the best of our knowledge, this is the first sound-based sleep staging study conducted in home environments. Moreover, the sounds were recorded by commercial smartphones and not through professional devices. Our proposed model introduced a reliable and convenient method for daily sleep tracking at home. Support (if any)

Multi-neuron connection using multi-terminal floating–gate memristor for unsupervised learning

Multi-terminal memristor and memtransistor (MT-MEMs) has successfully performed complex functions of heterosynaptic plasticity in synapse. However, theses MT-MEMs lack the ability to emulate membrane potential of neuron in multiple neuronal connections. Here, we demonstrate multi-neuron connection using a multi-terminal floating-gate memristor (MT-FGMEM). The variable Fermi level (EF) in graphene allows charging and discharging of MT-FGMEM using horizontally distant multiple electrodes. Our MT-FGMEM demonstrates high on/off ratio over 105 at 1000 s retention about ~10,000 times higher than other MT-MEMs. The linear behavior between current (ID) and floating gate potential (VFG) in triode region of MT-FGMEM allows for accurate spike integration at the neuron membrane. The MT-FGMEM fully mimics the temporal and spatial summation of multi-neuron connections based on leaky-integrate-and-fire (LIF) functionality. Our artificial neuron (150 pJ) significantly reduces the energy consumption by 100,000 times compared to conventional neurons based on silicon integrated circuits (11.7 μJ). By integrating neurons and synapses using MT-FGMEMs, a spiking neurosynaptic training and classification of directional lines functioned in visual area one (V1) is successfully emulated based on neuron’s LIF and synapse’s spike-timing-dependent plasticity (STDP) functions. Simulation of unsupervised learning based on our artificial neuron and synapse achieves a learning accuracy of 83.08% on the unlabeled MNIST handwritten dataset.

Targeted self-supervised attention network for coronavirus disease 2019 detection

The worldwide spread of the coronavirus disease 2019 (COVID-19) has had a significant impact on healthcare systems globally. Timely and accurate diagnosis is crucial for controlling and preventing outbreaks. Computed tomography (CT) has been demonstrated as an effective diagnostic tool for COVID-19. Deep learning techniques have been applied in related fields to solve relevant problems. Normally, they achieve good performance on large datasets directly. However, due to the need to protect patient privacy and the sudden outbreak of the epidemic, the amount of publicly available CT data is insufficient for utilizing these methods. In addition, some natural images, which are used for model pre-training, are quite different from chest CT images, causing some inevitable errors. To address these issues, a targeted self-supervised attention network based on an improved inception ResNet called SIRANet is proposed. SIRANet incorporates a self-supervised learning method during the pre-training process and introduces a attention module to improve feature selection and focus on key informative features. Moreover, a transfer learning-based strategy for COVID-19 detection is presented and experiments are conducted on three widely used datasets, including ImageNet, an unlabeled chest CT dataset, and a COVID-19 CT dataset. Experimental results show that SIRANet outperforms other state-of-the-art techniques in terms of accuracy, precision, recall, and f1-score, demonstrating its effectiveness in capturing key informative features from unlabeled and labeled images. The implementation of SIRANet is available at https://github.com/Xiiin6/COVID-19.

Unsupervised object discovery with pseudo label generated using K-means and self-supervised transformer

Instance segmentation is a fundamental task in the field of computer vision. It aims to assign every pixel to an appropriate class and localize objects within bounding boxes. However, expensive pixel-level segmentation labels are essential for training current state-of-the-art instance segmentation models. In this paper, we propose IMST, a simple method for generating instance-level pseudo-object mask labels without any form of human annotation. IMST leverages the fact that self-supervised transformers embed background and foreground into distinct cluster spaces. This characteristic allows us to discover class-agnostic object masks from unlabeled image datasets using cosine distance K-means clustering. We also present an object mask refinement method that employs ensemble results of K-means in a single image. Despite its simplicity, IMST achieves state-of-the-art performance in the unsupervised object mask discovery task. In unsupervised class-agnostic instance segmentation, IMST outperforms concurrent works by margins of 5.3 AP and 4.0 AP, respectively, on COCO20k and COCO val2017. Our proposed method can be extended to object box discovery tasks, such as unsupervised class-agnostic object detection and unsupervised single object discovery, surpassing previous works.

Deep learning based active learning technique for data annotation and improve the overall performance of classification models

Deep learning (DL) models achieved a lot of success due to the availability of labeled training data. In contrast, labeling a huge amount of data by a human is a time-consuming and expensive solution. Active Learning (AL) efficiently addresses the issue of labeled data collection at a low cost by picking the most useful samples from a large number of unlabeled datasets. However, current AL techniques largely depend on regular human involvement to annotate the most uncertain/informative samples in the collection. Therefore, we introduce a Deep Learning Based Active Learning (DLBAL) method that may incrementally learn from a small number of annotated training samples to build an effective classifier with optimum feature representation. The proposed DLBAL method improves the contemporary AL approaches in two steps. First, EfficientNet-B0 networks are integrated into AL. The incrementally annotated useful samples may be used to update both the representation of the feature and the classification algorithm concurrently. Second, we provide an effective technique for sample selection that improves the performance of the classifier with fewer manual annotations at a lower cost. In contrast to previous approaches that emphasize only uncertain instances with low prediction confidence, we obtain a significant number of instances with high-confidence from the unannotated set for pattern learning. Particularly, these instances are randomly picked and assigned labels to them continuously. Detailed experiments are conducted to show that the proposed DLBAL system obtained reasonable results on two difficult image classification datasets, namely Cross-Age Celebrity face recognition Dataset (CACD) and Caltech-256.

A novel density ratio‐based batch active learning fault diagnosis method integrated with adaptive Laplacian graph trimming

AbstractIn actual industrial processes, although a large number of original data are easy to obtain, only a few samples are effectively labelled, which is insufficient to construct a supervised fault diagnostic model. Facing the industrial demand of fault diagnosis, in this paper, a novel density ratio (DR)‐based batch active learning (BAL) fault diagnosis method integrated with adaptive Laplacian graph trimming (ALGT) method is proposed. First, under the active learning framework, a new index DR‐based on local reachability density (LRD) is proposed to search the low density and high uncertainty samples, in which the local outliers factor (LOF) is used to search the samples in low density region and the ratio of LRD and intra‐class LRD is calculated to search the samples with high uncertainty. Second, the samples are selected and manually labelled in batches according to the proposed index DR, and the labelled data set and the unlabelled data set are updated and reconstructed. Third, based on the reconstructed labelled dataset and remaining unlabelled dataset, a semi‐supervised classifier ALGT is constructed for fault diagnosis. In ALGT, the Laplacian weighted graph is initialized and iteratively optimized by ALGT. Finally, the proposed DR‐based BAL‐ALGT (DRBAL‐ALGT) fault diagnosis method is verified by the Tennessee Eastman process (TEP) and applied to grid‐connected photovoltaic systems (GPVS). The experimental results show that the proposed DRBAL‐ALGT method can achieve higher accuracy for fault diagnosis.

Detection of Pulmonary Embolism: Workflow Architecture and Comparative Analysis of the CNN Models

Machine learning has proven to be a practical medical image processing technique for pattern discovery in low-quality labelled and unlabeled datasets. Deep vein thrombosis and pulmonary embolism are both examples of venous thromboembolism, which is a key factor in patient mortality and necessitates prompt diagnosis by experts. An immediate diagnosis and course of treatment are necessary for the life-threatening cardiovascular condition known as pulmonary embolism (PE). In the study of medical imaging, especially the identification of PE, machine learning (ML) algorithms have produced encouraging results. This study's objective is to assess how well machine learning (ML) algorithms perform in identifying PE in computed tomography (CT) scans. A range of ML approaches were used to the dataset, including deep learning algorithms such as convolutional neural networks. The effectiveness of PE detection systems can be greatly enhanced by the use of cutting-edge methodologies like deep learning, which lowers the possibility of incorrect diagnoses and enables the quick administration of therapy to individuals who require it. This work contributes to the growing body of evidence that supports the use of ML in medical imaging and diagnosis. Future research should examine how these algorithms might be included into clinical workflows, resolving any potential implementation challenges, and making sure their adoption is done so in a secure and efficient way. In this study, we provide a thorough evaluation of three different models: the streamlined architecture MobileNetV2 with an accuracy of 96%, compared to other models like the Xception model with an accuracy of 91%, and the Efficientnet B5 model with an accuracy of 97%, after observation and process following.

DeepAstroUDA: semi-supervised universal domain adaptation for cross-survey galaxy morphology classification and anomaly detection

Artificial intelligence methods show great promise in increasing the quality and speed of work with large astronomical datasets, but the high complexity of these methods leads to the extraction of dataset-specific, non-robust features. Therefore, such methods do not generalize well across multiple datasets. We present a universal domain adaptation method, DeepAstroUDA, as an approach to overcome this challenge. This algorithm performs semi-supervised domain adaptation (DA) and can be applied to datasets with different data distributions and class overlaps. Non-overlapping classes can be present in any of the two datasets (the labeled source domain, or the unlabeled target domain), and the method can even be used in the presence of unknown classes. We apply our method to three examples of galaxy morphology classification tasks of different complexities (three-class and ten-class problems), with anomaly detection: (1) datasets created after different numbers of observing years from a single survey (Legacy Survey of Space and Time mock data of one and ten years of observations); (2) data from different surveys (Sloan Digital Sky Survey (SDSS) and DECaLS); and (3) data from observing fields with different depths within one survey (wide field and Stripe 82 deep field of SDSS). For the first time, we demonstrate the successful use of DA between very discrepant observational datasets. DeepAstroUDA is capable of bridging the gap between two astronomical surveys, increasing classification accuracy in both domains (up to on the unlabeled data), and making model performance consistent across datasets. Furthermore, our method also performs well as an anomaly detection algorithm and successfully clusters unknown class samples even in the unlabeled target dataset.

TransPolymer: a Transformer-based language model for polymer property predictions

Accurate and efficient prediction of polymer properties is of great significance in polymer design. Conventionally, expensive and time-consuming experiments or simulations are required to evaluate polymer functions. Recently, Transformer models, equipped with self-attention mechanisms, have exhibited superior performance in natural language processing. However, such methods have not been investigated in polymer sciences. Herein, we report TransPolymer, a Transformer-based language model for polymer property prediction. Our proposed polymer tokenizer with chemical awareness enables learning representations from polymer sequences. Rigorous experiments on ten polymer property prediction benchmarks demonstrate the superior performance of TransPolymer. Moreover, we show that TransPolymer benefits from pretraining on large unlabeled dataset via Masked Language Modeling. Experimental results further manifest the important role of self-attention in modeling polymer sequences. We highlight this model as a promising computational tool for promoting rational polymer design and understanding structure-property relationships from a data science view.

A semi-supervised material quantitative intelligent imaging algorithm for spectral CT based on prior information perception learning

To propose a semi-supervised material quantitative intelligent imaging algorithm based on prior information perception learning (SLMD-Net) to improve the quality and precision of spectral CT imaging. The algorithm includes a supervised and a self- supervised submodule. In the supervised submodule, the mapping relationship between low and high signal-to-noise ratio (SNR) data was constructed through mean square error loss function learning based on a small labeled dataset. In the self- supervised sub-module, an image recovery model was utilized to construct the loss function incorporating the prior information from a large unlabeled low SNR basic material image dataset, and the total variation (TV) model was used to to characterize the prior information of the images. The two submodules were combined to form the SLMD-Net method, and pre-clinical simulation data were used to validate the feasibility and effectiveness of the algorithm. Compared with the traditional model-driven quantitative imaging methods (FBP-DI, PWLS-PCG, and E3DTV), data-driven supervised-learning-based quantitative imaging methods (SUMD-Net and BFCNN), a material quantitative imaging method based on unsupervised learning (UNTV-Net) and semi-supervised learning-based cycle consistent generative adversarial network (Semi-CycleGAN), the proposed SLMD-Net method had better performance in both visual and quantitative assessments. For quantitative imaging of water and bone materials, the SLMD-Net method had the highest PSNR index (31.82 and 29.06), the highest FSIM index (0.95 and 0.90), and the lowest RMSE index (0.03 and 0.02), respectively) and achieved significantly higher image quality scores than the other 7 material decomposition methods (P < 0.05). The material quantitative imaging performance of SLMD-Net was close to that of the supervised network SUMD-Net trained with labeled data with a doubled size. A small labeled dataset and a large unlabeled low SNR material image dataset can be fully used to suppress noise amplification and artifacts in basic material decomposition in spectral CT and reduce the dependence on labeled data-driven network, which considers more realistic scenario in clinics.