Pre-training Approach Research Articles

Mitigating bias in artificial intelligence: Fair data generation via causal models for transparent and explainable decision-making

In the evolving field of Artificial Intelligence, concerns have arisen about the opacity of certain models and their potential biases. This study aims to improve fairness and explainability in AI decision making. Existing bias mitigation strategies are classified as pre-training, training, and post-training approaches. This paper proposes a novel technique to create a mitigated bias dataset. This is achieved using a mitigated causal model that adjusts cause-and-effect relationships and probabilities within a Bayesian network. Contributions of this work include (1) the introduction of a novel mitigation training algorithm for causal model; (2) a pioneering pretraining methodology for producing a fair dataset for Artificial Intelligence model training; (3) the diligent maintenance of sensitive features in the dataset, ensuring that these vital attributes are not overlooked during analysis and model training; (4) the enhancement of explainability and transparency around biases; and finally (5) the development of an interactive demonstration that vividly displays experimental results and provides the code for facilitating replication of the work.

A Dual-Stage Semi-Supervised Pre-Training Approach for Medical Image Segmentation

Deep neural networks have played a vital role in developing automated methods for addressing medical image segmentation. However, their reliance on labeled data impedes the practicability. Semi-supervised learning is gaining attention for its intrinsic ability to extract valuable information from labeled and unlabeled data with improved performance. Recently, consistency regularization methods have gained interest due to their efficient learning procedures. They are, however, confined to data or network-level perturbations, negating the benefit of having both forms in a single framework. In light of this, we ask an intriguing but unexplored question: can we have both network-level and data-level perturbation in the semi-supervised framework? To this end, we present a holistic approach that integrates data-level perturbation in the model pre-training stage, followed by implicit network-level perturbation in the fine-tuning stage. Furthermore, we incorporate networks with manifold learning paradigms throughout the training to facilitate the formation of robust data representations by ensuring local and global semantic affinities adhering to the theory of consensus. Notably, this may be the first attempt in the semi-supervised medical image segmentation archetype to use data and network-level perturbation with a model pre-training strategy. We extensively validated the efficacy of the proposed framework on three benchmark datasets, namely the Automated Cardiac Diagnosis Challenge, ISIC-2018, and Left Atrial Segmentation Challenge datasets, subjected to severely low-sampled labeled data. Notably, in ACDC (4%), ISIC-2018 (5%), and LA (6%) labeled cases, the proposed method outperforms the second-best method by 2.95%, 1.31%, and 0.71% in the Dice Similarity Metric.

Biometric contrastive learning for data-efficient deep learning from electrocardiographic images.

Artificial intelligence (AI) detects heart disease from images of electrocardiograms (ECGs). However, traditional supervised learning is limited by the need for large amounts of labeled data. We report the development of Biometric Contrastive Learning (BCL), a self-supervised pretraining approach for label-efficient deep learning on ECG images. Using pairs of ECGs from 78 288 individuals from Yale (2000-2015), we trained a convolutional neural network to identify temporally separated ECG pairs that varied in layouts from the same patient. We fine-tuned BCL-pretrained models to detect atrial fibrillation (AF), gender, and LVEF < 40%, using ECGs from 2015 to 2021. We externally tested the models in cohorts from Germany and the United States. We compared BCL with ImageNet initialization and general-purpose self-supervised contrastive learning for images (simCLR). While with 100% labeled training data, BCL performed similarly to other approaches for detecting AF/Gender/LVEF < 40% with an AUROC of 0.98/0.90/0.90 in the held-out test sets, it consistently outperformed other methods with smaller proportions of labeled data, reaching equivalent performance at 50% of data. With 0.1% data, BCL achieved AUROC of 0.88/0.79/0.75, compared with 0.51/0.52/0.60 (ImageNet) and 0.61/0.53/0.49 (simCLR). In external validation, BCL outperformed other methods even at 100% labeled training data, with an AUROC of 0.88/0.88 for Gender and LVEF < 40% compared with 0.83/0.83 (ImageNet) and 0.84/0.83 (simCLR). A pretraining strategy that leverages biometric signatures of different ECGs from the same patient enhances the efficiency of developing AI models for ECG images. This represents a major advance in detecting disorders from ECG images with limited labeled data.

LEBERT :Lite and Efficiently Optimized BERT PRetraining Approach

The extensive generalization of these models can lead to overfitting, causing the model to perform poorly on unseen data and thereby not realizing its full potential. To address this challenge systematically, we propose a novel approach for lightweight and efficient fine-tuning of BERT (Bidirectional Encoder Representations from Transformers) that aims to achieve improved generalization and harness the maximum capabilities of BERT. Our proposed approach incorporates various regularization techniques designed to adaptively manage the model's complexity. We plan to conduct experiments using this approach across various NLP tasks, including GLUE (Wang et al., 2019), RACE (Lai et al., 2017), and SQuAD (Rajpurkar et al., 2016).

Intelligent Recognition of Coastal Outfall Drainage Based on Sentinel-2/MSI Imagery

In this study, we developed an innovative and self-supervised pretraining approach using Sentinel-2/MSI satellite imagery specifically designed for the intelligent identification of drainage at sea discharge outlets. By integrating the geographical information from remote sensing images into our proposed methodology, we surpassed the classification accuracy of conventional models, such as MoCo (momentum contrast) and BYOL (bootstrap your own latent). Using Sentinel-2/MSI remote sensing imagery, we developed our model through an unsupervised dataset comprising 25,600 images. The model was further refined using a supervised dataset composed of 1100 images. After supervised fine-tuning, the resulting framework yielded an adept model that was capable of classifying outfall drainage with an accuracy rate of 90.54%, facilitating extensive outfall monitoring. A series of ablation experiments affirmed the effectiveness of our enhancement of the training framework, showing a 10.81% improvement in accuracy compared to traditional models. Furthermore, the authenticity of the learned features was further validated using visualization techniques. This study contributes an efficient approach to large-scale monitoring of coastal outfalls, with implications for augmenting environmental protection measures and reducing manual inspection efforts.

A robust deep learning detector for sleep spindles and K-complexes: towards population norms

Sleep spindles (SSs) and K-complexes (KCs) are brain patterns involved in cognitive functions that appear during sleep. Large-scale sleep studies would benefit from precise and robust automatic sleep event detectors, capable of adapting the variability in both electroencephalography (EEG) signals and expert annotation rules. We introduce the Sleep EEG Event Detector (SEED), a deep learning system that outperforms existing approaches in SS and KC detection, reaching an F1-score of 80.5% and 83.7%, respectively, on the MASS2 dataset. SEED transfers well and requires minimal fine-tuning for new datasets and annotation styles. Remarkably, SEED substantially reduces the required amount of annotated data by using a novel pretraining approach that leverages the rule-based detector A7. An analysis of 11,224 subjects revealed that SEED's detections provide better estimates of SS population statistics than existing approaches. SEED is a powerful resource for obtaining sleep-event statistics that could be useful for establishing population norms.

Fusion of Topic Modeling and RoBERTa for Detecting Signs of Depression from Social Media

Depression, or Major Depressive Disorder, is a serious and common medical condition that affects people worldwide. It negatively affects the person's feelings, thoughts, and actions. Depression causes a loss of interest in activities he enjoyed in the past. It can lead to physical and emotional problems that hamper the daily activities at work and home. In recent years, much research has been done to identify Depression through various modalities of image, speech, and text through artificial intelligence. Social media is an important medium where many discussions and mentions happen about Depression. The current study proposes a novel approach to understand how the depressed and non-depressed communicate differently with the help of Topic Modeling with latent-Dirichlet allocation (LDA) and also detect depression with the help of Robustly Optimized BERT Pretraining Approach (RoBERTa). The current study achieved an accuracy of 66.4% for the depression detection model, which outperformed the previous approaches with similar methodology. The current study is helpful for self-diagnosis of signs of Depression at very early stages.

Tower Masking MIM: A Self-Supervised Pretraining Method for Power Line Inspection

For intelligent inspection of power lines, a core task is to detect components in aerial images. Currently, deep supervised learning, a data-hungry paradigm, has attracted great attention. However, considering real-world scenarios, labeled data are usually limited, and the utilization of abundant unlabeled data is rarely investigated in this field. This study deploys a pretrained model for power line component detection based on a self-supervised pretraining approach, which exploits useful information from unannotated data. Concretely, we design a new masking strategy based on the structural characteristic of power lines to guide the pretraining process with meaningful semantic content. Meanwhile, a Siamese architecture is proposed to extract complete global features by using dual reconstruction with semantic targets provided by the proposed masking strategy. Then, the knowledge distillation is utilized to enable the pretrained model to learn both domain-specific and general representations. Moreover, a feature pyramid mechanism is adopted to capture multi-scale features, which can benefit the detection task. Experimental results show that the proposed approach can successfully improve the performance of a variety detection frameworks for power line components, and outperforms other self-supervised pretraining methods.

Masked Modeling-Based Ultrasound Image Classification via Self-Supervised Learning.

Recently, deep learning-based methods have emerged as the preferred approach for ultrasound data analysis. However, these methods often require large-scale annotated datasets for training deep models, which are not readily available in practical scenarios. Additionally, the presence of speckle noise and other imaging artifacts can introduce numerous hard examples for ultrasound data classification. In this paper, drawing inspiration from self-supervised learning techniques, we present a pre-training method based on mask modeling specifically designed for ultrasound data. Our study investigates three different mask modeling strategies: random masking, vertical masking, and horizontal masking. By employing these strategies, our pre-training approach aims to predict the masked portion of the ultrasound images. Notably, our method does not rely on externally labeled data, allowing us to extract representative features without the need for human annotation. Consequently, we can leverage unlabeled datasets for pre-training. Furthermore, to address the challenges posed by hard samples in ultrasound data, we propose a novel hard sample mining strategy. To evaluate the effectiveness of our proposed method, we conduct experiments on two datasets. The experimental results demonstrate that our approach outperforms other state-of-the-art methods in ultrasound image classification. This indicates the superiority of our pre-training method and its ability to extract discriminative features from ultrasound data, even in the presence of hard examples.

Improving First-stage Retrieval of Point-of-interest Search by Pre-training Models

Point-of-interest (POI) search is important for location-based services, such as navigation and online ride-hailing service. The goal of POI search is to find the most relevant destinations from a large-scale POI database given a text query. To improve the effectiveness and efficiency of POI search, most existing approaches are based on a multi-stage pipeline that consists of an efficiency-oriented retrieval stage and one or more effectiveness-oriented re-rank stages. In this article, we focus on the first efficiency-oriented retrieval stage of the POI search. We first identify the limitations of existing first-stage POI retrieval models in capturing the semantic-geography relationship and modeling the fine-grained geographical context information. Then, we propose a Geo-Enhanced Dense Retrieval framework for POI search to alleviate the above problems. Specifically, the proposed framework leverages the capacity of pre-trained language models (e.g., BERT) and designs a pre-training approach to better model the semantic match between the query prefix and POIs. With the POI collection, we first perform a token-level pre-training task based on a geographical-sensitive masked language prediction and design two retrieval-oriented pre-training tasks that link the address of each POI to its name and geo-location. With the user behavior logs collected from an online POI search system, we design two additional pre-training tasks based on users’ query reformulation behavior and the transitions between POIs. We also utilize a late-interaction network structure to model the fine-grained interactions between the text and geographical context information within an acceptable query latency. Extensive experiments on the real-world datasets collected from the Didichuxing application demonstrate that the proposed framework can achieve superior retrieval performance over existing first-stage POI retrieval methods.

Rethinking Person Re-Identification via Semantic-based Pretraining

Pretraining is a dominant paradigm in computer vision. Generally, supervised ImageNet pretraining is commonly used to initialize the backbones of person re-identification (Re-ID) models. However, recent works show a surprising result that CNN-based pretraining on ImageNet has limited impacts on Re-ID system due to the large domain gap between ImageNet and person Re-ID data. To seek an alternative to traditional pretraining, here we investigate semantic-based pretraining as another method to utilize additional textual data against ImageNet pretraining. Specifically, we manually construct a diversified FineGPR-C caption dataset for the first time on person Re-ID events. Based on it, a pure semantic-based pretraining approach named VTBR is proposed to adopt dense captions to learn visual representations with fewer images. We train convolutional neural networks from scratch on the captions of FineGPR-C dataset, and then transfer them to downstream Re-ID tasks. Comprehensive experiments conducted on benchmark datasets show that our VTBR can achieve competitive performance compared with ImageNet pretraining—despite using up to 1.4× fewer images, revealing its potential in Re-ID pretraining. Our source code is also publicly available at https://github.com/JeremyXSC/VTBR .

Social Media as an Instant Source of Feedback on Water Quality

This paper focuses on an important environmental challenge: the measurement of water quality, by analyzing the potential of social media to be harnessed as an immediate source of feedback. The goal of the work is to automatically analyze and retrieve social media posts relevant to water quality, with particular attention to posts describing different aspects of water quality, such as color, smell, taste, and water-related illnesses. To this aim, we propose a novel framework incorporating different preprocessing, data augmentation, and classification techniques. We use three Neural Network (NN) architectures for our framework, namely (i) Bidirectional Encoder Representations from Transformers (BERT), (ii) Robustly Optimized BERT Pretraining Approach (XLM-RoBERTa), and (iii) a custom Long short-term memory (LSTM) model. These are employed in a merit-based fusion scheme. For merit-based weight assignment to the models, several optimization and search techniques are compared including a Particle Swarm Optimization (PSO), a Genetic Algorithm (GA), Brute Force (BF), Nelder-Mead, and Powell&#x2019;s optimization methods. We also provide an evaluation of the individual models where the highest F1-score of 0.81 is obtained with the BERT model. Overall, in merit-based fusion, better results are obtained with BF achieving an F1-score score of 0.852. We also provide a comparison against existing methods, where a significant improvement for our proposed solutions is obtained. We believe such a rigorous analysis of this relatively new topic will provide a baseline for future research.

Towards real-world traffic prediction and data imputation: A multi-task pretraining and fine-tuning approach

Real-world traffic prediction is challenging and requires accuracy, efficiency, and generalizability for applications. Most studies used two-step data imputation-prediction procedures or recurrent networks to perform predictions based on imputed data, resulting in error accumulation. Training the same model independently for different missing rates and missing data patterns requires high computational costs. This strategy cannot be integrated with existing methods, causing low generalizability. We propose a multi-task pretraining and fine-tuning (MTPF) approach to address these issues. Specifically, we pretrain an encoder and decoders with different missing data patterns and missing rates and an adversarial noise to learn the correlations between different scenarios and generate robust hidden representations. The encoder and decoder perform fine-tuning for specific missing data scenarios after pretraining. Multiple parallel tasks, including prediction, data imputation, contrastive learning, and adaptive graph learning, are used for pretraining to enable the model to learn the hidden representation for prediction and data imputation during fine-tuning. The MTPF is compared with nine baseline models for prediction and five baseline models for data imputation using two real-world datasets. The experimental results show that MTPF outperforms many state-of-the-art baseline models, has a shorter training time, and is a model-agnostic off-the-shelf plug-in that improves baseline model performance.

Self-supervised pre-training with contrastive and masked autoencoder methods for dealing with small datasets in deep learning for medical imaging

Deep learning in medical imaging has the potential to minimize the risk of diagnostic errors, reduce radiologist workload, and accelerate diagnosis. Training such deep learning models requires large and accurate datasets, with annotations for all training samples. However, in the medical imaging domain, annotated datasets for specific tasks are often small due to the high complexity of annotations, limited access, or the rarity of diseases. To address this challenge, deep learning models can be pre-trained on large image datasets without annotations using methods from the field of self-supervised learning. After pre-training, small annotated datasets are sufficient to fine-tune the models for a specific task. The most popular self-supervised pre-training approaches in medical imaging are based on contrastive learning. However, recent studies in natural image processing indicate a strong potential for masked autoencoder approaches. Our work compares state-of-the-art contrastive learning methods with the recently introduced masked autoencoder approach “SparK” for convolutional neural networks (CNNs) on medical images. Therefore, we pre-train on a large unannotated CT image dataset and fine-tune on several CT classification tasks. Due to the challenge of obtaining sufficient annotated training data in medical imaging, it is of particular interest to evaluate how the self-supervised pre-training methods perform when fine-tuning on small datasets. By experimenting with gradually reducing the training dataset size for fine-tuning, we find that the reduction has different effects depending on the type of pre-training chosen. The SparK pre-training method is more robust to the training dataset size than the contrastive methods. Based on our results, we propose the SparK pre-training for medical imaging tasks with only small annotated datasets.

Improving the classification of veterinary thoracic radiographs through inter-species and inter-pathology self-supervised pre-training of deep learning models

The analysis of veterinary radiographic imaging data is an essential step in the diagnosis of many thoracic lesions. Given the limited time that physicians can devote to a single patient, it would be valuable to implement an automated system to help clinicians make faster but still accurate diagnoses. Currently, most of such systems are based on supervised deep learning approaches. However, the problem with these solutions is that they need a large database of labeled data. Access to such data is often limited, as it requires a great investment of both time and money. Therefore, in this work we present a solution that allows higher classification scores to be obtained using knowledge transfer from inter-species and inter-pathology self-supervised learning methods. Before training the network for classification, pretraining of the model was performed using self-supervised learning approaches on publicly available unlabeled radiographic data of human and dog images, which allowed substantially increasing the number of images for this phase. The self-supervised learning approaches included the Beta Variational Autoencoder, the Soft-Introspective Variational Autoencoder, and a Simple Framework for Contrastive Learning of Visual Representations. After the initial pretraining, fine-tuning was performed for the collected veterinary dataset using 20% of the available data. Next, a latent space exploration was performed for each model after which the encoding part of the model was fine-tuned again, this time in a supervised manner for classification. Simple Framework for Contrastive Learning of Visual Representations proved to be the most beneficial pretraining method. Therefore, it was for this method that experiments with various fine-tuning methods were carried out. We achieved a mean ROC AUC score of 0.77 and 0.66, respectively, for the laterolateral and dorsoventral projection datasets. The results show significant improvement compared to using the model without any pretraining approach.

Representation-centric approach for classification of Consumer Health Questions

Question Classification is the very first and pivotal step in Question Answering(QA) systems. It maps a given question into a predefined class. Classification of Consumer Health Questions contributes to answering medical queries of non-expert users. It plays a principal role in the performance of Consumer Health Question Answering systems. There are several approaches to the classification of questions. However, they do not consider the nature of Consumer Health Questions. In this study, we perform experiments focusing on the question representation methods to classify Consumer Health Questions accurately. These experiments provide an intuition of practical considerations for Consumer Health Question classification. The questions were represented through certain manually-designed features, word embedding, sentence embedding, and finetuning-based techniques. We create question classifiers that involve designing models for representing and categorizing questions. Based on the result, the fine-tuned Bidirectional Encoder Representations from Transformers(BERT)-large-based model with an accuracy of 86.00% on Genetic and Rare Diseases (GARD) and 80.40% on Yahoo! Answers-based datasets outperform previous works. In addition, the method based on A Robustly Optimized BERT Pretraining Approach(RoBERTa)-large model with 86.20% accuracy surpasses fine-tuned BERT-large model on the GARD dataset. Examining the outcomes of the models gives insight into decisive considerations for representing and classifying Consumer Health Questions.

Improving Clothing Product Quality and Reducing Waste Based on Consumer Review Using RoBERTa and BERTopic Language Model

The disposability of clothing has emerged as a critical concern, precipitating waste accumulation due to product quality degradation. Such consequences exert significant pressure on resources and challenge sustainability efforts. In response, this research focuses on empowering clothing companies to elevate product excellence by harnessing consumer feedback. Beyond insights, this research extends to sustainability by providing suggestions on refining product quality by improving material handling, gradually mitigating waste production, and cultivating longevity, therefore decreasing discarded clothes. Managing a vast influx of diverse reviews necessitates sophisticated natural language processing (NLP) techniques. Our study introduces a Robustly optimized BERT Pretraining Approach (RoBERTa) model calibrated for multilabel classification and BERTopic for topic modeling. The model adeptly distills vital themes from consumer reviews, exhibiting astounding accuracy in projecting concerns across various dimensions of clothing quality. NLP’s potential lies in endowing companies with insights into consumer review, augmented by the BERTopic to facilitate immersive exploration of harvested review topics. This research presents a thorough case for integrating machine learning to foster sustainability and waste reduction. The contribution of this research is notable for its integration of RoBERTa and BERTopic in multilabel classification tasks and topic modeling in the fashion industry. The results indicate that the RoBERTa model exhibits remarkable performance, as demonstrated by its macro-averaged F1 score of 0.87 and micro-averaged F1 score of 0.87. Likewise, BERTopic achieves a coherence score of 0.67, meaning the model can form an insightful topic.

PhiUSIIL: A diverse security profile empowered phishing URL detection framework based on similarity index and incremental learning

With the proliferation of the World Wide Web and the increasing sophistication of cyber threats, phishing attacks have emerged as a significant concern for individuals and organizations alike. Phishing attacks, commonly executed through deceptive URLs, aim to deceive users into divulging sensitive information, leading to financial loss, identity theft, or compromising sensitive data. It continues to pose a significant threat to individuals and organizations in today's digital landscape, necessitating the development of effective and efficient detection frameworks. This article presents PhiUSIIL, a Phishing URL detection framework based on Similarity Index and Incremental Learning. The similarity index helps effectively identify visual similarity-based attacks such as zero-width characters, homograph, punycode, homophone, bit squatting, and combosquatting attacks. The incremental learning approach allows the framework to continuously update its knowledge base with new data. Further, implementing diverse security profiles accommodates diverse security requirements of users or organizations. PhiUSIIL extracts URL features, downloads the webpage from URL to extract HTML features, and derives new features from existing information to construct a phishing URL dataset, named PhiUSIIL phishing URL dataset, encompassing 134850 legitimate and 100945 phishing URLs. The proposed phishing URL detection framework has extensively experimented with the PhiUSIIL phishing URL dataset. The constructed dataset helps to improve the detection accuracy when used during pre-training approach. PhiUSIIL achieved an accuracy of 99.24% when experimented with a fully incremental training approach and 99.79% when experimented with a pre-training approach. The experimental results show its effectiveness and ensure the framework remains effective and up-to-date against emerging and sophisticated phishing techniques.

Enhancing glomeruli segmentation through cross-species pre-training

The importance of kidney biopsy, a medical procedure in which a small tissue sample is extracted from the kidney for examination, is increasing due to the rising incidence of kidney disorders. This procedure helps diagnosing several kidney diseases which are cause of kidney function changes, as well as guiding treatment decisions, and evaluating the suitability of potential donor kidneys for transplantation. In this work, a deep learning system for the automatic segmentation of glomeruli in biopsy kidney images is presented. A novel cross-species transfer learning approach, in which a semantic segmentation network is trained on mouse kidney tissue images and then fine-tuned on human data, is proposed to boost the segmentation performance. The experiments conducted using two deep semantic segmentation networks, MobileNet and SegNeXt, demonstrated the effectiveness of the cross-species pre-training approach leading to an increased generalization ability of both models.

Exploring How AI Answering Models Understand and Respond in Context.

Abstract: Question answering (QA) is an important capability for artificial intelligence systems to assist humans by providing relevant information. In recent years, large pretrained language models like BERT and GPT have shown promising results on QA tasks. This paper explores how two state-of-the-art models, BERT and GPT-4, understand questions and generate answers in conversational contexts. We first provide an overview of the architectures and pretrained objectives of both models. Then we conduct experiments on two QA datasets to evaluate each model's ability to reason about questions, leverage context and background knowledge, and provide natural and logically consistent responses. Quantitative results reveal the strengths and weaknesses of each model, with BERT demonstrating stronger reasoning abilities but GPT-4 generating more human-like responses. Through qualitative error analysis, we identify cases where each model fails and propose explanations grounded in their underlying architectures and pretraining approaches. This analysis provides insights into the current capabilities and limitations of large pretrained models for open-domain conversational QA. The results suggest directions for improving both types of models, including combining their complementary strengths, increasing reasoning ability, and incorporating more conversational context. This work highlights important considerations in developing AI systems that can intelligently understand and respond to natural language questions.