Pre-training Method Research Articles

MÉTODO DE OTIMIZAÇÃO NO TREINO DE FORÇA DO TORNOZELO DURANTE O EXERCÍCIO

ABSTRACT Introduction Good ankle joint strength is a precondition for high-quality exercise and is an important factor in preventing joint injuries. Objective Explore the method of optimizing ankle strength training during exercise. Methods 40 volunteers were selected and randomly divided into an experimental group and a control group. The 20 athletes in the experimental group were trained three times a week for six weeks using a control variable method, while the control group performed only professional daily physical training. Pre-training and post-training methods were used to collect and investigate the data regarding the effect of strength training on the ankle joint and its impact on skill and strength tests submitted to the athletes. Results Ankle strength training can improve ankle muscle strength and athletes’ ability to run and jump (P > 0.05). Conclusion Ankle joint strength training may improve athletes’ baseline sporting ability, improve ankle joint muscle strength, reduce the likelihood of joint injuries, and contribute to improved outcomes of various abilities, meriting further study and replication. Level of evidence II; Therapeutic studies - investigation of treatment outcomes.

Classification of Brain Tumor Images Using CNN.

A brain tumor is a serious malignant condition caused by unregulated as well as aberrant cell partitioning. Recent advances in deep learning have aided the healthcare business, particularly, diagnostic imaging for the diagnosis of numerous disorders. The most frequent and widely utilized machine learning model for image recognition is probably task CNN. Similarly, in our study, we categorize brain MRI scanning images using CNN and data augmentation and image processing techniques. We compared the performance of the scratch CNN model with that of pretrained VGG-16 models using transfer learning. Even though the investigation is carried out on a small dataset, the results indicate that our model's accuracy is quite successful and has extremely low complexity rates, achieving 100 percent accuracy compared to 96 percent accuracy for VGG-16. Compared to existing pretrained methods, our model uses much less processing resources and produces substantially greater accuracy.

Change Detection Based on Supervised Contrastive Learning for High-Resolution Remote Sensing Imagery

Change detection (CD) is a challenging task on high-resolution bitemporal remote sensing images. Many recent studies of CD have focused on designing fully convolutional Siamese network architectures. However, most of these methods initialize their encoders by random values or an ImageNet pretrained model, without any prior for the CD task, thus limiting the performance of the CD model. In this article, the novel supervised contrastive pretraining and fine-tuning CD (SCPFCD) framework, which is made up of two cascaded stages, is presented to train a CD network based on a pretrained encoder. In the first supervised contrastive pretraining stage, the encoder of the Siamese network is asked to solve a joint pretext task introduced by the proposed CDContrast pretraining method on labeled CD data. The proposed CDContrast pretraining method includes land contrastive learning (LCL), which is based on supervised contrastive learning, and proxy CD learning. The LCL focuses on learning the spatial relationships among the land cover from bitemporal images by solving a land contrast task, while the proxy CD learning performs a proxy CD task on the top of the upsampling projector to avoid local optima for the LCL and learn features for the CD. Then, in the second fine-tuning stage, the whole Siamese network initialized with the pretrained encoder is fine-tuned to perform the CD task in an end-to-end manner. The proposed SCPFCD framework was verified with three CD datasets of high-resolution remote sensing images. The extensive experimental results consistently show that the proposed framework can effectively improve the ability to extract change information for Siamese networks.

MLAE: A Pretraining Method for Automatic Identification of Urban Public Space

MLAE: A Pretraining Method for Automatic Identification of Urban Public Space

Improving Medical Image Classification in Noisy Labels Using only Self-supervised Pretraining.

Noisy labels hurt deep learning-based supervised image classification performance as the models may overfit the noise and learn corrupted feature extractors. For natural image classification training with noisy labeled data, model initialization with contrastive self-supervised pretrained weights has shown to reduce feature corruption and improve classification performance. However, no works have explored: i) how other self-supervised approaches, such as pretext task-based pretraining, impact the learning with noisy label, and ii) any self-supervised pretraining methods alone for medical images in noisy label settings. Medical images often feature smaller datasets and subtle inter-class variations, requiring human expertise to ensure correct classification. Thus, it is not clear if the methods improving learning with noisy labels in natural image datasets such as CIFAR would also help with medical images. In this work, we explore contrastive and pretext task-based selfsupervised pretraining to initialize the weights of a deep learning classification model for two medical datasets with self-induced noisy labels-NCT-CRC-HE-100K tissue histological images and COVID-QU-Ex chest X-ray images. Our results show that models initialized with pretrained weights obtained from self-supervised learning can effectively learn better features and improve robustness against noisy labels.

An Empirical Study of Remote Sensing Pretraining

Deep learning has largely reshaped remote sensing (RS) research for aerial image understanding and made a great success. Nevertheless, most of the existing deep models are initialized with the ImageNet pretrained weights. Since natural images inevitably present a large domain gap relative to aerial images, probably limiting the finetuning performance on downstream aerial scene tasks. This issue motivates us to conduct an empirical study of remote sensing pretraining (RSP) on aerial images. To this end, we train different networks from scratch with the help of the largest RS scene recognition dataset up to now — MillionAID, to obtain a series of RS pretrained backbones, including both convolutional neural networks (CNN) and vision transformers such as Swin and ViTAE, which have shown promising performance on computer vision tasks. Then, we investigate the impact of RSP on representative downstream tasks including scene recognition, semantic segmentation, object detection, and change detection using these CNN and vision transformer backbones. Empirical study shows that RSP can help deliver distinctive performances in scene recognition tasks and in perceiving RS related semantics such as “Bridge” and “Airplane”. We also find that, although RSP mitigates the data discrepancies of traditional ImageNet pretraining on RS images, it may still suffer from task discrepancies, where downstream tasks require different representations from scene recognition tasks. These findings call for further research efforts on both large-scale pretraining datasets and effective pretraining methods. The codes and pretrained models will be released at https://github.com/ViTAE-Transformer/ViTAE-Transformer-Remote-Sensing.

Retina Disorders Classification via OCT Scan: A Comparative Study between Self-Supervised Learning and Transfer Learning

Retina disorders are among the common types of eye disease that occur due to several reasons such as aging, diabetes and premature born. Besides, Optical Coherence Tomography (OCT) is a medical imaging method that serves as a vehicle for capturing volumetric scans of the human eye retina for diagnoses purposes. This research compared two pretraining approaches including Self-Supervised Learning (SSL) and Transfer Learning (TL) to train ResNet34 neural architecture aiming at building computer aided diagnoses tool for retina disorders recognition. In addition, the research methodology employs convolutional auto-encoder model as a generative SSL pretraining method. The research efforts are implemented on a dataset that contains 109,309 retina OCT images with three medical conditions including Choroidal Neovascularization (CNV), Diabetic Macular Edema (DME), DRUSEN as well as NORMAL condition. The research outcomes showed better performance in terms of overall accuracy, sensitivity and specificity, namely, 95.2%, 95.2% and 98.4% respectively for SSL ResNet34 in comparison to scores of 90.7%, 90.7% and 96.9% respectively for TL ResNet34. In addition, SSL pretraining approach showed significant reduction in the number of epochs required for training in comparison to both TL pretraining as well as the previous research performed on the same dataset with comparable performance.

Spatiotemporal self-supervised pre-training on satellite imagery improves food insecurity prediction

Abstract Global warming will cause unprecedented changes to the world. Predicting events such as food insecurities in specific earth regions is a valuable way to face them with adequate policies. Existing food insecurity prediction models are based on handcrafted features such as population counts, food prices, or rainfall measurements. However, finding useful features is a challenging task, and data scarcity hinders accuracy. We leverage unsupervised pre-training of neural networks to automatically learn useful features from widely available Landsat-8 satellite images. We train neural feature extractors to predict whether pairs of images are coming from spatially close or distant regions on the assumption that close regions should have similar features. We also integrate a temporal dimension to our pre-training to capture the temporal trends of satellite images with improved accuracy. We show that with unsupervised pre-training on a large set of satellite images, neural feature extractors achieve a macro F1 of 65.4% on the Famine Early Warning Systems network dataset—a 24% improvement over handcrafted features. We further show that our pre-training method leads to better features than supervised learning and previous unsupervised pre-training techniques. We demonstrate the importance of the proposed time-aware pre-training and show that the pre-trained networks can predict food insecurity with limited availability of labeled data.

Cross-Receiver Radio Frequency Fingerprint Identification Based on Contrastive Learning and Subdomain Adaptation

Radio frequency fingerprint (RFF) identification is emerging as an attractive paradigm for physical layer security. Despite the exceptional accuracy achieved by deep learning (DL) based schemes, few works consider the cross-receiver scenario. The performance deteriorates significantly when the model is deployed on new receivers directly. To this end, a cross-receiver RFF learning scheme is proposed. First, an unsupervised pre-training method based on contrastive learning is utilized to extract receiver-agnostic features. Then, the model is optimized by subdomain adaptation to further improve identification performance. The proposed scheme does not require multiple labeled datasets from different receivers. And experimental results indicate that the proposed scheme effectively alleviates performance degradation in the cross-receiver scenario.

SSCLNet: A Self-Supervised Contrastive Loss-Based Pre-Trained Network for Brain MRI Classification

Brain magnetic resonance images (MRI) convey vital information for making diagnostic decisions and are widely used to detect brain tumors. This research proposes a self-supervised pre-training method based on feature representation learning through contrastive loss applied to unlabeled data. Self-supervised learning aims to understand vital features using the raw input, which is helpful since labeled data is scarce and expensive. For the contrastive loss-based pre-training, data augmentation is applied to the dataset, and positive and negative instance pairs are fed into a deep learning model for feature learning. Subsequently, the features are passed through a neural network model to maximize similarity and contrastive learning of the instances. This pre-trained model serves as an encoder for supervised training and then the classification of MRI images. Our results show that self-supervised pre-training with contrastive loss performs better than random or ImageNet initialization. We also show that contrastive learning performs better when the diversity of images in the pre-training dataset is more. We have taken three differently sized ResNet models as the base models. Further, experiments were also conducted to study the effect of changing the augmentation types for generating positive and negative samples for self-supervised training.

A brief analysis of ChatGPT：historical evolution， current applications，and future prospects

近年来，人工智能技术接连取得突破，尤其是在强化学习、大规模语言模型和人工智能生成内容技术等方面，正逐步成为各个行业的创新驱动力。OpenAI于2022年11月30日发布的ChatGPT由于具有惊人的自然语言理解和生成能力，引起全社会大范围的关注，成为全球热议的话题，并被广泛应用于各个行业。仅两个月后，ChatGPT的月活跃用户数便达1亿，成为史上用户数增长最快的消费者应用。鉴于ChatGPT目前造成的影响，对其进行全面的分析较为必要。本文从历史沿革、应用现状和前景展望这3个角度对ChatGPT进行剖析，探究其对社会的影响、技术的原理和挑战以及未来发展的可能性，并从模型能力的角度简要介绍GPT-4相对于ChatGPT的改进。作为一个现象级技术产品，从技术角度而言ChatGPT对相关领域具有里程碑式的重要意义，从应用角度而言其可能会给人类社会带来巨大的影响。ChatGPT有潜力成为计算机领域最伟大的成就之一。但就目前而言，ChatGPT仍然存在一些局限，尚未达到强人工智能的水平。在当前阶段，研究人员需要对人工智能技术持有自信和谦虚学习的态度，继续发展相关的技术研究和应用。;Artificial intelligence（AI）technology has been developing intensively，especially for such scenarios in relevance to its applications of 1）natural language processing，2）computer vision，3）recommendation systems，and 4）forecast analysis. AI technology has been challenging for human cognition over the past decade. In recent years，natural language processing techniques can be focused on more. ChatGPT，as a case of emerging generative AI technology，is launched in December of 2022. ChatGPT，as an advanced language model，is commonly used on the basis of its a）larger model sizes，b）advanced pre-training methods，c）faster computing resources，and d）more language processing tasks. This ChatGPT-related literature review is focused on its（1）public awareness and application status，（2）characteristics， （3）mechanisms，（4）scalability，（5）challenges and limitations，（6）future development and application prospects，and （7）improvements of GPT-4 relative to ChatGPT. Cognitive computing and AI-based ChatGPT can be as a sort of language model in terms of the Transformer architecture and Generative Pre-Training（GPT）. This GPT-trained model can be related to natural language processing，which can predict the probability distribution of the next token using a multi-layer Transformer to generate natural language text. It can be outreached by training the learned language patterns on a large corpus of text. The OpenAI’s language model has shown a significant improvement in their level of intelligence from GPT-1（117 million parameters）in 2018 to GPT-3（175 billion parameters）in 2020. The language processing and generation capabilities of GPT have been improving dramatically in terms of consistent optimization like its 1）model size，2）generative models， and 3）self-supervised learning. Thereafter，reinforcement learning-based InstructGPT is originated from Human Feedback and such probability of infeasible，untrue，and biased outputs can be significantly reduced in January 2022. In December 2022，ChatGPT is introduced as the sister model of InstructGPT. ChatGPT is not only add InstructGPT-based chat attributes，and a test version is opened to the public. The core technologies of ChatGPT can be linked to 1）reinforcement learning from human feedback（RLHF），2）supervised fine-tuning（SFT），3）instruction fine-tnning（IFT），and 4）chain-ofthought （CoT）as well. ChatGPT has attracked about 100 million active users per month after the launch of two months. In comparison，TikTok took nine months to achieve 100 million monthly active users，and Instagram took two and a half years. According to Similar Web，more than 13 million independent visitors use ChatGPT on average each day in January of 2023，which is more than twice in December of 2022. The leading US new media company Buzzfeed accurately seized the opportunity of ChatGPT and saw its stock price triple in two days. The ChatGPT-derived impact shows its potential preference for consumers. The ChatGPT can play mulitiple roles for such domain like clinics，translation，official administrations，and programming tasks. Such extensive application of ChatGPT is still to be developed. However，while ChatGPT has the potential for widespread application in various industries，it cannot be universally applied to all industries. For example，as certain industrial production processes typically rely on digitalization and do not necessitate the handling of human language，natural language processing techniques may not be required. Furthermore，various other factors，such as legal restrictions and data privacy concerns，may also impinge upon the application of natural language processing technologies within certain industries. For industries that require the processing of sensitive information，such as the healthcare industry，natural language processing technologies may need to comply with strict legal regulations to ensure data privacy and security. In addition to industry-specific reasons，it should be noted that ChatGPT has not yet achieved perfection in natural language processing tasks. In summary，as a phenomenal and technological product，AI-generated ChatGPT’s potentials are beneficial for textual and multi-modal AIGC applications to a certain extent，and it may have an impact on the a）survival of corporations，b）competition among countries，and c）entire social structure. However，the current various positive evaluations of ChatGPT can only be seen as a phenomenon of good rain after a long drought，and it cannot change the fact that ChatGPT is a questions and answers（Q&A）solution based on prior knowledge and models. It is required to be acknowledged that ChatGPT does not have its true recognition，intention，and creativity yet，and its true intelligence need to be tackled further.

Multimodal Pre-training Method for Vision-language Understanding and Generation

Multimodal Pre-training Method for Vision-language Understanding and Generation

Contrastive Learning via Local Activity

Contrastive learning (CL) helps deep networks discriminate between positive and negative pairs in learning. As a powerful unsupervised pretraining method, CL has greatly reduced the performance gap with supervised training. However, current CL approaches mainly rely on sophisticated augmentations, a large number of negative pairs and chained gradient calculations, which are complex to use. To address these issues, in this paper, we propose the local activity contrast (LAC) algorithm, which is an unsupervised method based on two forward passes and locally defined loss to learn meaningful representations. The learning target of each layer is to minimize the activation value difference between two forward passes, effectively overcoming the limitations of applying CL above mentioned. We demonstrated that LAC could be a very useful pretraining method using reconstruction as the pretext task. Moreover, through pretraining with LAC, the networks exhibited competitive performance in various downstream tasks compared with other unsupervised learning methods.

Contrastive Learning between Classical and Modern Chinese for Classical Chinese Machine Reading Comprehension

By leveraging self-supervised tasks, pre-trained language model (PLM) has made significant progress in the field of machine reading comprehension (MRC) . However, in classical Chinese MRC (CCMRC) , the passage is typically in classical style, but the question and options are given in modern style. Existing pre-trained methods seldom model the relationship between classical and modern styles, resulting in overall misunderstanding of the passage. In this paper, we propose a contrastive learning method between classical and modern Chinese in order to reach a deep understanding of the two different styles. In particular, a novel pre-training task and an enhanced co-matching network have been defined: (1) The synonym discrimination (SD) task is used to identify whether modern meaning corresponds to classical Chinese. (2) The enhanced dual co-matching (EDCM) network is employed for a more interactive understanding of the classical passage and the modern options. The experimental results show that our proposed method improves language understanding ability and outperforms existing PLMs on the Haihua, CCLUE, and ChID datasets.

Constructing and analyzing domain-specific language model for financial text mining

The application of natural language processing (NLP) to financial fields is advancing with an increase in the number of available financial documents. Transformer-based models such as Bidirectional Encoder Representations from Transformers (BERT) have been successful in NLP in recent years. These cutting-edge models have been adapted to the financial domain by applying financial corpora to existing pre-trained models and by pre-training with the financial corpora from scratch. In Japanese, by contrast, financial terminology cannot be applied from a general vocabulary without further processing. In this study, we construct language models suitable for the financial domain. Furthermore, we compare methods for adapting language models to the financial domain, such as pre-training methods and vocabulary adaptation. We confirm that the adaptation of a pre-training corpus and tokenizer vocabulary based on a corpus of financial text is effective in several downstream financial tasks. No significant difference is observed between pre-training with the financial corpus and continuous pre-training from the general language model with the financial corpus. We have released our source code and pre-trained models.

MCRN: A Multi-source Cross-modal Retrieval Network for remote sensing

Cross-modal remote sensing retrieval (RSCR) has an increasing importance due to the ability to quickly and flexibly retrieve valuable data from enormous remote sensing (RS) images. However, traditional RSCR methods tend to focus on the retrieval between two modalities, when the number of modalities increases, the contradiction between the increasing semantic gap and the small amount of paired data causes the model fail to learn a superior modal representation. In this paper, inspired by the visual-based modal center in RS, we construct a multi-source cross-modal retrieval network (MCRN) that manages to unify RS retrieval tasks under multiple retrieval sources. To solve the data heterogeneity caused by multiple data sources, we propose a shared pattern transfer module (SPTM) based on pattern memory and combine the theory of generative adversarial to achieve the semantic representation unbound from modality. Simultaneously, to cope with the lack of annotation data in the RS scenario, multiple unimodal self-supervised frameworks are unified to obtain robust pre-training parameters for the designed MCRN by combining domain alignment and contrastive learning. Finally, we come up with the multi-source triplet loss, the unimodal contrast loss, and the semantic consistency loss, which efficaciously make MCRN achieve competitive results through multitask learning for semantic alignment. We construct multimodal datasets M-RSICD and M-RSITMD, conduct extensive experiments and provide a complete benchmark to facilitate the development of RS multi-source cross-modal retrieval. The code of the MCRN method and the proposed dataset have been open to access at [Link] . • We construct a visual-based multi-source cross-modal retrieval network that manages to unify RS retrieval tasks under multiple retrieval sources. • To address semantic heterogeneity among multiple data sources, we propose a shared pattern transfer module based on pattern memorizers and combine the theory of generative adversarial to achieve the semantic representation unbound from modality. • To cope with the lack of annotation data in the RS scene, we construct an unified unimodal self-supervised pre-training method, and align semantics under different modalities through the constructed multi-modal RS dataset.

IUP-BERT: Identification of Umami Peptides Based on BERT Features.

Umami is an important widely-used taste component of food seasoning. Umami peptides are specific structural peptides endowing foods with a favorable umami taste. Laboratory approaches used to identify umami peptides are time-consuming and labor-intensive, which are not feasible for rapid screening. Here, we developed a novel peptide sequence-based umami peptide predictor, namely iUP-BERT, which was based on the deep learning pretrained neural network feature extraction method. After optimization, a single deep representation learning feature encoding method (BERT: bidirectional encoder representations from transformer) in conjugation with the synthetic minority over-sampling technique (SMOTE) and support vector machine (SVM) methods was adopted for model creation to generate predicted probabilistic scores of potential umami peptides. Further extensive empirical experiments on cross-validation and an independent test showed that iUP-BERT outperformed the existing methods with improvements, highlighting its effectiveness and robustness. Finally, an open-access iUP-BERT web server was built. To our knowledge, this is the first efficient sequence-based umami predictor created based on a single deep-learning pretrained neural network feature extraction method. By predicting umami peptides, iUP-BERT can help in further research to improve the palatability of dietary supplements in the future.

LPCL: Localized prominence contrastive learning for self-supervised dense visual pre-training

Self-supervised pre-training has attracted increasing attention given its promising performance in training backbone networks without using labels. By far, most methods focus on image classification with datasets containing iconic objects and simple background, e.g. ImageNet. However, these methods show sub-optimal performance for dense prediction tasks (e.g. object detection and scene parsing) when directly pre-training on datasets (e.g. PASCAL VOC and COCO) with multiple objects and cluttered backgrounds. Researchers explored self-supervised dense pre-training methods by adapting recent image pre-training methods. Nevertheless, they require a large number of negative samples and a long training time to reach reasonable performance. In this paper, we propose LPCL, a novel self-supervised representation learning method for dense predictions to settle these issues. To guide the instance information in multi-instance datasets, we define an online object patch selection module to select the local patches with the high possibility of containing instance area in the augmented views efficiently during learning. After obtaining the patches, we present a novel multi-level contrastive learning method considering the instance representation of global-level, local-level and position-level without using negative samples. We conduct extensive experiments with LPCL directly pre-trained on PASCAL VOC and COCO. For PASCAL VOC image classification task, our model achieves state-of-the-art 86.2% accuracy pre-trained on COCO(+9.7% top-1 accuracy compared with baseline BYOL). On object detection, instance segmentation and semantic segmentation task, our proposed model also achieved competitive results compared with other state-of-the-art methods.

Self-supervised transformer-based pre-training method using latent semantic masking auto-encoder for pest and disease classification

Pest and disease classification is a challenging issue in agriculture. Currently, the classification algorithms of pests and diseases based on CNN models have become popular. However, these methods have a limited performance improvement due to a lack of global information interaction and discriminative feature representation. Therefore, we propose a self-supervised transformer-based pre-training method using latent semantic masking auto-encoder (LSMAE). In this method, a feature relationship conditional filtering (FRCF) based on k-NN graph is proposed for filtering irrelevant data from the source domain and generating a subset of source domain. The data in this subset are similar to that of target domain which can supplement feature learning of the target domain. To further improve the performance, a novel auto-encoder based on latent semantic masking is proposed for transformer model pre-training. This auto-encoder can select key patches of each image in the subset of the source domain and let the transformer model learn a more discriminative feature representation. Finally, the target domain data are utilized to fine-tune the pre-trained transformer model. Experiments conducted on public datasets, such as IP102, CPB, and Plant Village, show that our method outperforms the state-of-the-art methods. For example, our method achieves 74.69%/76.99%/99.93% accuracy on IP102/CPB/Plant Village, demonstrating that the proposed self-supervised transformer-based pre-training method is more effective in the pest and disease classification field than CNN-based methods.

Cross-modal attention for multi-modal image registration.

In the past few years, convolutional neural networks (CNNs) have been proven powerful in extracting image features crucial for medical image registration. However, challenging applications and recent advances in computer vision suggest that CNNs are limited in their ability to understand the spatial correspondence between features, which is at the core of image registration. The issue is further exaggerated when it comes to multi-modal image registration, where the appearances of input images can differ significantly. This paper presents a novel cross-modal attention mechanism for correlating features extracted from the multi-modal input images and mapping such correlation to image registration transformation. To efficiently train the developed network, a contrastive learning-based pre-training method is also proposed to aid the network in extracting high-level features across the input modalities for the following cross-modal attention learning. We validated the proposed method on transrectal ultrasound (TRUS) to magnetic resonance (MR) registration, a clinically important procedure that benefits prostate cancer biopsy. Our experimental results demonstrate that for MR-TRUS registration, a deep neural network embedded with the cross-modal attention block outperforms other advanced CNN-based networks with ten times its size. We also incorporated visualization techniques to improve the interpretability of our network, which helps bring insights into the deep learning based image registration methods. The source code of our work is available at https://github.com/DIAL-RPI/Attention-Reg.