Pre-trained Network Research Articles

Normalizing flow sampling with Langevin dynamics in the latent space

AbstractNormalizing flows (NF) use a continuous generator to map a simple latent (e.g. Gaussian) distribution, towards an empirical target distribution associated with a training data set. Once trained by minimizing a variational objective, the learnt map provides an approximate generative model of the target distribution. Since standard NF implement differentiable maps, they may suffer from pathological behaviors when targeting complex distributions. For instance, such problems may appear for distributions on multi-component topologies or characterized by multiple modes with high probability regions separated by very unlikely areas. A typical symptom is the explosion of the Jacobian norm of the transformation in very low probability areas. This paper proposes to overcome this issue thanks to a new Markov chain Monte Carlo algorithm to sample from the target distribution in the latent domain before transporting it back to the target domain. The approach relies on a Metropolis adjusted Langevin algorithm whose dynamics explicitly exploits the Jacobian of the transformation. Contrary to alternative approaches, the proposed strategy preserves the tractability of the likelihood and it does not require a specific training. Notably, it can be straightforwardly used with any pre-trained NF network, regardless of the architecture. Experiments conducted on synthetic and high-dimensional real data sets illustrate the efficiency of the method.

3-1-3 Weight averaging technique-based performance evaluation of deep neural networks for Alzheimer’s disease detection using structural MRI

Alzheimer’s disease (AD) is a progressive neurological disorder. It is identified by the gradual shrinkage of the brain and the loss of brain cells. This leads to cognitive decline and impaired social functioning, making it a major contributor to dementia. While there are no treatments to reverse AD’s progression, spotting the disease’s onset can have a significant impact in the medical field. Deep learning (DL) has revolutionized medical image classification by automating feature engineering, removing the requirement for human experts in feature extraction. DL-based solutions are highly accurate but demand a lot of training data, which poses a common challenge. Transfer learning (TL) has gained attention for its knack for handling limited data and expediting model training. This study uses TL to classify AD using T1-weighted 3D Magnetic Resonance Imaging (MRI) from the Alzheimer’s Disease Neuroimaging (ADNI) database. Four modified pre-trained deep neural networks (DNN), VGG16, MobileNet, DenseNet121, and NASNetMobile, are trained and evaluated on the ADNI dataset. The 3-1-3 weight averaging technique and fine-tuning improve the performance of the classification models. The evaluated accuracies for AD classification are VGG16: 98.75%; MobileNet: 97.5%; DenseNet: 97.5%; and NASNetMobile: 96.25%. The receiver operating characteristic (ROC), precision–recall (PR), and Kolmogorov-Smirnov (KS) statistic plots validate the effectiveness of the modified pre-trained model. Modified VGG16 excels with area under the curve (AUC) values of 0.99 for ROC and 0.998 for PR curves. The proposed approach shows effective AD classification by achieving high accuracy using the 3-1-3 weight averaging technique and fine-tuning.

DGCL: dual-graph neural networks contrastive learning for molecular property prediction.

In this paper, we propose DGCL, a dual-graph neural networks (GNNs)-based contrastive learning (CL) integrated with mixed molecular fingerprints (MFPs) for molecular property prediction. The DGCL-MFP method contains two stages. In the first pretraining stage, we utilize two different GNNs as encoders to construct CL, rather than using the method of generating enhanced graphs as before. Precisely, DGCL aggregates and enhances features of the same molecule by the Graph Isomorphism Network and the Graph Attention Network, with representations extracted from the same molecule serving as positive samples, and others marked as negative ones. In the downstream tasks training stage, features extracted from the two above pretrained graph networks and the meticulously selected MFPs are concated together to predict molecular properties. Our experiments show that DGCL enhances the performance of existing GNNs by achieving or surpassing the state-of-the-art self-supervised learning models on multiple benchmark datasets. Specifically, DGCL increases the average performance of classification tasks by 3.73$\%$ and improves the performance of regression task Lipo by 0.126. Through ablation studies, we validate the impact of network fusion strategies and MFPs on model performance. In addition, DGCL's predictive performance is further enhanced by weighting different molecular features based on the Extended Connectivity Fingerprint. The code and datasets of DGCL will be made publicly available.

Representation learning with unconditional denoising diffusion models for dynamical systems

Abstract. We propose denoising diffusion models for data-driven representation learning of dynamical systems. In this type of generative deep learning, a neural network is trained to denoise and reverse a diffusion process, where Gaussian noise is added to states from the attractor of a dynamical system. Iteratively applied, the neural network can then map samples from isotropic Gaussian noise to the state distribution. We showcase the potential of such neural networks in proof-of-concept experiments with the Lorenz 1963 system. Trained for state generation, the neural network can produce samples that are almost indistinguishable from those on the attractor. The model has thereby learned an internal representation of the system, applicable for different tasks other than state generation. As a first task, we fine-tune the pre-trained neural network for surrogate modelling by retraining its last layer and keeping the remaining network as a fixed feature extractor. In these low-dimensional settings, such fine-tuned models perform similarly to deep neural networks trained from scratch. As a second task, we apply the pre-trained model to generate an ensemble out of a deterministic run. Diffusing the run, and then iteratively applying the neural network, conditions the state generation, which allows us to sample from the attractor in the run's neighbouring region. To control the resulting ensemble spread and Gaussianity, we tune the diffusion time and, thus, the sampled portion of the attractor. While easier to tune, this proposed ensemble sampler can outperform tuned static covariances in ensemble optimal interpolation. Therefore, these two applications show that denoising diffusion models are a promising way towards representation learning for dynamical systems.

General framework for unsteady aerodynamic prediction of airfoils based on deep transfer learning

Analyzing the unsteady aerodynamic performance of airfoils under dynamic stall using computational fluid dynamics (CFD) is computationally intensive. Although deep learning models can quickly predict aerodynamic parameters, their generalization capability on a small-scale dataset is often poor. This paper presents a novel deep transfer learning (TL) framework that combines model-based and instance-based transfer methods, termed synergistic instance-model TL. This framework facilitates rapid predictions of unsteady aerodynamic performance for various airfoils and pitch oscillations from the small-scale dataset. The framework integrates the accelerated training speed of model-based methods with the dynamic dataset expansion benefits of instance-based approaches. Initially, a pre-trained Wasserstein-deep convolutional generative adversarial network (W-DCGAN) is developed, combining a convolutional neural network with a generative adversarial network to predict aerodynamic hysteresis loops for the SC1095 airfoil in the source domain. The framework then fine-tunes the pre-trained model and incorporates weighted source domain dataset into the small-scale target domain dataset, producing the transferred model W-DCGAN-TL. This approach significantly reduces prediction inaccuracies compared to model-based and non-TL methods when applied to the small-scale dataset. The framework's flexibility allows the use of pre-trained models and datasets from related aerodynamic problems to address issues with insufficient data. Consequently, it is expected to reduce the dependency on extensive datasets, enhance design efficiency, and minimize resource requirements.

Classification of Lung Cancer using Pre-Trained Deep Learning Models

Many Computer Assisted Diagnosis (CAD) systems have been designed and used in recent past for diagnosingdifferenttypes of cancer. Identification of carcinoma at an earlier stage is more important, and it is made possible due to the use modern image processing and deep learning methods.The occurrence of Lung cancer is seen to be increased and Computed Tomography (CT) scan images were utilized in investigation to locate and classify lung cancer, also for determining the severity of those cancer. This study is aimed at employing pre-trained deep neural networks for classification of lung cancer images. A gaussian-based approach is used to segment CT scan images. In this research exploits a transfer learning-based classification method for the chest CT images acquired from Cancer Image Archive and available in the Kaggle platform. Pre- trained models VGG and RESNET were trained using segmented chest CT images, and their performance was evaluated using different optimization algorithms were analyzed. Keywords: Computer Aided Diagnosis, lung cancer, deep learning, CT image, gaussian, transfer learning, pre-trained models, optimization algorithms.

A Meta-Learning Approach for Classifying Multimodal Retinal Images of Retinal Vein Occlusion With Limited Data.

To propose and validate a meta-learning approach for detecting retinal vein occlusion (RVO) from multimodal images with only a few samples. In this cross-sectional study, we formulate the problem as meta-learning. The meta-training dataset consists of 1254 color fundus (CF) images from 39 different fundus diseases. Two meta-testing datasets include a public domain dataset and an independent dataset from Kandze Prefecture People's Hospital. The proposed meta-learning models comprise two modules: the feature extraction networks and the prototypical networks (PNs). We use two deep learning models (the ResNet and the Contrastive Language-Image Pre-Training networks [CLIP]) for feature extraction. We evaluate the performance of the algorithms using accuracy, area under the receiver operating characteristic curve (AUCROC), F1-score, and recall. CLIP-based PNs outperform across all meta-testing datasets. For the public APTOS dataset, meta-learning algorithms achieve good results with an accuracy of 86.06% and an AUCROC of 0.87 with only 16 training images. In the hospital datasets, meta-learning algorithms show excellent diagnostic capability for detecting RVO with a very low number of shots (AUCROC above 0.99 for n = 4, 8, and 16, respectively). Notably, even though the meta-training dataset does not include fluorescein angiography (FA) images, meta-learning algorithms also have excellent diagnostic capability for detecting RVO from images with a different modality (AUCROC above 0.93 for n = 4, 8, and 16, respectively). The proposed meta-learning models excel in detecting RVO, not only on CF images but also on FA images from a different imaging modality. The proposed meta-learning models could be useful in automatically detecting RVO on CF and FA images.

Investigation of protein family relationships with deep learning.

In this article, we propose a method for finding similarities between Pfam families based on the pre-trained neural network ProtENN2. We use the model ProtENN2 per-residue embeddings to produce new high-dimensional per-family embeddings and develop an approach for calculating inter-family similarity scores based on these embeddings, and evaluate its predictions using structure comparison. We apply our method toPfam annotation by refining clan membership for Pfam families, suggesting both new members of existing clans and potential new clans for future Pfam releases. We investigate some of the failure modes of our approach, which suggests directions for future improvements. Our method is relatively simple with few parameters and could be applied to other protein family classification models. Overall, our work suggests potential benefits of employing deep learning for improving our understanding of protein family relationships and functions of previously uncharacterized families. github.com/iponamareva/ProtCNNSim, 10.5281/zenodo.10091909.

Facing Differences of Similarity: Intra- and Inter-Correlation Unsupervised Learning for Chest X-Ray Anomaly Detection.

Anomaly detection can significantly aid doctors in interpreting chest X-rays. The commonly used strategy involves utilizing the pre-trained network to extract features from normal data to establish feature representations. However, when a pre-trained network is applied to more detailed X-rays, differences of similarity can limit the robustness of these feature representations. Therefore, we propose an intra- and inter-correlation learning framework for chest X-ray anomaly detection. Firstly, to better leverage the similar anatomical structure information in chest X-rays, we introduce the Anatomical-Feature Pyramid Fusion Module for feature fusion. This module aims to obtain fusion features with both local details and global contextual information. These fusion features are initialized by a trainable feature mapper and stored in a feature bank to serve as centers for learning. Furthermore, to Facing Differences of Similarity (FDS) introduced by the pre-trained network, we propose an intra- and inter-correlation learning strategy: (1) We use intra-correlation learning to establish intra-correlation between mapped features of individual images and semantic centers, thereby initially discovering lesions; (2) We employ inter-correlation learning to establish inter-correlation between mapped features of different images, further mitigating the differences of similarity introduced by the pre-trained network, and achieving effective detection results even in diverse chest disease environments. Finally, a comparison with 18 state-of-the-art methods on three datasets demonstrates the superiority and effectiveness of the proposed method across various scenarios.

Enhancing alzheimer’s diagnosis through optimized brain lesion classification in MRI with attention-driven grid feature fusion

This paper explores cognitive interface technology, aiming to tackle current challenges and shed light on the prospects of brain-computer interfaces (BCIs). It provides a comprehensive examination of their transformative impact on medical technology and patient well-being. Specifically, this study contributes to addressing challenges in classifying brain lesion images arising from the complex nature of lesions and limitations of traditional deep learning approaches. It introduces advanced feature fusion models that leverage deep learning algorithms, including the African vulture optimization (AVO) algorithm. These models integrate informative features from multiple pre-trained networks and employ innovative fusion techniques, including the attention-driven grid feature fusion (ADGFF) model. The ADGFF model incorporates an attention mechanism based on the optimized weights obtained using AVO. The objective is to improve the overall accuracy by providing fine-grained control over different regions of interest in the input image through a grid-based technique. This grid-based technique divides the image into vertical and horizontal grids, simplifying the exemplar feature generation process without compromising performance. Experimental results demonstrate that the proposed feature fusion strategies consistently outperform individual pre-trained models in terms of accuracy, sensitivity, specificity, and F1-score. The optimized feature fusion strategies, particularly the GRU-ADGFF model, further enhance classification performance, outperforming CNN and RNN classifiers. The learning progress analysis shows convergence, indicating the effectiveness of the feature fusion strategies in capturing lesion patterns. AUC-ROC curves highlight the superior discriminatory capabilities of the ADGFF-AVO strategy. Five-fold cross-validation is employed to assess the performance of the proposed models, demonstrating their accuracy, and few other accuracy-based measures. The GRU-ADGFF model optimized with AVO consistently achieves high accuracy, sensitivity, and AUC values, demonstrating its effectiveness and generalization capability. The GRU-ADGFF model also outperforms the majority voting ensemble technique in terms of accuracy and discriminative ability. Additionally, execution time analysis reveals good scalability and resource utilization of the proposed models. The Friedman rank test confirms significant differences in classifier performance, with the GRU-ADGFF model emerging as the top-performing method across different feature fusion strategies and optimization algorithms.

E-DFu-Net: An efficient deep convolutional neural network models for Diabetic Foot Ulcer classification.

The Diabetic Foot Ulcer (DFU) is a severe complication that affects approximately 33% of diabetes patients globally, often leading to limb amputation if not detected early. This study introduces an automated approach for identifying and classifying DFU using transfer learning. DFU is typically categorized into ischemic and infection states, which are challenging to distinguish visually. We evaluate the effectiveness of pre-trained Deep Convolutional Neural Network (DCNN) models for autonomous DFU detection. Seven models are compared: EfficientNetB0, DenseNet121, ResNet101, VGG16, MobileNetV2, InceptionV3, and InceptionResNetV2. Additionally, we propose E-DFu-Net, a novel model derived from existing architectures, designed to mitigate overfitting. Experimental results demonstrate that E-DFu-Net achieves remarkable performance, with 97% accuracy in ischemia classification and 92% in infection classification. This advancement enhances current methodologies and aids practitioners in effectively detecting DFU cases.

Barley disease recognition using deep neural networks

Plant disease negatively impacts food production and quality. It is crucial to detect and recognise plant diseases correctly. Traditional approaches do not offer a rapid and comprehensive management system for detecting plant diseases. Deep learning techniques (DL) have achieved encouraging results in discriminating patterns and anomalies in visual samples. This ability provides an effective method to diagnose any plant disease symptoms automatically. However, one of the limitations of recent studies is that in-field disease detection is underexplored, so developing a model that performs well for in-field samples is necessary. The objective of this study is to develop and investigate DL techniques for in-field disease detection of barley (Hordeum vulgare L.), one of the main crops in Australia, given visual samples captured at barley trials using a consumer-grade RGB camera. Consequently, A dataset was captured from test-bed trials across multiple paddocks infected with three diseases: net form net blotch (NFNB), spot form net blotch (SFNB), and scald, in various weather conditions. The collected data, 312 images (6000 × 4000 pixels), are divided into patches of 448 × 448 pixels, which are manually annotated into four classes: no-disease, scald, NFNB and SFNB. Finally, the data was augmented using random rotation and flip to increase the dataset size. The generated barley disease dataset is then applied to several well-known pre-trained DL networks such as DenseNet, ResNet, InceptionV3, Xception, and MobileNet as the network backbone. Given limited data, these methods can be trained to detect anomalies in visual samples. The results show that MobileNet, Xception, and InceptionV3 performed well in barley disease detection. On the other hand, ResNet showed poor classification ability. Moreover, Augmenting the data improves the performance of DL networks, particularly for underperforming backbones like ResNet, and mitigates the limited data access for these data-intensive networks. The augmentation step improved MobileNet performance by approximately 6 %. MobileNet achieved the highest accuracy of 98.63 % (the average of the three diseases) in binary classification and an accuracy of 93.50 % in multi-class classification. Even though classifying SFNB and NFNB is challenging in the early stages, MobileNet achieved the minimum misclassification rate among the two diseases. The results show the efficiency of this model in diagnosing barley diseases using complex data collected from the field environment. In addition, the model is lighter and comprises fewer trainable parameters. Consequently, MobileNet is suitable for small training datasets, reducing data acquisition costs.

Meta transfer evidence deep learning for trustworthy few-shot classification

Although the standard meta-learning methods have demonstrated strong performance in few-shot image classification scenarios, the models typically lack the capability to assess the reliability of their predictions, which can lead to risks in certain applications. Aiming at this problem, we first propose a meta-learning-based Evidential Deep Learning (EDL) called Meta Evidence Deep Learning (MetaEDL), which enables reliable prediction in the few-show image classification scenario. Being the same as general meta-learning methods, MetaEDL commonly employed shallow neural networks as feature extractors to avoid overfitting when dealing with few-shot samples, which significantly restricts the model’s ability to extract features. To further address this limitation, we propose a Meta Transfer Evidence Deep Learning (MetaTEDL) to address the few-shot trustworthy classification issue. MetaTEDL adopts a large-scale pre-trained neural network as its feature extractor. In the meta-training process, we only train two lightweight neuron operations Scaling and Shifting to reduce the risk of over-fitting. Then, two evidential head neural networks are trained to integrate evidence from different sources, aiming to improve the quality of the evidence output. We conduct comprehensive experiments on several challenging few-shot classification benchmarks. The results indicate that our proposed method not only outperforms other conventional meta-learning methods in terms of few-shot classification performance, but also has good UQ (uncertainty quantification), Uncertainty-guided active learning, and OOD (Out-of-Distribution) detection capabilities.

Automated medical diagnosis of dementia through fine-tuned EfficientNet

Automated models using deep learning are more extensively used in medical imaging in the last decade. The present study contributes to the diagnosis of dementia using MRI. Dementia is a syndrome that deteriorates the cognitive function of the brain. The disease has no cure, till now, except for the prior diagnosis. The present study aims for classifying the MRI scans of two datasets OASIS and ADNI into 2 categories: binary and multi-classification. To achieve the objective, the EfficientNetB0 architecture of deep learning is fine-tuned by adding three dense layers on the top of the network. The swish activation function is used in the inner dense layers added. The dropout and batch normalization layers are also added for dealing with the problem of overfitting. This architecture offers high accuracy and high efficiency compared to other pre-trained networks. The model is assessed on various performance measures and outperformed the state of art techniques. For the OASIS dataset, the best testing accuracy for binary classification is 93.10% with a 0.01 learning rate. The sensitivity is 95.93%, specificity is 90.08%, false-negative rate is 4.07, the false-positive rate is 9.92 and the F1-score is 93.48%. The best testing accuracy of multi-classification is 84.50% with a 0.001 learning rate. For the ADNI dataset, the best testing accuracy is 96.08% with a learning rate of 0.001. The sensitivity is 94.74%, specificity is 99.32%, false-negative rate is 5.26, the false-positive rate is 0.68 and the f1-score is 97.16%. The best testing accuracy of multi-classification is 98.10 with a 0.01 learning rate. The proposed model can be utilized for developing an automated framework to help medical services to improve decision-making.

Multimodal Social Sensing for the Spatio-Temporal Evolution and Assessment of Nature Disasters.

Social sensing, using humans as sensors to collect disaster data, has emerged as a timely, cost-effective, and reliable data source. However, research has focused on the textual data. With advances in information technology, multimodal data such as images and videos are now shared on media platforms, aiding in-depth analysis of social sensing systems. This study proposed an analytical framework to extract disaster-related spatiotemporal information from multimodal social media data. Using a pre-trained multimodal neural network and a location entity recognition model, the framework integrates disaster semantics with spatiotemporal information, enhancing situational awareness. A case study of the April 2024 heavy rain event in Guangdong, China, using Weibo data, demonstrates that multimodal content correlates more strongly with rainfall patterns than textual data alone, offering a dynamic perception of disasters. These findings confirm the utility of multimodal social media data and offer a foundation for future research. The proposed framework offers valuable applications for emergency response, disaster relief, risk assessment, and witness discovery, and presents a viable approach for safety risk monitoring and early warning systems.

Crack analysis of tall concrete wind towers using an ad-hoc deep multiscale encoder–decoder with depth separable convolutions under severely imbalanced data

An accurate and timely cracking assessment, including the presence, location and crack geometric feature measurement, is crucial for evaluating concrete wind towers. Therefore, the early identification of cracks is a critical procedure in promptly evaluating structural integrity. This study proposed an ad-hoc encoder–decoder network based on DeepLabv3+ with depth separable convolutions to automatically segment cracks from real-world images captured from various concrete wind towers. The combined advantages of the improved DeepLabv3+ and the lightweight MobileNet v2 are suitable as a benchmark due to their high performance and universality. Four experiments were conducted to determine the model design choice and crack feature measurement capability: (1) six parametric tests using various pre-trained base networks and algorithm optimisers, (2) the influence of complex background noise (i.e., handwriting script) on crack segmentation performance, (3) comparative studies with cutting-edge pixel-wise segmentation models and (4) crack feature measurement (i.e., length and width). The research outcome demonstrated that DeepLabv3+ with MobileNet v2 can potentially be applied for efficient and accurate crack segmentation in concrete wind towers with complex backgrounds.

Research on the vector DOA estimation method with limited number of snapshots

The number of sample snapshots of array signals directly affects the performance of direction of arrival (DOA) estimation methods, and smaller snapshots often cannot represent all the features of array signals. However, in practical applications, owing to short-time abrupt changes, low intensity, large noise interference, and other factors of the target signal, the acoustic vector array sometimes cannot obtain sufficient signal data, making it difficult to achieve accurate DOA estimation. Therefore, this study proposes a transfer-learning-based DOA estimation method for acoustic vector arrays. This method extracts the spatial–temporal features of existing signal data by constructing a pre-trained network model based on a convolutional neural network (CNN) and long short-term memory (LSTM), and transfers the trained model to scenes with limited snapshot data through model fine-tuning, achieving the goal of improving the DOA estimation accuracy under a small number of snapshots. Simulation experiments show that the accuracy and RMSE of the proposed DOA estimation method are superior to those of traditional methods when only 1% of the target data are used. This indicates that the pre-training model based on LSTM and CNN can preserve the effective information of signal data and provides a new solution for the real-time prediction of acoustic vector arrays in scenes with a limited number of snapshots through transfer learning.

Prediction of Covid-19 From Lung CT Images Using Fine-Tuned Deep Learning Models

Objectives: This research article's principal objective is to improve and create a fully automated methodology for the recognition and diagnosis of infections in the lungs Computed Tomography pictures using pre-trained neural network architectures. Methods: The proposed model have employed a total 746 CT scan sample which is openly accessible, comprising 349 scans who tested positive for COVID-19 and 397 scans that are either healthy or show different respiratory illnesses. Convolutional neural networks (CNNs) with four different architectures—InceptionV3, ResNet-50, Xception, and VGG19 are employed and refined based on the intended objective. After then, in order to increase performance metrics, an ensemble technique employing the Adam optimizer have developed. Findings: These outcomes demonstrated the utility of Artificial intelligence in identifying COVID-19 instances. When measured against every other deep learning model the N-VGG-19 revealed an accuracy of 90%, indicating its better performance. Novelty: Using the Adamax optimizer and data augmentation approach, The proposed method have contributed a thorough assessment of many already trained neural network modules such as ResNet50, InceptionV3, N-VGG19, and Xception. Keywords: Artificial intelligence (AI), COVID-19, Convolutional Neural Network, CT Scan, Deep Learning

Graph neural network coarse-grain force field for the molecular crystal RDX

Condense phase molecular systems organize in wide range of distinct molecular configurations, including amorphous melt and glass as well as crystals often exhibiting polymorphism, that originate from their intricate intra- and intermolecular forces. While accurate coarse-grain (CG) models for these materials are critical to understand phenomena beyond the reach of all-atom simulations, current models cannot capture the diversity of molecular structures. We introduce a generally applicable approach to develop CG force fields for molecular crystals combining graph neural networks (GNN) and data from an all-atom simulations and apply it to the high-energy density material RDX. We address the challenge of expanding the training data with relevant configurations via an iterative procedure that performs CG molecular dynamics of processes of interest and reconstructs the atomistic configurations using a pre-trained neural network decoder. The multi-site CG model uses a GNN architecture constructed to satisfy translational invariance and rotational covariance for forces. The resulting model captures both crystalline and amorphous states for a wide range of temperatures and densities.

A Vision-Language Model-Based Traffic Sign Detection Method for High-Resolution Drone Images: A Case Study in Guyuan, China.

As a fundamental element of the transportation system, traffic signs are widely used to guide traffic behaviors. In recent years, drones have emerged as an important tool for monitoring the conditions of traffic signs. However, the existing image processing technique is heavily reliant on image annotations. It is time consuming to build a high-quality dataset with diverse training images and human annotations. In this paper, we introduce the utilization of Vision-language Models (VLMs) in the traffic sign detection task. Without the need for discrete image labels, the rapid deployment is fulfilled by the multi-modal learning and large-scale pretrained networks. First, we compile a keyword dictionary to explain traffic signs. The Chinese national standard is used to suggest the shape and color information. Our program conducts Bootstrapping Language-image Pretraining v2 (BLIPv2) to translate representative images into text descriptions. Second, a Contrastive Language-image Pretraining (CLIP) framework is applied to characterize not only drone images but also text descriptions. Our method utilizes the pretrained encoder network to create visual features and word embeddings. Third, the category of each traffic sign is predicted according to the similarity between drone images and keywords. Cosine distance and softmax function are performed to calculate the class probability distribution. To evaluate the performance, we apply the proposed method in a practical application. The drone images captured from Guyuan, China, are employed to record the conditions of traffic signs. Further experiments include two widely used public datasets. The calculation results indicate that our vision-language model-based method has an acceptable prediction accuracy and low training cost.