Local Feature Extraction Research Articles

Personalized Text-to-Image Model Enhancement Strategies: SOD Preprocessing and CNN Local Feature Integration

Recent advancements in text-to-image models have been substantial, generating new images based on personalized datasets. However, even within a single category, such as furniture, where the structures vary and the patterns are not uniform, the ability of the generated images to preserve the detailed information of the input images remains unsatisfactory. This study introduces a novel method to enhance the quality of the results produced by text-image models. The method utilizes mask preprocessing with an image pyramid-based salient object detection model, incorporates visual information into input prompts using concept image embeddings and a CNN local feature extractor, and includes a filtering process based on similarity measures. When using this approach, we observed both visual and quantitative improvements in CLIP text alignment and DINO metrics, suggesting that the generated images more closely follow the text prompts and more accurately reflect the input image’s details. The significance of this research lies in addressing one of the prevailing challenges in the field of personalized image generation: enhancing the capability to consistently and accurately represent the detailed characteristics of input images in the output. This method enables more realistic visualizations through textual prompts enhanced with visual information, additional local features, and unnecessary area removal using a SOD mask; it can also be beneficial in fields that prioritize the accuracy of visual data.

DS-Former: A dual-stream encoding-based transformer for 3D medical image segmentation

Models that utilize self-attention mechanisms, including but not limited to Vision Transformers (ViTs), have shown promising performance in visual tasks like semantic segmentation. This is attributed to their capacity to capture global features of images, enabling them to learn more comprehensive representations. However, transformer-based models typically demand a considerable amount of training data to achieve satisfactory performance, while being deficient in the ability to efficiently extract local image features. As a result, these models may not be as effective in some computer vision tasks that involve small-scale datasets, like medical image segmentation. To address these issues, this paper proposes a dual-stream encoding-based transformer dubbed as Dual-stream Transformer (DS-Former). The dual-stream module in DS-Former can simultaneously acquire local and global features in the image and construct relation between the two kinds of features via self-attention. Compared with the simple splicing or serial connection, the dual-stream module can extract more comprehensive and hierarchical feature information from the fusion interaction of the two features. Our method is evaluated on the UK Biobank (UKBB) cardiac magnetic resonance imaging (CMR) dataset and The Beyond the Cranial Vault (BTCV) abdominal challenge dataset. The experimental results indicate that our DS-Former outperforms other state-of-the-art approaches on both datasets, indicating its potential for medical images semantic segmentation.

Enhancing Workplace Safety: PPE_Swin—A Robust Swin Transformer Approach for Automated Personal Protective Equipment Detection

Accidents occur in the construction industry as a result of non-compliance with personal protective equipment (PPE). As a result of diverse environments, it is difficult to detect PPE automatically. Traditional image detection models like convolutional neural network (CNN) and vision transformer (ViT) struggle to capture both local and global features in construction safety. This study introduces a new approach for automating the detection of personal protective equipment (PPE) in the construction industry, called PPE_Swin. By combining global and local feature extraction using the self-attention mechanism based on Swin-Unet, we address challenges related to accurate segmentation, robustness to image variations, and generalization across different environments. In order to train and evaluate our system, we have compiled a new dataset, which provides more reliable and accurate detection of personal protective equipment (PPE) in diverse construction scenarios. Our approach achieves a remarkable 97% accuracy in detecting workers with and without PPE, surpassing existing state-of-the-art methods. This research presents an effective solution for enhancing worker safety on construction sites by automating PPE compliance detection.

An approach for assisting diagnosis of Alzheimer's disease based on natural language processing.

Alzheimer's Disease (AD) is a common dementia which affects linguistic function, memory, cognitive and visual spatial ability of the patients. Language is proved to have the relationship with AD, so the time that AD can be diagnosed in a doctor's office is coming. In this study, the Pitt datasets are used to detect AD which is balanced in gender and age. First bidirectional Encoder Representation from Transformers (Bert) pretrained model is used to acquire the word vector. Then two channels are constructed in the feature extraction layer, which is, convolutional neural networks (CNN) and long and short time memory (LSTM) model to extract local features and global features respectively. The local features and global features are concatenated to generate feature vectors containing rich semantics, which are sent to softmax classifier for classification. Finally, we obtain a best accuracy of 89.3% which is comparative compared to other studies. In the meanwhile, we do the comparative experiments with TextCNN and LSTM model respectively, the combined model manifests best and TextCNN takes the second place. The performance illustrates the feasibility to predict AD effectively by using acoustic and linguistic datasets.

A lightweight transformer and depthwise separable convolution model for remaining useful life prediction of turbofan engines

The degradation of turbofan engines under complex operating conditions makes it difficult to predict their remaining useful life (RUL), which affects aircraft maintenance efficiency and reliability. To maintain prediction accuracy while improving prediction speed under the limited computing power and memory resources of edge devices, a lightweight Transformer and depthwise separable convolutional neural network (DSCformer) prediction model has been proposed. In the proposed DSCformer method, a probsparse self-attention mechanism with convolutional transformation of the Value branch is developed to improve the efficiency of dot-product, and depthwise separable convolution is employed to extract local spatiotemporal features replace the decoder in Transformer. Additionally, the model’s ability to capture overall trends is improved by incorporating a scaling factor in the Bayesian optimization algorithm, which also accelerates the search for the smoothing coefficient. The evaluation on the C-MAPSS dataset shows that the proposed method achieves a root mean square error of 11.33 and 12.44, as well as scores of 634.22 and 947.35 for predicting FD002 and FD004, respectively, within a shorter training time. These results indicate that the proposed method outperforms state-of-the-art prediction methods under multiple operating conditions for aero engine RUL prediction.

ShuffleFormer: An efficient shuffle meta framework for automatic modulation classification

Automatic modulation classification (AMC) plays an essential and fundamental part in wireless communication systems, which can greatly enhance the efficiency of spectrum utilization. With the rapid development of internet of things, it has been an urgent task to effectively deploy AMC method in edge devices which usually have limited computing power. Hence, there is a pressing need for a high-performance AMC method, especially in terms of high accuracy and low computation complexity in modulation recognition. Although numerous AMC methods have been proposed, it is still a challenge to develop a novel AMC method with both high recognition accuracy and limited parameters. Recently, MetaFormer, as the promising technology, has achieved outstanding performance in processing long sequences. Inspired by MetaFormer, in this paper, an efficient end-to-end deep ShuffleFormer AMC approach is proposed to realize high performance modulation recognition. In the proposed approach, firstly, the PreBlock is proposed to extract deep information from input signals, in which both the channel and temporal attention are utilized to enrich the useful features. Then, bidirectional gated recurrent unit (BiGRU) is employed to further refine the abundant information for feature extraction. Furthermore, ShuffleFormer module is proposed, which innovatively fuse shuffle blocks into MetaFormer to further enhance its both local and global feature extraction ability with less parameters. Experimental results demonstrate that the ShuffleFormer AMC approach has more superior recognition performance than other contrastive methods, i.e., at least 5.97% average accuracy improvement with almost the same number of parameters, and less 41.4% parameters with a competitive average accuracy.

Local-global aware-transformer for occluded person re-identification

Recently, security protection is import in many scenarios. Occluded person re-identification (Re-ID) involves identifying obscured pedestrians from images captured by multiple cameras, even when the images are partially or fully occluded. Many state-of-the-art models for occluded Re-ID utilize auxiliary modules such as pose estimation, feature pyramid, and graph matching to address occlusion challenges. However, this approach results in complex models that struggle to generalize to diverse occlusions and may not effectively handle non-occluded pedestrians. Furthermore, real-world Re-ID applications frequently involve both occluded and non-occluded pedestrians, making it difficult to develop versatile models. To tackle these issues, we introduce a novel Re-ID model that learns discriminative features on both local and global scales for occluded pedestrian identification. Our proposed model, the Local-aware Transformer (LAT) for occluded person Re-ID, comprises three modules: a Discriminative Feature Extraction Module (DFEM), a Local Feature Extraction Module (LFEM), and a Global Feature Extraction Module (GFEM). Our experimental results on three occluded and two general Re-ID benchmarks demonstrate that our model surpasses existing state-of-the-art methods and achieves exceptional performance in both occluded and non-occluded Re-ID tasks.

Joint graph convolution networks and transformer for human pose estimation in sports technique analysis

Human pose estimation has various applications in domains such as sports technology analysis, virtual reality, and education. However, most previous studies focused on the respective feature representations of keypoints, but disregarded the topological relationship among keypoints. To address this challenge, we propose GTPose, a network structure that integrates graph convolutional networks and Transform. First of all, a set of multi-scale convolution operations are applied to extract local feature maps of images. Secondly, the positions of keypoints are roughly estimated by using Transform to process the sequential relations between feature maps. Finally, GCN is adopted to model the topological structure between keypoints to accurately locate the location of keypoints and learn feature representations. The performance of GTPose is evaluated on two real datasets: MS COCO and MPII. Experimental results demonstrate that GTPose outperforms other methods in human pose estimation tasks. In addition, experimental results also show that the spatial relationship between keypoints is effective for accurately characterizing keypoints.

Text sentiment analysis based on BERT-CBLBGA

Current word vector techniques can generate effective word representations from massive text data, and different deep learning models can extract various textual semantic and grammatical features from different perspectives and levels. To extract richer text sentiment features, we propose a fused model of text sentiment classification called BERT-CBLBGA, which combines BERT with multiple feature extraction deep learning models. First, BERT is utilized to generate word embedding of the text, and the obtained feature representations are then fed into TextCNN, BiLSTM, and BiGRU to extract local and global sentiment features of the text. Then we use self-attention to fuse the output of TextCNN, BiLSTM, and BiGRU to enhance the ability to extract text sentiment classification features. Finally, a classifier is employed to classify the extracted features. Experimental results on four different domain text datasets demonstrate that the BERT-CBLBGA outperforms the baseline models in terms of extracting text sentiment features and achieves the best classification performance.

A CNN-LSTM and Attention-Mechanism-Based Resistance Spot Welding Quality Online Detection Method for Automotive Bodies

Resistance spot welding poses potential challenges for automotive manufacturing enterprises with regard to ensuring the real-time and accurate quality detection of each welding spot. Nowadays, many machine learning and deep learning methods have been proposed to utilize monitored sensor data to solve these challenges. However, poor detection results or process interpretations are still unaddressed key issues. To bridge the gap, this paper takes the automotive bodies as objects, and proposes a resistance spot welding quality online detection method with dynamic current and resistance data based on a combined convolutional neural network (CNN), long short-term memory network (LSTM), and an attention mechanism. First, an overall online detection framework using an edge–cloud collaboration was proposed. Second, an online quality detection model was established. In it, the combined CNN and LSTM network were used to extract local detail features and temporal correlation features of the data. The attention mechanism was introduced to improve the interpretability of the model. Moreover, the imbalanced data problem was also solved with a multiclass imbalance algorithm and weighted cross-entropy loss function. Finally, an experimental verification and analysis were conducted. The results show that the quality detection accuracy was 98.5%. The proposed method has good detection performance and real-time detection abilities for the in-site welding processes of automobile bodies.

Image super resolution by double dictionary learning and its application to tool wear monitoring in micro milling

It is an effective means to improve the machining quality of product by monitoring the tool wear conditions in micro milling. However, due to the small size and high speed of the micro milling cutter, it is difficult to capture the clear image of tool wear by CCD camera. Therefore, a double dictionary sparse coding super-resolution (SR) method is proposed to improve the resolution of micro milling tool wear images in this work. Based on the traditional single dictionary, this work proposes the residual dictionary to reconstruct the residual image of the tool, achieving the double dictionary coefficient encoding reconstruction. In order to extract local features of low resolution (LR) tool wear images, the Gobor filter banks are designed, and then the PCA method is used to reduce dimension of extracted features. The dictionary learning based on a single high resolution (HR) image is realized, which reduces the dependence of super-resolution algorithm on large datasets. The effectiveness of Gabor filter and double dictionary for restoration tool wear image details is verified by experiments. Compared with proposed method, traditional interpolation method, sparse coding super resolution model [32] and super resolution convolutional neural network model [26], the PSNR mean value of those methods are 43.62, 42.19, 43.20, 43.21, respectively, which shows that the proposed method has excellent performance in super-resolution reconstruction of micro milling tool images for tool wear monitoring.

Improved semantic segmentation network using normal vector guidance for LiDAR point clouds

Abstract As Light Detection and Ranging (LiDAR) sensors become increasingly prevalent in the field of autonomous driving, the need for accurate semantic segmentation of three-dimensional points grows accordingly. To address this challenge, we propose a novel network model that enhances segmentation performance by utilizing normal vector information. Firstly, we present a method to improve the accuracy of normal estimation by using the intensity and reflection angles of the light emitted from the LiDAR sensor. Secondly, we introduce a novel local feature aggregation module that integrates normal vector information into the network to improve the performance of local feature extraction. The normal information is closely related to the local structure of the shape of an object, which helps the network to associate unique features with corresponding objects. We propose four different structures for local feature aggregation, evaluate them, and choose the one that shows the best performance. Experiments using the SemanticKITTI dataset demonstrate that the proposed architecture outperforms both the baseline models, RandLA-Net, and other existing methods, achieving mean intersection over union of 57.9%. Furthermore, it shows highly competitive performance compared with RandLA-Net for small and dynamic objects in a real road environment. For example, it yielded 95.2% for cars, 47.4% for bicycles, 41.0% for motorcycles, 57.4% for bicycles, and 53.2% for pedestrians.

Automatic delineation of laryngeal squamous cell carcinoma during endoscopy

White Light (WL) and Narrow Band Imaging (NBI) endoscopy are widely used to assess the superficial spreading of laryngeal squamous cell carcinoma (LSCC). However, the analysis of images requires a high level of attention and extensive clinical expertise, leading to inter-clinician variability on the assessment of tumor margins. Computer-aided segmentation can automate the identification of LSCC margins, supporting clinicians in this challenging task. In this paper, we present SegMENT-Plus, a Deep Learning segmentation convolutional network specifically developed and optimized for accurate delineation of LSCC. SegMENT-Plus uses EfficienstNetB5 as encoder with a new modified Atrous Spatial Pyramid Pooling (m-ASPP) block that integrates Channel Block Attention Module (CBAM) and Squeeze Excitation (SE). In this new architecture, CBAM extracts local and global LSCC features from the encoder, while the SE block refines cancer segmentation on each dilated convolution output. SegMENT-Plus was trained and evaluated on a multi-center dataset including clinical data from three different hospitals. A total of 4289 annotated laryngeal images from 766 patients were included in this study. The experiments showed that SegMENT-Plus achieved a Dice Similarity Coefficient (DSC) between 81.4% and 84.9% and an Intersection over Union (IOU) between 81.8% and 85.7% on the data from the different hospitals, attesting its high performance and generalization capability. The proposed segmentation architecture also demonstrated statistically significant improvement in DSC and IoU compared to other state of the art architectures, showing that this work is a concrete foundation towards a clinical system for the automatic delineation of LSCC margins in endoscopic images.

A deep learning-based local feature extraction method for improved image matching and surface reconstruction from Yutu-2 PCAM images on the Moon

Topographic reconstruction of the lunar surface or other planets is important to engineering applications and scientific research in a planetary exploration mission. The typical methods of terrain reconstruction are usually based on photogrammetry techniques. Structure-from-Motion (SfM) is one of the most effective and commonly used photogrammetric technologies that estimate three-dimensional structures from two-dimensional image sequences. To find correspondences and align photos, SfM approaches require invariant features, such as Scale Invariant Feature Transform (SIFT). The hand-crafted features, however, seriously degrade performance due to weak texture and low light conditions causing image matching failure, which will directly affect the accuracy and robustness of reconstruction. Robust local descriptors based on deep learning outperform hand-crafted descriptors since convolutional neural networks are more robust than hand-engineered representations. Therefore, a novel and robust deep learning-based local feature extraction method is proposed, comprising two branch networks integrated with attention mechanisms for generating reliable keypoints and descriptors, respectively. Furthermore, a 3D terrain surface reconstruction workflow is constructed by combining it with the modern advanced image matching method and SfM system. The effectiveness of the proposed method and the workflow were verified in experiments using Panoramic Camera (PCAM) images acquired from three waypoints explored by the Yutu-2 lunar rover during the Chang’e-4 mission. We also illustrate how our approach supports other applications, such as creating panoramic mosaics of surface imagery. This provides a new and powerful method for planetary terrain reconstruction at a high spatial resolution that can meet the requirements for rover navigation and positioning, as well as geological analysis of the Moon and other planets. The source codes developed in this study are openly available at https://github.com/Atypical-Programmer/Deep_Reconstruction_Workflow.

Image De-raining Transformer.

Existing deep learning based de-raining approaches have resorted to the convolutional architectures. However, the intrinsic limitations of convolution, including local receptive fields and independence of input content, hinder the model's ability to capture long-range and complicated rainy artifacts. To overcome these limitations, we propose an effective and efficient transformer-based architecture for the image de-raining. Firstly, we introduce general priors of vision tasks, i.e., locality and hierarchy, into the network architecture so that our model can achieve excellent de-raining performance without costly pre-training. Secondly, since the geometric appearance of rainy artifacts is complicated and of significant variance in space, it is essential for de-raining models to extract both local and non-local features. Therefore, we design the complementary window-based transformer and spatial transformer to enhance locality while capturing long-range dependencies. Besides, to compensate for the positional blindness of self-attention, we establish a separate representative space for modeling positional relationship, and design a new relative position enhanced multi-head self-attention. In this way, our model enjoys powerful abilities to capture dependencies from both content and position, so as to achieve better image content recovery while removing rainy artifacts. Experiments substantiate that our approach attains more appealing results than state-of-the-art methods quantitatively and qualitatively.

Online State-of-Health Estimation for Fast-Charging Lithium-Ion Batteries Based on a Transformer–Long Short-Term Memory Neural Network

With the rapid development of machine learning and cloud computing, deep learning methods based on big data have been widely applied in the assessment of lithium-ion battery health status. However, enhancing the accuracy and robustness of assessment models remains a challenge. This study introduces an innovative T-LSTM prediction network. Initially, a one-dimensional convolutional neural network (1DCNN) is employed to effectively extract local and global features from raw battery data, providing enriched inputs for subsequent networks. Subsequently, LSTM and transformer models are ingeniously combined to fully utilize their unique advantages in sequence modeling, further enhancing the accurate prediction of battery health status. Experiments were conducted using both proprietary and open-source datasets, and the results validated the accuracy and robustness of the proposed method. The experimental results on the proprietary dataset show that the T-LSTM-based estimation method exhibits excellent performance in various evaluation metrics, with MSE, RMSE, MAE, MAPE, and MAXE values of 0.43, 0.66, 0.53, 0.58, and 1.65, respectively. The performance improved by 30–50% compared to that of the other models. The method demonstrated superior performance in comparative experiments, offering novel insights for optimizing intelligent battery management and maintenance strategies.

Dual attention transformer network for hyperspectral image classification

Hyperspectral image classification (HSIC) has been a significant topic in the field of remote sensing in the past few years. Convolutional neural networks have shown promising performance in HSIC applications due to their strong local feature extraction ability. However, they struggle to extract global information from HSIs, thereby resulting in classification performance limitations. Recently, vision transformers have been used to solve HSIC problems, and its advantage is to adopt the multi-head self-attention mechanism to explore global dependencies. Nevertheless, the extracted features using MHSA usually exhibit over-dispersion due to the abundance of band information hidden in HSIs. In this work, we propose a novel method, called dual attention transformer network (DATN), for HSIC problems. It consists of two types of modules, namely the spatial–spectral hybrid transformer (SSHT) module and the spectral local-conv block (SLCB) module. Specifically, the SSHT module aims to utilize the MHSA to capture spatial and spectral feature information. Therefore, it can effectively utilize global spatial–spectral features and embed the local spatial information, simultaneously. Besides, we design a SLCB module to extract the local spectral information of HSIs effectively. Then the SSHT and SLCB modules are integrated into an end-to-end framework. Finally, the global and local spatial–spectral features extracted from this framework are input into the fully connected layer, and then classification results of HSIs are obtained. A series of experiments on three HSI datasets have demonstrated that our DATN approach outperforms several state-of-the-art HSIC approaches.

PMG—Pyramidal Multi-Granular Matching for Text-Based Person Re-Identification

Given a textual query, text-based person re-identification is supposed to search for the targeted pedestrian images from a large-scale visual database. Due to the inherent heterogeneity between different modalities, it is challenging to measure the cross-modal affinity between visual and textual data. Existing works typically employ single-granular methods to extract local features and align image regions with relevant words/phrases. Nevertheless, the limited robustness of single-granular methods cannot adapt to the imprecision and variances of visual and textual features, which are usually influenced by the background clutter, position transformation, posture diversity, and occlusion in surveillance videos, thereby leading to the deterioration of cross-modal matching accuracy. In this paper, we propose a Pyramidal Multi-Granular matching network (PMG) that incorporates a gradual transition process between the coarsest global information and the finest local information by a coarse-to-fine pyramidal method for multi-granular cross-modal features extraction and affinities learning. For each body part of a pedestrian, PMG is adequate in ensuring the integrity of local information while minimizing the surrounding interference signals at a certain scale and can adapt to capture discriminative signals of different body parts and achieve semantically alignment between image strips with relevant textual descriptions, thus suppressing the variances of feature extraction and improving the robustness of feature matching. Comprehensive experiments are conducted on the CUHK-PEDES and RSTPReid datasets to validate the effectiveness of the proposed method and results show that PMG outperforms state-of-the-art (SOTA) methods significantly and yields competitive accuracy of cross-modal retrieval.

GLTF-Net: Deep-Learning Network for Thick Cloud Removal of Remote Sensing Images via Global–Local Temporality and Features

Remote sensing images are very vulnerable to cloud interference during the imaging process. Cloud occlusion, especially thick cloud occlusion, significantly reduces the imaging quality of remote sensing images, which in turn affects a variety of subsequent tasks using the remote sensing images. The remote sensing images miss ground information due to thick cloud occlusion. The thick cloud removal method based on a temporality global–local structure is initially suggested as a solution to this problem. This method includes two stages: the global multi-temporal feature fusion (GMFF) stage and the local single-temporal information restoration (LSIR) stage. It adopts the fusion feature of global multi-temporal to restore the thick cloud occlusion information of local single temporal images. Then, the featured global–local structure is created in both two stages, fusing the global feature capture ability of Transformer with the local feature extraction ability of CNN, with the goal of effectively retaining the detailed information of the remote sensing images. Finally, the local feature extraction (LFE) module and global–local feature extraction (GLFE) module is designed according to the global–local characteristics, and the different module details are designed in this two stages. Experimental results indicate that the proposed method performs significantly better than the compared methods in the established data set for the task of multi-temporal thick cloud removal. In the four scenes, when compared to the best method CMSN, the peak signal-to-noise ratio (PSNR) index improved by 2.675, 5.2255, and 4.9823 dB in the first, second, and third temporal images, respectively. The average improvement of these three temporal images is 9.65%. In the first, second, and third temporal images, the correlation coefficient (CC) index improved by 0.016, 0.0658, and 0.0145, respectively, and the average improvement for the three temporal images is 3.35%. Structural similarity (SSIM) and root mean square (RMSE) are improved 0.33% and 34.29%, respectively. Consequently, in the field of multi-temporal cloud removal, the proposed method enhances the utilization of multi-temporal information and achieves better effectiveness of thick cloud restoration.

MFF-CNER: A Multi-feature Fusion Model for Chinese Named Entity Recognition in Finance Securities

The objective of Chinese financial securities named entity recognition is to extract relevant entities from unstructured Chinese text, such as news, announcements, and research reports, that impact security prices. Recognizing entities in this field is challenging due to the abundance of specialized terms, diverse expressions, and the limited feature extraction capabilities of traditional models. To address this, we propose MFF-CNER, a multi-feature fusion model, to improve the effectiveness of Chinese financial securities named entity recognition. MFF-CNER encompasses several key steps. Firstly, it leverages a BERT pre-training model to capture semantic features at the character level. Secondly, a BiLSTM network is utilized to capture contextual features specific to financial securities text. Additionally, we introduce an Iterated Dilated Convolutional Neural Network (IDCNN) to blend, and extract local features, incorporating an Attention mechanism for weighted feature integration. Finally, the predicted sequences are optimized, and decoded using the Conditional Random Field (CRF). To validate the state-of-the-art performance of MFF-CNER in this domain, we compare it with five popular methods on a Chinese financial securities dataset annotated with the BIO labeling scheme. Notably, MFF-CNER demonstrates superior performance while maintaining compatibility among its components. Furthermore, we evaluate the applicability of MFF-CNER in the Chinese financial securities domain by utilizing public datasets from diverse domains, including social media (WEIBO), and news (MSRA). This research holds practical significance for downstream applications, such as constructing financial securities knowledge graphs, and analyzing factors that influence security prices.