Local Visual Features Research Articles

Efficient pretraining model based on multi-scale local visual field feature reconstruction for PCB CT image element segmentation

Element segmentation is a key step in nondestructive testing of printed circuit boards (PCBs) based on computed tomography (CT) technology. In recent years, there has been rapid development of self-supervised pretraining technology that can obtain general image features without labeled samples and then uses a small amount of labeled samples to solve downstream tasks, which has good potential in PCB element segmentation. At present, a masked image modeling (MIM) pretraining model has been initially applied in PCB CT image element segmentation. However, due to the small and regular size of PCB elements such as vias, wires, and pads, the global visual field has redundancy for a single-element reconstruction, which may damage the performance of the model. Based on this issue, we propose an efficient pretraining model based on multi-scale local visual field feature reconstruction for PCB CT image element segmentation (EMLR-seg). In this model, the teacher-guided MIM pretraining model is introduced into PCB CT image element segmentation for the first time, to our knowledge, and a multi-scale local visual field extraction (MVE) module is proposed to reduce redundancy by focusing on local visual fields. At the same time, a simple four-Transformer-blocks decoder is used. Experiments show that EMLR-seg can achieve 88.6% mIoU on the PCB CT image dataset we proposed, which exceeds 1.2% by the baseline model, and the training time is reduced by 29.6 h, a reduction of 17.4% under the same experimental condition, which reflects the advantage of EMLR-seg in terms of performance and efficiency.

Progressively global–local fusion with explicit guidance for accurate and robust 3d hand pose reconstruction

Parametric and non-parametric methods are two commonly used strategies in current 3D hand pose reconstruction. Parametric methods predict low-dimensional parameters to fit a predefined hand model to the input image. Benefiting from the prior knowledge of hand models, parametric methods guarantee plausible hand poses, whereas the pose estimation accuracy is limited due to nonlinear regression and spatial information loss. Differently, non-parametric methods directly estimate the coordinates of keypoints or mesh vertices from the input image. The reconstructed 3D hand poses show high precision but may be less robust. In this paper, we integrate the advantages of two methods for accurate and robust hand pose reconstruction. Specifically, we disentangle the hand pose reconstruction into global modeling and local refinement, which is performed in a coarse-to-fine manner. Firstly, we utilize global features from the encoder to generate the initial estimation by a parametric method, which aims to provide the prior knowledge of the human hand for subsequent processes. Then, we gradually fuse multi-scale contextual features for local refinement by explicitly integrating global prior information and local visual features. In particular, we introduce a consecutive pixel-aligned feature retrieval module to extract fine-grained information from visual features, thereby achieving pixel-level alignment. Furthermore, we demonstrate that our method can be extended to weakly-supervised learning where only sparse pose annotations are needed, potentially alleviating the burden of meticulous mesh annotation. The effectiveness and robustness of our method are substantiated through both fully- and weakly-supervised experiments, demonstrating superior performance compared to state-of-the-art methods. We plan to release our code at https://github.com/Kun-Gao/P_GLFnet.

Mitigating biases in long-tailed recognition via semantic-guided feature transfer

Dealing with significant class imbalance poses a significant challenge in various real-world applications, particularly when the accurate classification and generalization of minority classes are of crucial. One key factor that results in model biases is the inadequate representation of intra-class diversity in samples from tail classes. To tackle this challenge, transfer learning strategies have emerged as a viable approach to promote the data distribution of tail classes by transferring feature learned from head classes. However, existing methods often assume that transferability exists between any head class and tail class, and randomly transfer information during training, which can be problematic when the head and tail classes have uncorrelated semantics. To tackle this problem, we propose a novel feature transfer strategy to promote the learning of discriminative representations for tail classes. Firstly, we compute the semantic similarity in a hybrid manner by combining local visual features and global semantic features. Secondly, we utilize the Gaussian mixture model to represent each class with multiple and anisotropic prototypes, which helps us fit the actual distribution accurately. We then carry out distribution-aware head-to-tail feature transfer, leveraging an optimization objective that can derive a computationally efficient closed-form upper bound. This procedure mitigates representation biases in the feature space and achieves state-of-the-art results on CIFAR100-LT, ImageNet-LT, and iNaturalist long-tailed benchmarks.

Cross-Modal Match for Language Conditioned 3D Object Grounding

Language conditioned 3D object grounding aims to find the object within the 3D scene mentioned by natural language descriptions, which mainly depends on the matching between visual and natural language. Considerable improvement in grounding performance is achieved by improving the multimodal fusion mechanism or bridging the gap between detection and matching. However, several mismatches are ignored, i.e., mismatch in local visual representation and global sentence representation, and mismatch in visual space and corresponding label word space. In this paper, we propose crossmodal match for 3D grounding from mitigating these mismatches perspective. Specifically, to match local visual features with the global description sentence, we propose BEV (Bird’s-eye-view) based global information embedding module. It projects multiple object proposal features into the BEV and the relations of different objects are accessed by the visual transformer which can model both positions and features with long-range dependencies. To circumvent the mismatch in feature spaces of different modalities, we propose crossmodal consistency learning. It performs cross-modal consistency constraints to convert the visual feature space into the label word feature space resulting in easier matching. Besides, we introduce label distillation loss and global distillation loss to drive these matches learning in a distillation way. We evaluate our method in mainstream evaluation settings on three datasets, and the results demonstrate the effectiveness of the proposed method.

Global–local feature learning for fine-grained food classification based on Swin Transformer

Separable object parts, such as the head and tail in a bird, are vital for fine-grained visual classifications. For those objects without separable parts, the classification task relies only on local and global textural image features. Although the Swin Transformer architecture was proposed to efficiently capture both local and global visual features, it still exhibits a bias towards global features. Therefore, our goal is to enhance the local feature learning capability of the Swin Transformer by adding four new modules of the Local Feature Extraction Network (L-FEN), Convolution Patch-Merging (CP), Multi-Path (MP), and Multi-View (MV). The L-FEN enhances the Swin transformer with the improved local feature capture. The CP is a localized and hierarchical adaptation of the Swin’s Patch Merging technique. The MP method integrates features across various Swin stages to accentuate local details. Meanwhile, the MV Swin transformer block supersedes traditional Swin blocks with those incorporating varied receptive fields, ensuring a broader scope of local feature capture. Our enhanced architecture, named Global–Local Swin Transformer (GL-Swin), is applied to solve a fine-grained food classification task. On three major food datasets: ISIA Food-500 UEC Food-256, and Food-101, our GL-Swin achieved accuracies of 66.75%, 85.78%, and 92.93% respectively, consistently outperforming other leading methods.

Explanatory Object Part Aggregation for Zero-Shot Learning.

Zero-shot learning (ZSL) aims to recognize objects from unseen classes only based on labeled images from seen classes. Most existing ZSL methods focus on optimizing feature spaces or generating visual features of unseen classes, both in conventional ZSL and generalized zero-shot learning (GZSL). However, since the learned feature spaces are suboptimal, there exists many virtual connections where visual features and semantic attributes are not corresponding to each other. To reduce virtual connections, in this paper, we propose to discover comprehensive and fine-grained object parts by building explanatory graphs based on convolutional feature maps, then aggregate object parts to train a part-net to obtain prediction results. Since the aggregated object parts contain comprehensive visual features for activating semantic attributes, the virtual connections can be reduced by a large extent. Since part-net aims to extract local fine-grained visual features, some attributes related to global structures are ignored. To take advantage of both local and global visual features, we design a feature distiller to distill local features into a master-net which aims to extract global features. The experimental results on AWA2, CUB, FLO, and SUN dataset demonstrate that our proposed method obviously outperforms the state-of-the-arts in both conventional ZSL and GZSL tasks.

Hierarchical Sampling for the Visualization of Large Scale-Free Graphs

Graph sampling frequently compresses a large graph into a limited screen space. This paper proposes a hierarchical structure model that partitions scale-free graphs into three blocks: the core, which captures the underlying community structure, the vertical graph, which represents minority structures that are important in visual analysis, and the periphery, which describes the connection structure between low-degree nodes. A new algorithm named hierarchical structure sampling (HSS) was then designed to preserve the characteristics of the three blocks, including complete replication of the connection relationship between high-degree nodes in the core, joint node/degree distribution between high- and low-degree nodes in the vertical graph, and proportional replication of the connection relationship between low-degree nodes in the periphery. Finally, the importance of some global statistical properties in visualization was analyzed. Both the global statistical properties and local visual features were used to evaluate the proposed algorithm, which verify that the algorithm can be applied to sample scale-free graphs with hundreds to one million nodes from a visualization perspective.

GLCM: Global-Local Captioning Model for Remote Sensing Image Captioning.

Remote sensing image captioning (RSIC), which describes a remote sensing image with a semantically related sentence, has been a cross-modal challenge between computer vision and natural language processing. For visual features extracted from remote sensing images, global features provide the complete and comprehensive visual relevance of all the words of a sentence simultaneously, while local features can emphasize the discrimination of these words individually. Therefore, not only global features are important for caption generation but also local features are meaningful for making the words more discriminative. In order to make full use of the advantages of both global and local features, in this article, we propose an attention-based global-local captioning model (GLCM) to obtain global-local visual feature representation for RSIC. Based on the proposed GLCM, the correlation of all the generated words and the relation of each separate word and the most related local visual features can be visualized in a similarity-based manner, which provides more interpretability for RSIC. In the extensive experiments, our method achieves comparable results in UCM-captions and superior results in Sydney-captions and RSICD which is the largest RSIC dataset.

Generalized Zero-Shot Space Target Recognition Based on Global-Local Visual Feature Embedding Network

Existing deep learning-based space target recognition methods rely on abundantly labeled samples and are not capable of recognizing samples from unseen classes without training. In this article, based on generalized zero-shot learning (GZSL), we propose a space target recognition framework to simultaneously recognize space targets from both seen and unseen classes. First, we defined semantic attributes to describe the characteristics of different categories of space targets. Second, we constructed a dual-branch neural network, termed the global-local visual feature embedding network (GLVFENet), which jointly learns global and local visual features to obtain discriminative feature representations, thereby achieving GZSL for space targets with higher accuracy. Specifically, the global visual feature embedding subnetwork (GVFE-Subnet) calculates the compatibility score by measuring the cosine similarity between the projection of global visual features in the semantic space and various semantic vectors, thereby obtaining global visual embeddings. The local visual feature embedding subnetwork (LVFE-Subnet) introduces soft space attention, and an encoder discovers the semantic-guided local regions in the image to then generate local visual embeddings. Finally, the visual embeddings from both branches were combined and matched with semantics. The calibrated stacking method is introduced to achieve GZSL recognition of space targets. Extensive experiments were conducted on an electromagnetic simulation dataset of nine categories of space targets, and the effectiveness of our GLVFENet is confirmed.

End-to-End Dual-Stream Transformer with a Parallel Encoder for Video Captioning

In this paper, we propose an end-to-end dual-stream transformer with a parallel encoder (DST-PE) for video captioning, which combines multimodal features and global–local representations to generate coherent captions. First, we design a parallel encoder that includes a local visual encoder and a bridge module, which simultaneously generates refined local and global visual features. Second, we devise a multimodal encoder to enhance the representation ability of our model. Finally, we adopt a transformer decoder with multimodal features as inputs and local visual features fused with textual features using a cross-attention block. Extensive experimental results demonstrate that our model achieves state-of-the-art performance with low training costs on several widely used datasets.

Correction to: Metaheuristic-Enabled Artificial Neural Network Framework For Multimodal Biometric Recognition With Local Fusion Visual Features

Correction to: Metaheuristic-Enabled Artificial Neural Network Framework For Multimodal Biometric Recognition With Local Fusion Visual Features

CICHMKG: a large-scale and comprehensive Chinese intangible cultural heritage multimodal knowledge graph

Intangible Cultural Heritage (ICH) witnesses human creativity and wisdom in long histories, composed of a variety of immaterial manifestations. The rapid development of digital technologies accelerates the record of ICH, generating a sheer number of heterogenous data but in a state of fragmentation. To resolve that, existing studies mainly adopt approaches of knowledge graphs (KGs) which can provide rich knowledge representation. However, most KGs are text-based and text-derived, and incapable to give related images and empower downstream multimodal tasks, which is also unbeneficial for the public to establish the visual perception and comprehend ICH completely especially when they do not have the related ICH knowledge. Hence, aimed at that, we propose to, taking the Chinese nation-level ICH list as an example, construct a large-scale and comprehensive Multimodal Knowledge Graph (CICHMKG) combining text and image entities from multiple data sources and give a practical construction framework. Additionally, in this paper, to select representative images for ICH entities, we propose a method composed of the denoising algorithm (CNIFA) and a series of criteria, utilizing global and local visual features of images and textual features of captions. Extensive empirical experiments demonstrate its effectiveness. Lastly, we construct the CICHMKG, consisting of 1,774,005 triples, and visualize it to facilitate the interactions and help the public dive into ICH deeply.

GAGCN: Generative adversarial graph convolutional network for non‐homogeneous texture extension synthesis

AbstractIn the non‐homogeneous texture synthesis task, the overall visual characteristics should be consistent when extending the local patterns of the exemplar. The existing methods mainly focus on the local visual features of patterns but ignore the relative position features that are important for non‐homogeneous texture synthesis. Although these methods have achieved success on homogeneous textures, they cannot perform well on non‐homogeneous textures. Thus, it is desirable to model the dependence between pixels to improve the synthesis performance. To ensure synthesis results from both the local detail structure and the overall structure, this paper proposes a non‐homogeneous texture extended synthesis model (GAGCN) combining the generate adversarial network (GAN) and the graph convolutional network (GCN). The GAN learns the internal distribution of image patches, which makes the synthetic image have rich local details. The GCN learns the latent dependence between pixels according to the statistical characteristics of the image. Based on this, a novel graph similarity loss is proposed. This loss describes the latent spatial differences between the sample image and the generated image, which helps the model to better capture global features. Experiments show that our method outperforms existing methods on non‐homogeneous textures.

An adaptive spatiotemporal correlation filtering visual tracking method.

Discriminative correlation filter (DCF) tracking algorithms are commonly used for visual tracking. However, we observed that different spatio-temporal targets exhibit varied visual appearances, and most DCF-based trackers neglect to exploit this spatio-temporal information during the tracking process. To address the above-mentioned issues, we propose a three-way adaptive spatio-temporal correlation filtering tracker, named ASCF, that makes fuller use of the spatio-temporal information during tracking. To be specific, we extract rich local and global visual features based on the Conformer network, establish three correlation filters at different spatio-temporal locations during the tracking process, and the three correlation filters independently track the target. Then, to adaptively select the correlation filter to achieve target tracking, we employ the average peak-to-correlation energy (APCE) and the peak-to-sidelobe ratio (PSR) to measure the reliability of the tracking results. In addition, we propose an adaptive model update strategy that adjusts the update frequency of the three correlation filters in different ways to avoid model drift due to the introduction of similar objects or background noise. Extensive experimental results on five benchmarks demonstrate that our algorithm achieves excellent performance compared to state-of-the-art trackers.

Chest X-Ray Quality Assessment Method With Medical Domain Knowledge Fusion

Digital X-ray radiography is widely used in clinical diagnosis. High quality chest X-ray is conducive to the accurate diagnosis of diseases by clinicians. However, the quality assessment of the chest X-ray images mainly depends on the subjective evaluation of doctors, and the results are influenced by the skill level and experience of the evaluators, involving many issues such as heavy workload and various uncertain factors in such subjective judgment. In this paper, we propose a chest X-ray quality assessment method that combines image-text contrastive learning with medical domain knowledge fusion. Based on pretraining the model from contrastive text-image pairs, large-scale real clinical chest X-ray and diagnostic report text information are fused, and the model is fine-tuned to achieve cross domain transfer learning. While improving the prediction accuracy of the algorithm, the cost of massive sample data annotation is avoided. The local visual patch features of the X-ray images are aligned with multiple text features to ensure that the visual features contain more fine-grained image information. Theoretical analysis and experimental results show that the contrastive learning algorithm based on the fusion of triplet information in medical knowledge graph and chest X-ray multi-modal data has achieved good performance in terms of accuracy. In addition, the method proposed in this paper can be easily extended to complete other tasks such as medical image multi-lesion segmentation and disease progression prediction.

Dual Relation Network for Scene Text Recognition

Local visual and long-range contextual features yield two complementary cues for human reading text in natural scene. Existing scene text recognition methods mainly extract local features in low level features and then model long-range dependencies in high level, this sequential pipeline may be sub-optimal to construct complete and effective representation. Except for high level features, long-range contextual relation is of importance in low level features as well since it can help separate different characters based on the intervals between characters and thus enhance the features in character area. To address this issue, we develop a dual relation module to extract complementary features in a parallel manner for scene text recognition. Dual relation module consists of a local visual branch and a long-range contextual branch from low level to high level. The local visual branch employs a topological-aware operation to model intra-character characteristic and extract discriminative features of different characters. Meanwhile, the long-range contextual branch utilizes a simple but effective strategy to incorporate inter-character relations into feature maps. Our dual relation module is a plug-and-play block which can be easily incorporated into modern deep architectures. Experimental results demonstrate that our methods achieved top performance on several standard benchmarks. Code and models will become publicly available in the future.

Visual and motor signatures of locomotion dynamically shape a population code for feature detection in Drosophila.

Natural vision is dynamic: as an animal moves, its visual input changes dramatically. How can the visual system reliably extract local features from an input dominated by self-generated signals? In Drosophila, diverse local visual features are represented by a group of projection neurons with distinct tuning properties. Here, we describe a connectome-based volumetric imaging strategy to measure visually evoked neural activity across this population. We show that local visual features are jointly represented across the population, and a shared gain factor improves trial-to-trial coding fidelity. A subset of these neurons, tuned to small objects, is modulated by two independent signals associated with self-movement, a motor-related signal, and a visual motion signal associated with rotation of the animal. These two inputs adjust the sensitivity of these feature detectors across the locomotor cycle, selectively reducing their gain during saccades and restoring it during intersaccadic intervals. This work reveals a strategy for reliable feature detection during locomotion.

Local Visual and Global Deep Features Based Blind Stitched Panoramic Image Quality Evaluation Using Ensemble Learning

Image stitching technology aims to generate a Stitched Panoramic Image (SPI) by stitching multiple narrow-view images containing over-lapping regions. However, heterogeneous artifacts may be introduced during the process of image stitching, which affects the visual perception. To automatically and accurately evaluate the perceptual quality of SPIs, a novel local visual and global deep features based blind SPI quality evaluation method is proposed in this paper. To be specific, with the consideration that image stitching mainly destroys structure, texture and color information, we first design a color structure-texture joint dictionary trained from the constructed stitched-specific image patches dataset. Given an input SPI, its local visual and global deep features are extracted to characterize the stitched-specific distortions. For local visual features, the trained dictionary is employed to capture structure, texture and color distortions by sparse features extraction. Then, considering that sparse features are insensitive to weak structural distortions, weighted local binary pattern features are extracted to measure various weak distortions. For global perceptual features, deep features are extracted via a pre-trained convolutional neural network model to represent the high-level semantics. Finally, considering the diversity of the extracted features, an ensemble learning strategy is adopted to promote the generalization performance and prediction accuracy of the proposed model. Experimental results show that, compared with the conventional 2D and SPI quality measurement methods, the proposed method can measure the stitched-specific distortions more accurately, and is more coincident with subjective ratings.

Visual Map-Based Localization for Intelligent Vehicles Using Around View Monitoring in Underground Parking Lots

Accurate and robust self-localization is a crucial task for intelligent vehicles. Because of limited access to GPS signals, localization in underground parking lots remains a problem. In this paper, fusion localization for intelligent vehicles using the widely available around view monitoring (AVM) is conducted by Kalman filter based on second-order Markov motion model (KF-MM2). The proposed method consists of two steps, one for visual map construction from AVM images and the other for map-based multi-scale localization. The proposed visual map consists of a series of nodes. Each node encodes both holistic and local visual features computed from AVM images, three-dimensional structure, and vehicle pose. In the localization step, the process of image-level localization is modeled as a Hidden Markov Model (HMM), in which the map nodes are hidden states. The result of image-level localization is calculated using forward algorithm by the given AVM image sequence. Then the metric localization is computed from local features matching. Finally, the metric localization is fused with the prediction by KF-MM2. The proposed method has been verified in two typical underground parking lots. Experimental results demonstrate that the proposed method can achieve an average error of 0.39 m in underground parking lots.

Multi-level control of adaptive camouflage by European cuttlefish

To camouflage themselves on the seafloor, European cuttlefish Sepia officinalis control the expression of about 30 pattern components to produce a range of body patterns.1 If each component were under independent control, cuttlefish could produce at least 230 patterns. To examine how cuttlefish deploy this vast potential, we recorded cuttlefish on seven experimental backgrounds, each designed to resemble a pattern component, and then compared their responses to predictions of two models of sensory control of component expression. The body pattern model proposes that cuttlefish integrate low-level sensory cues to categorize the background and co-ordinate component expression to produce a small number of overall body patterns.2-4 The feature matching model proposes that each component is expressed in response to one (or more) local visual features, and the overall pattern depends upon the combination of features in the background. Consistent with the feature matching model, six of the backgrounds elicited a specific set of one to four components, whereas the seventh elicited eleven components typical of a disruptive body pattern. This evidence suggests that both modes of control are important, and we suggest how they can be implemented by a recent hierarchical model of the cuttlefish motor system.5,6.