Local Visual Features Research Articles

A marker-less monocular vision point positioning method for industrial manual operation environments

Vision-assisted technologies in industrial manual operation such as augmented reality (AR) are increasingly popular. They require high positioning accuracy and robustness to operate normally. However, narrow spaces, moving hands, or tools may occlude or obscure local visual features of operation environments and thus affect the positioning accuracy and robustness of operating position. The resultant misguidance may even cause misoperation of operators. This paper proposes a marker-less monocular vision point positioning method for vision-assisted manual operation in industrial environments. The proposed method can accurately and robustly locate the target point of operation using constraint minimization method even if the target area has no corresponding visual features in the case of occlusion and improper illumination. The proposed method has three phases: intersection generation, intersection optimization, and target point solving. In the intersection generation phase, a certain number of intersections of epipolar lines are generated as candidate target points using fundamental matrices. Here, the solving constraint is converted from point-to-line to point-to-points. In the intersection optimization phase, the intersections are optimized to two different sets through the iterative linear fitting and geometric mean absolute error methods. Here, the solving constraint is further converted from point-to-points to point-to-point sets. In the target point solving phase, the target point is solved as a constrained minimization problem based on the distribution constraint of the two intersection sets. Here, the solving constraint is finally converted from point-to-point sets to point-to-point, and the unique optimal solution is obtained as the target point. The experimental results show that this method has a better accuracy and robustness than the traditional homography matrix method for the practical industrial operation scenes.

Cross Encoder-Decoder Transformer with Global-Local Visual Extractor for Medical Image Captioning.

Transformer-based approaches have shown good results in image captioning tasks. However, current approaches have a limitation in generating text from global features of an entire image. Therefore, we propose novel methods for generating better image captioning as follows: (1) The Global-Local Visual Extractor (GLVE) to capture both global features and local features. (2) The Cross Encoder-Decoder Transformer (CEDT) for injecting multiple-level encoder features into the decoding process. GLVE extracts not only global visual features that can be obtained from an entire image, such as size of organ or bone structure, but also local visual features that can be generated from a local region, such as lesion area. Given an image, CEDT can create a detailed description of the overall features by injecting both low-level and high-level encoder outputs into the decoder. Each method contributes to performance improvement and generates a description such as organ size and bone structure. The proposed model was evaluated on the IU X-ray dataset and achieved better performance than the transformer-based baseline results, by 5.6% in BLEU score, by 0.56% in METEOR, and by 1.98% in ROUGE-L.

Global Contextual Dependency Network for Object Detection

Current two-stage object detectors extract the local visual features of Regions of Interest (RoIs) for object recognition and bounding-box regression. However, only using local visual features will lose global contextual dependencies, which are helpful to recognize objects with featureless appearances and restrain false detections. To tackle the problem, a simple framework, named Global Contextual Dependency Network (GCDN), is presented to enhance the classification ability of two-stage detectors. Our GCDN mainly consists of two components, Context Representation Module (CRM) and Context Dependency Module (CDM). Specifically, a CRM is proposed to construct multi-scale context representations. With CRM, contextual information can be fully explored at different scales. Moreover, the CDM is designed to capture global contextual dependencies. Our GCDN includes multiple CDMs. Each CDM utilizes local Region of Interest (RoI) features and single-scale context representation to generate single-scale contextual RoI features via the attention mechanism. Finally, the contextual RoI features generated by parallel CDMs independently are combined with the original RoI features to help classification. Experiments on MS-COCO 2017 benchmark dataset show that our approach brings continuous improvements for two-stage detectors.

Region-attentive multimodal neural machine translation

We propose a multimodal neural machine translation (MNMT) method with semantic image regions called region-attentive multimodal neural machine translation (RA-NMT). Existing studies on MNMT have mainly focused on employing global visual features or equally sized grid local visual features extracted by convolutional neural networks (CNNs) to improve translation performance. However, they neglect the effect of semantic information captured inside the visual features. This study utilizes semantic image regions extracted by object detection for MNMT and integrates visual and textual features using two modality-dependent attention mechanisms. The proposed method was implemented and verified on two neural architectures of neural machine translation (NMT): recurrent neural network (RNN) and self-attention network (SAN). Experimental results on different language pairs of Multi30k dataset show that our proposed method improves over baselines and outperforms most of the state-of-the-art MNMT methods. Further analysis demonstrates that the proposed method can achieve better translation performance because of its better visual feature use.

Fusion of Novelty Detectors Using Deep and Local Invariant Visual Features for Inspection Task

In this study, a novel framework using multiple novelty detection filters is developed to learn a model of the normality of a robot’s visual perception, which is called multichannel novelty detection. Subsequently, the acquired model was used to highlight dissimilar perceptions when the robot explored an environment. The main purpose of fusing multiple novelty filters is that each novelty filter performs well in detecting specific types of novelties; therefore, a new framework is proposed that demonstrates a new way to combine multiple different purposed novelty detection filters together in order to yield an overall more robust novelty status on the visual features. To develop a multichannel novelty detection system, expectation- and appearance-based novelty detection models were used in this study. To become experts in detecting different types of novelty using these models, different features from the input image were extracted as inputs for the models. The expectation-based novelty detection model uses the MobileNetV2 deep network to extract the deep features of the input image, which is subsequently used to learn a sequential and temporal model of normality to detect novelty. By contrast, the appearance-based novelty detection model uses speeded up robust features (SURF), which provide more region-focused features within the input image, to identify whether a specific region of the image is novel. The proposed multichannel novelty detection system is a completely online and real-time approach that is very important for mobile robotics applications. The proposed framework was tested in three novel environments, and it was reported that the proposed multichannel novelty detection system performs better than expectation-based and appearance-based novelty filters separately. Statistically, in the three novel environments, the Matthews correlation coefficients are reported to be 0.94, 0.97, and 0.93, and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> scores are reported to be 0.95, 0.97, and 0.93, respectively, which proves that and can be concluded as almost perfect statistically.

Global Visual Feature and Linguistic State Guided Attention for Remote Sensing Image Captioning

The encoder–decoder framework is prevalent in existing remote-sensing image captioning (RSIC) models. The appearance of attention mechanisms brings significant results. However, current attention-based caption models only build up the relationships between the local features without introducing the global visual feature and removing redundant feature components. It will cause caption models to generate descriptive sentences that are weakly related to the scene of images. To solve the problems, this article proposed a global visual feature-guided attention (GVFGA) mechanism. First, GVFGA introduces the global visual feature and fuses them with local visual features to build up their relationships between them. Second, an attention gate utilizing the global visual feature is proposed in GVFGA to filter out redundant feature components in the fused image features and provide more salient image features. In addition, to relieve the hidden state’s burden, a linguistic state (LS) is proposed to specifically provide textual features, making the hidden state only guiding visual–textual attention process. What’s more, to further refine the fusion of visual features and textual features, a LS-Guided Attention (LSGA) mechanism is proposed. It can also filter out the irrelevant information in the fused visual–textual feature with the help of an attention gate. The experimental results show that this proposed image captioning model can achieve better results on three RSIC datasets, UCM-Captions, Sydney-Captions, and RSICD datasets.

An advanced YOLOv3 method for small-scale road object detection

Road target detection is a very challenging task in the field of computer vision because it is easily affected by complex backgrounds and sparse features of small targets. YOLOv3 (You Only Look Once v3) is currently one of the state-of-the-art object detection methods of deep learning. However, because the k-means clustering algorithm is sensitive to the initial clustering center, the local fragile visual field features related to small objects in the prediction map are severely lost and the final decision-making theory (The grid located in the center of the foreground object is responsible for predicting this object) of the network ignores the detailed information of the neighboring grid, there are still many problems in object detection. In this paper, we propose an improved algorithm based on YOLOv3 for small-scale object detection. We use the improved k-medians clustering method instead of the previous k-means to improve the model instability caused by the singularity; We propose a local enhancement method to strengthen weak features for small-scale object detection by paralleling a branch on the backbone. Besides, a flexible offset sampling structure added in parallel for information compensation is also designed. A series of experiments showing that our system has achieved good detection results on the KITTI and UA-DETRAC public datasets, and the distinguishing performance for small-scale objects is significantly improved. Therefore, our method is effective in road target detection tasks.

Integrated generalized zero-shot learning for fine-grained classification

Embedding learning (EL) and feature synthesizing (FS) are two of the popular categories of fine-grained GZSL methods. EL or FS using global features cannot discriminate fine details in the absence of local features. On the other hand, EL or FS methods exploiting local features either neglect direct attribute guidance or global information. Consequently, neither method performs well. In this paper, we propose to explore global and direct attribute-supervised local visual features for both EL and FS categories in an integrated manner for fine-grained GZSL. The proposed integrated network has an EL sub-network and a FS sub-network. Consequently, the proposed integrated network can be tested in two ways. We propose a novel two-step dense attention mechanism to discover attribute-guided local visual features. We introduce new mutual learning between the sub-networks to exploit mutually beneficial information for optimization. Moreover, we propose to compute source-target class similarity based on mutual information and transfer-learn the target classes to reduce bias towards the source domain during testing. We demonstrate that our proposed method outperforms contemporary methods on benchmark datasets.

A Saliency Detection and Gram Matrix Transform-Based Convolutional Neural Network for Image Emotion Classification

Using the convolutional neural network (CNN) method for image emotion recognition is a research hotspot of deep learning. Previous studies tend to use visual features obtained from a global perspective and ignore the role of local visual features in emotional arousal. Moreover, the CNN shallow feature maps contain image content information; such maps obtained from shallow layers directly to describe low-level visual features may lead to redundancy. In order to enhance image emotion recognition performance, an improved CNN is proposed in this work. Firstly, the saliency detection algorithm is used to locate the emotional region of the image, which is served as the supplementary information to conduct emotion recognition better. Secondly, the Gram matrix transform is performed on the CNN shallow feature maps to decrease the redundancy of image content information. Finally, a new loss function is designed by using hard labels and probability labels of image emotion category to reduce the influence of image emotion subjectivity. Extensive experiments have been conducted on benchmark datasets, including FI (Flickr and Instagram), IAPSsubset, ArtPhoto, and Abstract. The experimental results show that compared with the existing approaches, our method has a good application prospect.

Semantic-diversity transfer network for generalized zero-shot learning via inner disagreement based OOD detector

Zero-shot learning (ZSL) aims to recognize objects from unseen classes, where the key is to transfer knowledge from seen classes to unseen classes by establishing appropriate mappings between visual and semantic features. Currently, the knowledge transfer in many existing works is rather limited due to various factors: (i) the widely used visual features are global ones, and they are not completely consistent with semantic attributes; (ii) only one mapping is learned, which is not able to effectively model diverse visual–semantic relations; (iii) the bias problem in the generalized ZSL (GZSL) could not be effectively handled. In this paper, we propose two techniques to alleviate these limitations. Firstly, we propose a Semantic-diversity transfer Network (SetNet) addressing the first two limitations, where (1) a multiple-attention architecture and a diversity regularizer are proposed to learn multiple local visual features being more consistent with semantic attributes and (2) a projector ensemble which geometrically takes diverse local features as inputs is proposed to diversify visual–semantic relations. Secondly, we propose an inner disagreement based domain detection module (ID3M) for GZSL to alleviate the bias problem, which picks out unseen-class data before class-level classification. Due to the lack of unseen-class data in the training stage, ID3M employs a novel self-contained training scheme and detects out unseen-class data based on a proposed inner disagreement criterion. Experimental results on three public datasets show that the proposed SetNet with the explored ID3M achieves a significant improvement against many state-of-the-art methods.

Cross Modal Few-Shot Contextual Transfer for Heterogenous Image Classification.

Deep transfer learning aims at dealing with challenges in new tasks with insufficient samples. However, when it comes to few-shot learning scenarios, due to the low diversity of several known training samples, they are prone to be dominated by specificity, thus leading to one-sidedness local features instead of the reliable global feature of the actual categories they belong to. To alleviate the difficulty, we propose a cross-modal few-shot contextual transfer method that leverages the contextual information as a supplement and learns context awareness transfer in few-shot image classification scenes, which fully utilizes the information in heterogeneous data. The similarity measure in the image classification task is reformulated via fusing textual semantic modal information and visual semantic modal information extracted from images. This performs as a supplement and helps to inhibit the sample specificity. Besides, to better extract local visual features and reorganize the recognition pattern, the deep transfer scheme is also used for reusing a powerful extractor from the pre-trained model. Simulation experiments show that the introduction of cross-modal and intra-modal contextual information can effectively suppress the deviation of defining category features with few samples and improve the accuracy of few-shot image classification tasks.

Detecting individual abandoned houses from google street view: A hierarchical deep learning approach

Abandoned houses (AH) are focal points in urban communities by threatening local security, destroying housing markets, and burdening government finance in the U.S. legacy cities. In particular, individual-level AH detection provides essential information for fine-resolution urban studies, government decision-makers, and private sector practitioners. However, three primary conventional data sources (field data, utility data, and remote sensing data) cannot suffice to collect such fine-resolution data in the large spatial area via a cost-effective approach. To this end, Google Street View (GSV) imagery, which emerges as the mainstream open-access data source with global coverage, provides an opportunity to address this issue. Subsequently, a follow-up challenge confronting the detection of AH arises from the fact that it lacks an effective method that can discern authentic visual features from the redundant noise in GSV images. In this study, we aim to develop an effective method to detect individual-level AH from GSV imagery. Specifically, we developed a new hierarchical deep learning method to leverage both global and local visual features of AH in the detection. The method can be further divided into three steps: (1) Scene-based classification that can extract global visual features of AH was implemented through fine-tuning a pre-trained deep convolutional neural network (CNN) model. (2) We developed a patch-based classification method that can extract specific local features of AH. In this method, patches were generated from GSV images based on auto-detected local features, followed by being labeled as three categories: building patches, vegetation patches, and others. Two deep CNN models were employed to identify deteriorated building façade patches and overgrown vegetation patches, respectively. (3) Individual-level AH were detected by integrating scene classification results and patch classification results in a decision-tree model. Experimental results showed that the F-score of AH was 0.84 in a well-prepared dataset collected from five different Rust Belt cities. The proposed hierarchical deep learning approach effectively improved the accuracy comparing with the traditional scene-based method. In addition, the proposed method was applied to generate an AH map in a new site in Detroit, MI. Our study demonstrated the feasibility of GSV imagery in AH detection and showed great potential to detect AH in a large spatial extent.

Cross-Modal Semantic Matching Generative Adversarial Networks for Text-to-Image Synthesis

Synthesizing photo-realistic images based on text descriptions is a challenging image generation problem. Although many recent approaches have significantly advanced the performance of text-to-image generation, to guarantee semantic matchings between the text description and synthesized image remains very challenging. In this paper, we propose a new model, Cross-modal Semantic Matching Generative Adversarial Networks (CSM-GAN), to improve the semantic consistency between text description and synthesized image for a fine-grained text-to-image generation. Two new modules are proposed in CSM-GAN: Text Encoder Module (TEM) and Textual-Visual Semantic Matching Module (TVSMM). TVSMM is aimed at making the distance of the pairs of synthesized image and its corresponding text description closer, in global semantic embedding space, than those of mismatched pairs. This improves the semantic consistency and consequently, the generalizability of CSM-GAN. In TEM, we introduce Text Convolutional Neural Networks (Text_CNNs) to capture and highlight local visual features in textual descriptions. Thorough experiments on two public benchmark datasets demonstrated the superiority of CSM-GAN over other representative state-of-the-art methods.

Robust and effective multiple copy-move forgeries detection and localization

Copy-move (or copy-paste or cloning) is one of the most common image forgeries, wherein one or more region are copied and pasted within the same image. The motivations of such forgery include hiding an element in the image or emphasizing a particular object. Copy-move image forgery is more challenging to detect than other types, such as splicing and retouching. In recent years, keypoint based copy-move forgery detection, which extracts image keypoints and uses local visual features to identify duplicated regions, exhibits remarkable performance with respect to memory requirement and robustness against various attacks. However, these approaches usually have poor detection ability when copy-move forgeries only involve small or smooth regions. Moreover, they cannot always effectively deal with multiple copy-move forgeries. To tackle these challenges, we propose a robust and effective multiple copy-move forgeries detection and localization method through adaptive keypoint extraction, robust local feature representation, and offsets clustering based post-processing. Firstly, we develop a new image keypoint detector, named generic features from accelerated segment test, and extract adaptively the uniform distribution keypoints from the forged image by employing the adaptive-thresholding and non-maximum suppression. Then, we introduce fast quaternion polar complex exponential transform to describe the image keypoints compactly and distinctively, and utilize the KD tree based K-nearest neighbor matching to find possible correspondences. Finally, the falsely matched pairs are removed by employing the offsets information based candidate clustering, and the duplicated regions are localized using RANSAC and ZNCC algorithm. We conduct extensive experiments to evaluate the performance of the proposed approach, in which encouraging results validate the effectiveness of the proposed technique, especially for plain/multiple copy-move forgeries, in comparison with the state-of-the-art approaches recently proposed in the literature.

Change detection using weighted features for image-based localization

Autonomous mobile robots are becoming increasingly important in many industrial and domestic environments. Dealing with unforeseen situations is a difficult problem that must be tackled to achieve long-term robot autonomy. In vision-based localization and navigation methods, one of the major issues is the scene dynamics. The autonomous operation of the robot may become unreliable if the changes occurring in dynamic environments are not detected and managed. Moving chairs, opening and closing doors or windows, replacing objects and other changes make many conventional methods fail. To deal with these challenges, we present a novel method for change detection based on weighted local visual features. The core idea of the algorithm is to distinguish the valuable information in stable regions of the scene from the potentially misleading information in the regions that are changing. We evaluate the change detection algorithm in a visual localization framework based on feature matching by performing a series of long-term localization experiments in various real-world environments. The results show that the change detection method yields an improvement in the localization accuracy, compared to the baseline method without change detection. In addition, an experimental evaluation on a public long-term localization data set with more than 10000 images reveals that the proposed method outperforms two alternative localization methods on images recorded several months after the initial mapping.

Part-Guided Attention Learning for Vehicle Instance Retrieval

Vehicle instance retrieval (IR) often requires one to recognize the fine-grained visual differences between vehicles. Besides the holistic appearance of vehicles which is easily affected by the viewpoint variation and distortion, vehicle parts also provide crucial cues to differentiate near-identical vehicles. Motivated by these observations, we introduce a <i>Part-Guided Attention Network</i> (PGAN) to pinpoint the prominent part regions and effectively combine the global and local information for discriminative feature learning. PGAN first detects the locations of different part components and salient regions regardless of the vehicle identity, which serves as the <i>bottom-up attention</i> to narrow down the possible searching regions. To estimate the importance of detected parts, we propose a <i>Part Attention Module</i> (PAM) to adaptively locate the most discriminative regions with high-attention weights and suppress the distraction of irrelevant parts with relatively low weights. The PAM is guided by the identification loss and therefore provides <i>top-down attention</i> that enables attention to be calculated at the level of car parts and other salient regions. Finally, we aggregate the global appearance and local features together to improve the feature performance further. The PGAN combines part-guided bottom-up and top-down attention, global and local visual features in an end-to-end framework. Extensive experiments demonstrate that the proposed method achieves new state-of-the-art vehicle IR performance on four large-scale benchmark datasets.<xref ref-type="fn" rid="fn1">¹</xref>

Mouse Higher Visual Areas Provide Both Distributed and Specialized Contributions to Visually Guided Behaviors

Cortical parallel processing streams segregate many diverse features of a sensory scene. However, some features are distributed across streams, begging the question of whether and how such distributed representations contribute to perception. We determined the necessity of the primary visual cortex (V1) and three key higher visual areas (lateromedial [LM], anterolateral [AL], and posteromedial [PM]) for perception of orientation and contrast, two features that are robustly encoded across all four areas. Suppressing V1, LM, or AL decreased sensitivity for both orientation discrimination and contrast detection, consistent with a role for these areas in sensory perception. In comparison, suppressing PM selectively increased false alarm (FA) rates during contrast detection, without any effect on orientation discrimination. This effect was not retinotopically specific, suggesting that suppression of PM altered sensory integration or the decision-making process rather than processing of local visual features. Thus, we find that distributed representations in the visual system can nonetheless support specialized perceptual roles for higher visual cortical areas.

Computer Vision Based on Computational Intelligence and Its Application Analysis

Under the background of modern information technology intelligence, the comprehensive processing technology formed by the processing and calculation of information and data. The purpose of this paper is to analyze the vision application under the intelligent computer, and the computer and technology application based on intelligent integration is to scientifically design various computer sectors by expanding and extending the functions of the computer. The method used in this paper is based on face recognition in computer vision. First, image recognition is used as the image description, and then the feature points are used to extract the K-means clustering algorithm. The clustering algorithm is used to determine the k value of M., Then clustering can get M keywords, generate a classification model using the independent features of the histogram, and then use local and global visual features to extract computer vision recognition data with obstacles. Experimental research results show that computer vision is an important development direction of artificial intelligence, which uses cameras and computers instead of human eyes to perceive, recognize, track and measure targets, obtain three-dimensional information of corresponding scenes, and process captured pictures or videos.

Multi-label zero-shot learning with graph convolutional networks

The goal of zero-shot learning (ZSL) is to build a classifier that recognizes novel categories with no corresponding annotated training data. The typical routine is to transfer knowledge from seen classes to unseen ones by learning a visual-semantic embedding. Existing multi-label zero-shot learning approaches either ignore correlations among labels, suffer from large label combinations, or learn the embedding using only local or global visual features. In this paper, we propose a Graph Convolution Networks based Multi-label Zero-Shot Learning model, abbreviated as MZSL-GCN. Our model first constructs a label relation graph using label co-occurrences and compensates the absence of unseen labels in the training phase by semantic similarity. It then takes the graph and the word embedding of each seen (unseen) label as inputs to the GCN to learn the label semantic embedding, and to obtain a set of inter-dependent object classifiers. MZSL-GCN simultaneously trains another attention network to learn compatible local and global visual features of objects with respect to the classifiers, and thus makes the whole network end-to-end trainable. In addition, the use of unlabeled training data can reduce the bias toward seen labels and boost the generalization ability. Experimental results on benchmark datasets show that our MZSL-GCN competes with state-of-the-art approaches.

IMG-Net: inner-cross-modal attentional multigranular network for description-based person re-identification

Given a natural language description, description-based person re-identification aims to retrieve images of the matched person from a large-scale visual database. Due to the existing modality heterogeneity, it is challenging to measure the cross-modal similarity between images and text descriptions. Many of the existing approaches usually utilize a deep-learning model to encode local and global fine-grained features with a strict uniform partition strategy. This breaks the part coherence, making it difficult to capture meaningful information from the within-part and semantic information among body parts. To address this issue, we proposed an inner-cross-modal attentional multigranular network (IMG-Net) to incorporate inner-modal self-attention and cross-modal hard-region attention with the fine-grained model for extracting the multigranular semantic information. Specifically, the inner-modal self-attention module is proposed to address the within-part consistency broken problem using both spatial-wise and channel-wise information. Following it is a multigranular feature extraction module, which is used to extract rich local and global visual and textual features with the help of group normalization (GN). Then a cross-modal hard-region attention module is proposed to obtain the local visual representation and phrase representation. Furthermore, a GN is used instead of batch normalization for the accurate batch statistics estimation. Comprehensive experiments with ablation analysis demonstrate that IMG-Net achieves the state-of-the-art performance on the CUHK-PEDES dataset and outperforms other previous methods significantly.