Computer Vision Tasks Research Articles

Object Detection Using Capsule Neural Network: An Overview

Appropriate remedies for tasks like language translation, object identification, object segmentation, picture recognition, and natural language processing are needed for today's computer vision tasks. This paper explores the use of Capsule neural Networks (Caps Nets) in object identification and offers a thorough analysis of their developments and uses. It presents an analysis of the transformative impact of Caps Nets on object identification tasks. This distinctive the architecture of neural network is presented as an alternative to convolutional neural networks (CNNs). The paper highlights that how Caps Nets are unique in capturing spatial correlations and hierarchical patterns in visualization data by analyzing the fundamental ideas of the technology. Additionally, it demonstrates how Caps net out perform CNNs in terms of generalization, interpretability, and last in the resistance to spatial distortions, confirming their goodness at object detection. By integrating the Caps networks with the latest scientific findings and advances, the paper shows the current status and potential future paths for object detection methods that use this leading neural network architecture.

Adaptive search for broad attention based vision transformers

Vision Transformer (ViT) has prevailed among computer vision tasks for its powerful capability of image representation recently. Frustratingly, the manual design of efficient architectures for ViTs can be laborious, often involving repetitive trial and error processes. Furthermore, the exploration of lightweight ViTs remains limited, resulting in inferior performance compared to convolutional neural networks. To tackle these challenges, we propose Adaptive Search for Broad attention based Vision Transformers, called ASB, which automates the design of efficient ViT architectures by utilizing the broad search space and an adaptive evolutionary algorithm. The broad search space facilitates the exploration of a novel connection paradigm, enabling more comprehensive integration of attention information to improve ViT performance. Additionally, an efficient adaptive evolutionary algorithm is developed to efficiently explore architectures by dynamically learning the probability distribution of candidate operators. Our experimental results demonstrate that the adaptive evolution in ASB efficiently learns excellent lightweight models, achieving a 55% improvement in convergence speed over traditional evolutionary algorithms. Moreover, the effectiveness of ASB is validated across several visual tasks. For instance, on ImageNet classification, the searched model attains a performance of 77.8% with 6.5M parameters and outperforms state-of-the-art models, including EfficientNet and EfficientViT networks. On mobile COCO panoptic segmentation, our approach delivers 43.7% PQ. On mobile ADE20K semantic segmentation, our method attains 40.9% mIoU. The code and pre-trained models will be available soon in ASB-Code.

Visualizing and Understanding Patch Interactions in Vision Transformer.

Vision transformer (ViT) has become a leading tool in various computer vision tasks, owing to its unique self-attention mechanism that learns visual representations explicitly through cross-patch information interactions. Despite having good success, the literature seldom explores the explainability of ViT, and there is no clear picture of how the attention mechanism with respect to the correlation across comprehensive patches will impact the performance and what is the further potential. In this work, we propose a novel explainable visualization approach to analyze and interpret the crucial attention interactions among patches for ViT. Specifically, we first introduce a quantification indicator to measure the impact of patch interaction and verify such quantification on attention window design and indiscriminative patches removal. Then, we exploit the effective responsive field of each patch in ViT and devise a window-free transformer (WinfT) architecture accordingly. Extensive experiments on ImageNet demonstrate that the exquisitely designed quantitative method is shown able to facilitate ViT model learning, leading the top-1 accuracy by 4.28% at most. More remarkably, the results on downstream fine-grained recognition tasks further validate the generalization of our proposal.

Rotation Equivariant Proximal Operator for Deep Unfolding Methods in Image Restoration.

The deep unfolding approach has attracted significant attention in computer vision tasks, which well connects conventional image processing modeling manners with more recent deep learning techniques. Specifically, by establishing a direct correspondence between algorithm operators at each implementation step and network modules within each layer, one can rationally construct an almost "white box" network architecture with high interpretability. In this architecture, only the predefined component of the proximal operator, known as a proximal network, needs manual configuration, enabling the network to automatically extract intrinsic image priors in a data-driven manner. In current deep unfolding methods, such a proximal network is generally designed as a CNN architecture, whose necessity has been proven by a recent theory. That is, CNN structure substantially delivers the translational symmetry image prior, which is the most universally possessed structural prior across various types of images. However, standard CNN-based proximal networks have essential limitations in capturing the rotation symmetry prior, another universal structural prior underlying general images. This leaves a large room for further performance improvement in deep unfolding approaches. To address this issue, this study makes efforts to suggest a high-accuracy rotation equivariant proximal network that effectively embeds rotation symmetry priors into the deep unfolding framework. Especially, we deduce, for the first time, the theoretical equivariant error for such a designed proximal network with arbitrary layers under arbitrary rotation degrees. This analysis should be the most refined theoretical conclusion for such error evaluation to date and is also indispensable for supporting the rationale behind such networks with intrinsic interpretability requirements. Through experimental validation on different vision tasks, including blind image super-resolution, medical image reconstruction, and image de-raining, the proposed method is validated to be capable of directly replacing the proximal network in current deep unfolding architecture and readily enhancing their state-of-the-art performance. This indicates its potential usability in general vision tasks.

Visual place recognition for aerial imagery: A survey

Aerial imagery and its direct application to visual localization is an essential problem for many Robotics and Computer Vision tasks. While Global Navigation Satellite Systems (GNSS) are the standard default solution for solving the aerial localization problem, it is subject to a number of limitations, such as, signal instability or solution unreliability that make this option not so desirable. Consequently, visual geolocalization is emerging as a viable alternative. However, adapting Visual Place Recognition (VPR) task to aerial imagery presents significant challenges, including weather variations and repetitive patterns. Current VPR reviews largely neglect the specific context of aerial data. This paper introduces a methodology tailored for evaluating VPR techniques specifically in the domain of aerial imagery, providing a comprehensive assessment of various methods and their performance. However, we not only compare various VPR methods, but also demonstrate the importance of selecting appropriate zoom and overlap levels when constructing map tiles to achieve maximum efficiency of VPR algorithms in the case of aerial imagery. The code is available on our GitHub repository — https://github.com/prime-slam/aero-vloc.

Feature-level Fusion vs. Score-level Fusion for Image Retrieval Based on Pre-trained Deep Neural Networks

Today’s complex multimedia content made retrieving images similar to the user’s query from the database a challenging task. The performance of a Content-Based Image Retrieval System (CBIR) system highly depends on the image representation in a form of low-level features and similarity measurement. The traditional visual descriptors that do not provide good prior domain knowledge could lead to poor performance retrieval results. On the other hand, Deep Convolutional Neural Networks (DCNNs) have recently achieved a remarkable success as methods for image classification in various domains. Recently, pre-trained deep convolution neural networks on thousands of classes have the ability to extract very accurate and representative features which, in addition to classification, can also be successfully used in image retrieval systems. ResNet152, GoogLeNet and InceptionV3 are some of the effective and successful examples of pre-trained DCNNs recently applied in a computer vision tasks such as object recognition, clustering, and classification. In this paper, two approaches for a CBIR system, namely early fusion and late fusion, have been presented and compared. The early fusion utilizes concatenation of the features extracted by each possible pair of DCNNs, that is ResNet152-GoogLeNet, ResNet152-InceptionV3, and GoogLeNet-InceptionV3, and the late fusion apply CombSum method with Z-Score standardization to combine the score results provided by each DCNN of the aforementioned pairs. In the experiments on a popular WANG dataset it has been shown that late fusion approach slightly outperforms early fusion approach. The best performance of our experiments in terms of Average Precision (AP) for the top 20 results reaches 96.82%.

MT-Net: Single image dehazing based on meta learning, knowledge transfer and contrastive learning

Single image dehazing is becoming increasingly important as its results impact the efficiency of subsequent computer vision tasks. While many methods have been proposed to address this challenge, existing dehazing approaches often exhibit limited adaptability to different types of images and lack future learnability. In light of this, we propose a dehazing network based on meta-learning, knowledge transfer, and contrastive learning, abbreviated as MT-Net. In our approach, we combine knowledge transfer with meta-learning to tackle these challenges, thus enhancing the network’s generalization performance. We refine the structure of knowledge transfer by introducing a two-phases approach to facilitate learning under the guidance of teacher networks and learning committee networks. We also optimize the negative examples of contrastive learning to reduce the contrast space. Extensive experiments conducted on synthetic and real datasets demonstrate the remarkable performance of our method in both quantitative and qualitative comparisons. The code has been released on https://github.com/71717171fan/MT-Net.

Meta Invariance Defense Towards Generalizable Robustness to Unknown Adversarial Attacks.

Despite providing high-performance solutions for computer vision tasks, the deep neural network (DNN) model has been proved to be extremely vulnerable to adversarial attacks. Current defense mainly focuses on the known attacks, but the adversarial robustness to the unknown attacks is seriously overlooked. Besides, commonly used adaptive learning and fine-tuning technique is unsuitable for adversarial defense since it is essentially a zero-shot problem when deployed. Thus, to tackle this challenge, we propose an attack-agnostic defense method named Meta Invariance Defense (MID). Specifically, various combinations of adversarial attacks are randomly sampled from a manually constructed Attacker Pool to constitute different defense tasks against unknown attacks, in which a student encoder is supervised by multi-consistency distillation to learn the attack-invariant features via a meta principle. The proposed MID has two merits: 1) Full distillation from pixel-, feature- and prediction-level between benign and adversarial samples facilitates the discovery of attack-invariance. 2) The model simultaneously achieves robustness to the imperceptible adversarial perturbations in high-level image classification and attack-suppression in low-level robust image regeneration. Theoretical and empirical studies on numerous benchmarks such as ImageNet verify the generalizable robustness and superiority of MID under various attacks.

A new two-stage low-light enhancement network with progressive attention fusion strategy

Low-light image enhancement is a very challenging subject in the field of computer vision. It has a large number of degradation problems such as accumulated noise, artifacts, and color distortion. Therefore, how to solve the degradation problems and obtain clear images with high visual quality has become an important issue. It can effectively improve the performance of high-level computer vision tasks. In this study, we propose a new two-stage low-light enhancement network with a progressive attention fusion strategy, and the two hallmarks of this method are the use of global feature fusion (GFF) and local detail restoration (LDR), which can enrich the global content of the image and restore local details. Experimental results on the LOL dataset show that the proposed model can achieve good enhancement effects. Moreover, on the benchmark dataset without reference images, the proposed model also obtains a better NIQE score, which outperforms most existing state-of-the-art methods in both quantitative and qualitative evaluations. All these verify the effectiveness and superiority of the proposed method.

Design and Practice of Virtual Experimental Scenes Integrating Computer Vision and Image Processing Technologies

In this work, we introduce a novel framework, GAN-based Image Processing (GAN-IP), to design and manipulate digital experimental settings that effectively combine computer vision and image processing technology. Using the skills of Generative Adversarial Networks (GANs), GAN-IP creates and complements virtual scenes and affords rich, adaptive surroundings for analysing and creating computationally smart and perceptive algorithms. GAN-IP solves the pressing troubles of variability and absence of data through synthesising realistic pictures and scenarios, enabling simulations of the diverse environments in which computer vision structures have to function. Our approach improves the fidelity and sort of digital scenes and introduces a way to mechanically alter and evoke a pleasant image, enabling more accurate and powerful computer vision and prescient models. Through extensive experimentation, GAN-IP demonstrates remarkable improvements in the performance of computer vision tasks, including object detection, segmentation, and recognition in complicated virtual environments. This research lays the foundation for future studies in this field and provides an adaptive tool for researchers and practitioners to simulate and test superior computer imaginative and prescient image processing technologies.

An efficient hyperspectral image classification method using retentive network

In recent computer vision tasks, the vision transformer (ViT) has demonstrated competitive ability. However, ViT still has problems: the computational complexity of the self-attention layer leads to expensive and slow interference, and processing all tokens for high-resolution images may slow down due to the layer’s quadratic complexity. Recently, a retentive network with excellent performance, training parallelism, and an inexpensive inference cost was proposed. For hyperspectral image (HSI) classification, this paper proposesa retention-based network model called the HSI retentive network (HSIRN). The proposed model allows memory usage independent of the token’s sequence, facilitating the efficient processing of high-resolution images with low inference and computational costs. Although the retention encoder can extract global data, it pays limited attention to local data. A powerful tool for extracting local information is a convolutional neural network (CNN). The proposed HSIRN model uses a specific CNN-based block to extract local spectral-spatial information. To maintain degradation between successive vertical and horizontal positions with the depth dimension of the HSI, we propose a three-dimensional retention mechanism for the three-dimensional HSI dataset in the retention encoder. By efficiently using both local and global spectral-spatial information, the proposed method offers a potent tool for HSI classification. We evaluated the classification performance of the proposed HSIRN approach on four datasets through comprehensive examinations, and the results demonstrated its superiority over state-of-the-art methods. At https://github.com/RajatArya22/HSIRN, the source code will be available to the public.

Knowledge graph construction in hyperbolic space for automatic image annotation

Automatic image annotation (AIA) is a fundamental and challenging task in computer vision. Considering the correlations between tags can lead to more accurate image understanding, benefiting various applications, including image retrieval and visual search. While many attempts have been made to incorporate tag correlations in annotation models, the method of constructing a knowledge graph based on external knowledge sources and hyperbolic space has not been explored. In this paper, we create an attributed knowledge graph based on vocabulary, integrate external knowledge sources such as WordNet, and utilize hyperbolic word embeddings for the tag representations. These embeddings provide a sophisticated tag representation that captures hierarchical and complex correlations more effectively, enhancing the image annotation results. In addition, leveraging external knowledge sources enhances contextuality and significantly enriches existing AIA datasets. We exploit two deep learning-based models, the Relational Graph Convolutional Network (R-GCN) and the Vision Transformer (ViT), to extract the input features. We apply two R-GCN operations to obtain word descriptors and fuse them with the extracted visual features. We evaluate the proposed approach using three public benchmark datasets. Our experimental results demonstrate that the proposed architecture achieves state-of-the-art performance across most metrics on Corel5k, ESP Game, and IAPRTC-12.

Hierarchical Graph Neural Network: A Lightweight Image Matching Model with Enhanced Message Passing of Local and Global Information in Hierarchical Graph Neural Networks

Graph Neural Networks (GNNs) have gained popularity in image matching methods, proving useful for various computer vision tasks like Structure from Motion (SfM) and 3D reconstruction. A well-known example is SuperGlue. Lightweight variants, such as LightGlue, have been developed with a focus on stacking fewer GNN layers compared to SuperGlue. This paper proposes the h-GNN, a lightweight image matching model, with improvements in the two processing modules, the GNN and matching modules. After image features are detected and described as keypoint nodes of a base graph, the GNN module, which primarily aims at increasing the h-GNN’s depth, creates successive hierarchies of compressed-size graphs from the base graph through a clustering technique termed SC+PCA. SC+PCA combines Principal Component Analysis (PCA) with Spectral Clustering (SC) to enrich nodes with local and global information during graph clustering. A dual non-contrastive clustering loss is used to optimize graph clustering. Additionally, four message-passing mechanisms have been proposed to only update node representations within a graph cluster at the same hierarchical level or to update node representations across graph clusters at different hierarchical levels. The matching module performs iterative pairwise matching on the enriched node representations to obtain a scoring matrix. This matrix comprises scores indicating potential correct matches between the image keypoint nodes. The score matrix is refined with a ‘dustbin’ to further suppress unmatched features. There is a reprojection loss used to optimize keypoint match positions. The Sinkhorn algorithm generates a final partial assignment from the refined score matrix. Experimental results demonstrate the performance of the proposed h-GNN against competing state-of-the-art (SOTA) GNN-based methods on several image matching tasks under homography, estimation, indoor and outdoor camera pose estimation, and 3D reconstruction on multiple datasets. Experiments also demonstrate improved computational memory and runtime, approximately 38.1% and 26.14% lower than SuperGlue, and an average of about 6.8% and 7.1% lower than LightGlue. Future research will explore the effects of integrating more recent simplicial message-passing mechanisms, which concurrently update both node and edge representations, into our proposed model.

Low-Light Image Enhancement Network Using Informative Feature Stretch and Attention

Low-light images often exhibit reduced brightness, weak contrast, and color distortion. Consequently, enhancing low-light images is essential to make them suitable for computer vision tasks. Nevertheless, addressing this task is particularly challenging because of the inherent constraints posed by low-light environments. In this study, we propose a novel low-light image enhancement network using adaptive feature stretching and informative attention. The proposed network architecture mainly includes an adaptive feature stretch block designed to extend the narrow range of image features to a broader range. To achieve improved image restoration, an informative attention block is introduced to assign weight to the output features from the adaptive feature stretch block. We conduct comprehensive experiments on widely used benchmark datasets to assess the effectiveness of the proposed network. The experimental results show that the proposed low-light image enhancement network yields satisfactory results compared with existing state-of-the-art methods from both subjective and objective perspectives while maintaining acceptable network complexity.

Multiscale implicit frequency selective network for single‐image dehazing

AbstractImage dehazing is aimed to reconstruct a clear latent image from a degraded image affected by haze. Although vision transformers have achieved impressive success in various computer vision tasks, the limitations in scale and quality of available datasets have hindered the transformer effectiveness for image dehazing. Thus, convolutional neural networks (CNNs) remain the mainstream approach for image dehazing, offering robust performance and adaptability. We further explore the potential of CNNs in image dehazing by proposing a multiscale implicit frequency selection network (MIFSN). The proposed MIFSN enhances multiscale representation learning based on U‐shaped networks. As hazy and clear images considerably differ in high‐frequency components, we introduce an implicit frequency selection module to amplify high‐frequency components of features and generate candidate feature maps. Implicit frequency selection attention is then used to emphasize and merge beneficial frequency components. Results from extensive experiments on synthetic and real‐world datasets demonstrate the superior performance of MIFSN for image dehazing.

Toward identity preserving in face sketch-photo synthesis using a hybrid CNN-Mamba framework

The synthesis of facial sketch-photo has important applications in practical life, such as crime investigation. Many convolutional neural networks (CNNs) based methods have been proposed to address this issue. However, due to the substantial modal differences between sketch and photo, the CNN’s insensitivity to global information, and insufficient utilization of hierarchical features, synthesized photos struggle to balance both identity preservation and image quality. Recently, State Space Sequence Models (SSMs) have achieved exciting results in computer vision (CV) tasks. Inspired by SSMs, we design a hybrid CNN–SSM model called FaceMamba for the Face Sketch-Photo Synthesis (FSPS) task. It includes an original Face Vision Mamba Attention for modeling in latent space using SSM. Additionally, it incorporates a general auxiliary method called Attention Feature Injection that combines encoding features, decoding features, and external auxiliary features using attention mechanisms. FaceMamba combines Mamba’s modeling ability for long-range dependencies with CNN’s powerful local feature extraction ability, and utilizes hierarchical features at the appropriate position. Adequate experimental and evaluation results reveal that FaceMamba has strong competitiveness in FSPS task, achieving the best balance between identity preservation and image quality.

Hyperbolic prototype rectification for few-shot 3D point cloud classification

Few-shot point cloud classification is a challenging task in 3D computer vision and has received widespread attention from researchers. Most of the works on deep learning models rely heavily on Euclidean spatial metrics. However, point cloud objects often have complex non-Euclidean geometric structures, with underlying inter/intra-class hierarchical structures, which are difficult to capture by current Euclidean-based deep learning models. Moreover, due to the lack of training samples, many few-shot learning methods often suffer from the overfitting problem. Given the Hyperbolic metric of non-Euclidean geometry offering hierarchical structural prior, as we assume to be able to assist FSL task, we propose Hyperbolic Prototype Rectification (HPR) for few-shot point cloud classification, without requiring extra learnable parameter. Firstly, point clouds are embedded into hyperbolic space to better describe hierarchical similarity relationships in data. Secondly, the HPR utilizes hyperbolic spatial and distributional information to enhance the feature representation and improve the generalization capability, with more appropriate hyperbolic prototypes. The few-shot classification experiments and further ablation studies conducted on widely used point cloud datasets demonstrate the effectiveness of our method. On the real-world ScanObjectNN(-PB) datasets, the average classification accuracy outperforms the SOTA method by 2.08%(0.66%), respectively, indicating that the proposed HPR has great generalization capability and strong robustness against perturbed data. Our code is available at: https://github.com/Jonathan-UCAS/HPR.

Spatial attention for human-centric visual understanding: An Information Bottleneck method

The selective visual attention mechanism in the Human Visual System (HVS) restricts the amount of information that reaches human visual awareness, allowing the brain to perceive high-fidelity natural scenes in real-time with limited computational cost. This selectivity acts as an “Information Bottleneck (IB)” that balances information compression and predictive accuracy. However, such information constraints are rarely explored in the attention mechanism for deep neural networks (DNNs). This paper introduces an IB-inspired spatial attention module for DNNs, which generates an attention map by minimizing the mutual information (MI) between the attentive content and the input while maximizing that between the attentive content and the output. We develop this IB-inspired attention mechanism based on a novel graphical model and explore various implementations of the framework. We show that our approach can yield attention maps that neatly highlight the regions of interest while suppressing the backgrounds, and are interpretable for the decision-making of the DNNs. To validate the effectiveness of the proposed IB-inspired attention mechanism, we apply it to various computer vision tasks including image classification, fine-grained recognition, cross-domain classification, semantic segmentation, and object detection. Extensive experiments demonstrate that it bootstraps standard DNN structures quantitatively and qualitatively for these tasks.

A comprehensive survey on contrastive learning

Contrastive Learning is self-supervised representation learning by training a model to differentiate between similar and dissimilar samples. It has been shown to be effective and has gained significant attention in various computer vision and natural language processing tasks. In this paper, we comprehensively and systematically sort out the main ideas, recent developments and application areas of contrastive learning. Specifically, we firstly provide an overview of the research activity of contrastive learning in recent years. Secondly, we describe the basic principles and summarize a universal framework of contrastive learning. Thirdly, we further introduce and discuss the latest advances of each functional component in detail, including data augmentation, positive/negative samples,network structure, and loss function. Finally, we summarize contrastive learning and discuss the challenges, future research trends and development directions in the area of contrastive learning.

Specular highlight removal using Quaternion transformer

Specular highlight removal is a very important issue, because specular highlight reflections in images with illumination changes can give very negative effects on various computer vision and image processing tasks. Numerous state-of-the-art networks for the specular removal use convolutional neural networks (CNN), which cannot learn global context effectively. They capture spatial information while overlooking 3D structural correlation information of an RGB image. To address this problem, we introduce a specular highlight removal network based on Quaternion transformer (QformerSHR), which employs a transformer architecture based on Quaternion representation. In particular, a depth-wise separable Quaternion convolutional layer (DSQConv) is proposed to enhance computational performance of QformerSHR, while efficiently preserving the structural correlation of an RGB image by utilizing the Quaternion representation. In addition, a Quaternion transformer block (QTB) based on DSQConv learns global context. As a result, QformerSHR consisting of DSQConv and QTB performs the specular removal from natural and text image datasets effectively. Experimental results demonstrate that it is significantly more effective than state-of-the-art networks for the specular removal, in terms of both quantitative performance and subjective quality.