Vision Community Research Articles

E2-VINS: An Event-Enhanced Visual–Inertial SLAM Scheme for Dynamic Environments

Simultaneous Localization and Mapping (SLAM) technology has garnered significant interest in the robotic vision community over the past few decades. The rapid development of SLAM technology has resulted in its widespread application across various fields, including autonomous driving, robot navigation, and virtual reality. Although SLAM, especially Visual–Inertial SLAM (VI-SLAM), has made substantial progress, most classic algorithms in this field are designed based on the assumption that the observed scene is static. In complex real-world environments, the presence of dynamic objects such as pedestrians and vehicles can seriously affect the robustness and accuracy of such systems. Event cameras, which use recently introduced motion-sensitive biomimetic sensors, efficiently capture scene changes (referred to as “events”) with high temporal resolution, offering new opportunities to enhance VI-SLAM performance in dynamic environments. Integrating this kind of innovative sensor, we propose the first event-enhanced Visual–Inertial SLAM framework specifically designed for dynamic environments, termed E2-VINS. Specifically, the system uses visual–inertial alignment strategy to estimate IMU biases and correct IMU measurements. The calibrated IMU measurements are used to assist in motion compensation, achieving spatiotemporal alignment of events. The event-based dynamicity metrics, which measure the dynamicity of each pixel, are then generated on these aligned events. Based on these metrics, the visual residual terms of different pixels are adaptively assigned weights, namely, dynamicity weights. Subsequently, E2-VINS jointly and alternately optimizes the system state (camera poses and map points) and dynamicity weights, effectively filtering out dynamic features through a soft-threshold mechanism. Our scheme enhances the robustness of classic VI-SLAM against dynamic features, which significantly enhances VI-SLAM performance in dynamic environments, resulting in an average improvement of 1.884% in the mean position error compared to state-of-the-art methods. The superior performance of E2-VINS is validated through both qualitative and quantitative experimental results. To ensure that our results are fully reproducible, all the relevant data and codes have been released.

Research Advanced in Image Recognition under Autonomous Driving Scene Based on Deep Learning

Image recognition has always been a fundamental research task in the computer vision community, aimed at identifying the categories of objects in images and has been widely used in many fields, especially in autonomous driving. Early image recognition technologies were mostly based on machine learning, and their recognition speed and accuracy could not meet the application requirements of complex autonomous driving scenarios. With the great success of convolutional neural networks, image recognition technology based on deep learning has attracted increasing research interest. Taking the autonomous driving scenario as an example, this article introduces the latest research progress of image recognition technology, including representative methods and their basic pipelines. In addition, this paper also introduces the commonly used dataset in image recognition and discusses the existing problems of image recognition in autonomous driving tasks. Finally, this paper looks forward to the future development directions of this field, hoping bring some new insight to advance the further development of image recognition under autonomous driving scene.

Research Advanced in Image Inpainting based on Deep Learning

Image inpainting has always been a hotspot in the computer vision community, which aims to restore damaged or missing parts of an image, ensuring that the restored image is physically reasonable, visually pleasing, and consistent in texture and structure with the original image. Early inpainting methods primarily depend on diverse image processing technologies, such as sample replication, partial differential equation solving, etc. As artificial intelligence rapidly progresses, image inpainting methods based on deep learning exploit robust feature extraction capabilities to extract useful information from vast amounts of data, contributing to more robust and natural inpainting effects, especially when dealing with complex and high-dimensional scenes. This paper provides a review of deep learning-based image inpainting technologies, categorized into four main types: autoencoders, generative adversarial networks (GANs), recurrent neural networks (RNNs), and convolutional neural networks (CNNs). It summarizes and analyzes the basic principles, advantages, disadvantages, and performance of these methods in image inpainting, as well as introduces commonly used datasets and quantitative evaluation metrics. Additionally, the paper discusses the principal challenges currently faced in this domain and provides insights into future research directions for deep learning-based image inpainting.

Research Advanced in Object Detection based on Deep Learning

Object detection has always been a fundamental research topic in the computer vision community, which focuses on predicting the category and location of all objects in the scene. In last several years, progressing from the rapid development of deep learning, the speed and accuracy of general object detection methods have also achieved significant breakthroughs. This paper aims to report the latest research progress in the field of object detection based on deep learning to inspire and promote subsequent research. Specifically, this paper systematically introduces the research progress of predecessors from four aspects: dual-stage, single-stage, Transformer-based and key point, including the design ideas and basic processes of representative algorithms. In addition, this paper also quantitatively compares the performance of different methods on common data sets to further distinguish the benefits and disadvantages of different categories of methods. Finally, this paper summarizes the challenges that still exist in the field of object detection and looks forward to future development directions.

TRANSFORMATIONAL LEADERSHIP IN ISLAMIC EDUCATION INSTITUTION THROUGH SOCIAL MEDIA ENGAGEMENT

Transformational leadership in Islamic value-based educational institutions, such as Rumah Tahfidz Qur'an (RTQ), faces challenges in integrating traditional practices with modern innovations to improve academic quality. This study explores the effectiveness of transformational leadership in RTQ Surau Lakuak, focusing on the strategic use of social media. Using a descriptive qualitative approach, the research involves in-depth interviews and document analysis, guided by Kouzes and Posner's (2017) Leadership Challenge framework, which includes five core dimensions: Model the Way, Inspire a Shared Vision, Challenge the Process, Enable Others to Act, and Encourage the Heart. The findings indicate that leaders at RTQ Surau Lakuak effectively utilize social media to articulate their vision, enhance transparency, and foster community engagement. Key initiatives, such as digital donation campaigns, symbolic recognition, and community empowerment programs, have strengthened collaboration and ensured the sustainability of Qur'anic education programs. These practices are grounded in Islamic principles like ta'awun (cooperation) and deliberation, providing an adaptable leadership model for contemporary Islamic education. The study highlights the potential of applying the Leadership Challenge framework to address digital-era challenges, offering a replicable and innovative strategy for Islamic educational institutions in diverse contexts.

SEMPNet: enhancing few-shot remote sensing image semantic segmentation through the integration of the segment anything model

ABSTRACT Few-shot semantic segmentation has attracted increasing attention due to its potential for low dependence on annotated samples. While extensively explored in the computer vision community, these techniques are primarily designed for natural images, resulting in limited generalization to remote sensing images. In contrast to the mostly individual and distinct objects presented in natural images, remote sensing images often feature clustered and regular patterns of objects. To bridge this gap, we propose a novel approach for few-shot remote sensing image semantic segmentation, which takes into account the specific characteristics of remote sensing imagery. Our approach introduces a mask classification pipeline, which initially extracts all independent objects within an image and subsequently assigns specific categories to each object guided by semantic information derived from few support images. To accomplish this, a robust mask extractor is imperative. Fortunately, the impressive segment anything model (SAM) possesses the potential to fulfill this role. Leveraging its remarkable zero-shot segmentation capabilities, we present the SAM-enhanced mask parsing network (SEMPNet), a novel few-shot remote sensing image semantic segmentation model. The method generates a set of masks for each image using SAM, transforming the segmentation task into a mask classification problem. To precisely classify each mask, we calculate pixel-wise correlations between each mask and the support features through cross-image position attention. Finally, a mask parsing module is utilized to decode the correlation maps and generate the segmentation results. The experiments on two remote sensing datasets testify the superiority of our method. Our code will be available at https://github.com/TinyAway/SEMPNet.

Deep Learning for Visual Localization and Mapping: A Survey.

Deep-learning-based localization and mapping approaches have recently emerged as a new research direction and receive significant attention from both industry and academia. Instead of creating hand-designed algorithms based on physical models or geometric theories, deep learning solutions provide an alternative to solve the problem in a data-driven way. Benefiting from the ever-increasing volumes of data and computational power on devices, these learning methods are fast evolving into a new area that shows potential to track self-motion and estimate environmental models accurately and robustly for mobile agents. In this work, we provide a comprehensive survey and propose a taxonomy for the localization and mapping methods using deep learning. This survey aims to discuss two basic questions: whether deep learning is promising for localization and mapping, and how deep learning should be applied to solve this problem. To this end, a series of localization and mapping topics are investigated, from the learning-based visual odometry and global relocalization to mapping, and simultaneous localization and mapping (SLAM). It is our hope that this survey organically weaves together the recent works in this vein from robotics, computer vision, and machine learning communities and serves as a guideline for future researchers to apply deep learning to tackle the problem of visual localization and mapping.

Geometric Understanding of Discriminability and Transferability for Visual Domain Adaptation.

To overcome the restriction of identical distribution assumption, invariant representation learning for unsupervised domain adaptation (UDA) has made significant advances in computer vision and pattern recognition communities. In UDA scenario, the training and test data belong to different domains while the task model is learned to be invariant. Recently, empirical connections between transferability and discriminability have received increasing attention, which is the key to understand the invariant representations. However, theoretical study of these abilities and in-depth analysis of the learned feature structures are unexplored yet. In this work, we systematically analyze the essentials of transferability and discriminability from the geometric perspective. Our theoretical results provide insights into understanding the co-regularization relation and prove the possibility of learning these abilities. From methodology aspect, the abilities are formulated as geometric properties between domain/cluster subspaces (i.e., orthogonality and equivalence) and characterized as the relation between the norms/ranks of multiple matrices. Two optimization-friendly learning principles are derived, which also ensure some intuitive explanations. Moreover, a feasible range for the co-regularization parameters is deduced to balance the learning of geometric structures. Based on the theoretical results, a geometry-oriented model is proposed for enhancing the transferability and discriminability via nuclear norm optimization. Extensive experiment results validate the effectiveness of the proposed model in empirical applications, and verify that the geometric abilities can be sufficiently learned in the derived feasible range.

NDDepth: Normal-Distance Assisted Monocular Depth Estimation and Completion.

Over the past few years, monocular depth estimation and completion have been paid more and more attention from the computer vision community because of their widespread applications. In this paper, we introduce novel physics (geometry)-driven deep learning frameworks for these two tasks by assuming that 3D scenes are constituted with piece-wise planes. Instead of directly estimating the depth map or completing the sparse depth map, we propose to estimate the surface normal and plane-to-origin distance maps or complete the sparse surface normal and distance maps as intermediate outputs. To this end, we develop a normal-distance head that outputs pixel-level surface normal and distance. Afterthat, the surface normal and distance maps are regularized by a developed plane-aware consistency constraint, which are then transformed into depth maps. Furthermore, we integrate an additional depth head to strengthen the robustness of the proposed frameworks. Extensive experiments on the NYU-Depth-v2, KITTI and SUN RGB-D datasets demonstrate that our method exceeds in performance prior state-of-the-art monocular depth estimation and completion competitors.

Towards Context-Aware Emotion Recognition Debiasing From a Causal Demystification Perspective via De-Confounded Training.

Understanding emotions from diverse contexts has received widespread attention in computer vision communities. The core philosophy of Context-Aware Emotion Recognition (CAER) is to provide valuable semantic cues for recognizing the emotions of target persons by leveraging rich contextual information. Current approaches invariably focus on designing sophisticated structures to extract perceptually critical representations from contexts. Nevertheless, a long-neglected dilemma is that a severe context bias in existing datasets results in an unbalanced distribution of emotional states among different contexts, causing biased visual representation learning. From a causal demystification perspective, the harmful bias is identified as a confounder that misleads existing models to learn spurious correlations based on likelihood estimation, limiting the models' performance. To address the issue, we embrace causal inference to disentangle the models from the impact of such bias, and formulate the causalities among variables in the CAER task via a customized causal graph. Subsequently, we present a Contextual Causal Intervention Module (CCIM) to de-confound the confounder, which is built upon backdoor adjustment theory to facilitate seeking approximate causal effects during model training. As a plug-and-play component, CCIM can easily integrate with existing approaches and bring significant improvements. Systematic experiments on three datasets demonstrate the effectiveness of our CCIM.

MVImgNet2.0: A Larger-scale Dataset of Multi-view Images

MVImgNet is a large-scale dataset that contains multi-view images of ~220k real-world objects in 238 classes. As a counterpart of ImageNet, it introduces 3D visual signals via multi-view shooting, making a soft bridge between 2D and 3D vision. This paper constructs the MVImgNet2.0 dataset that expands MVImgNet into a total of ~520k objects and 515 categories, which derives a 3D dataset with a larger scale that is more comparable to ones in the 2D domain. In addition to the expanded dataset scale and category range, MVImgNet2.0 is of a higher quality than MVImgNet owing to four new features: (i) most shoots capture 360° views of the objects, which can support the learning of object reconstruction with completeness; (ii) the segmentation manner is advanced to produce foreground object masks of higher accuracy; (iii) a more powerful structure-from-motion method is adopted to derive the camera pose for each frame of a lower estimation error; (iv) higher-quality dense point clouds are reconstructed via advanced methods for objects captured in 360 ° views, which can serve for downstream applications. Extensive experiments confirm the value of the proposed MVImgNet2.0 in boosting the performance of large 3D reconstruction models. MVImgNet2.0 will be public at luyues.github.io/mvimgnet2 , including multi-view images of all 520k objects, the reconstructed high-quality point clouds, and data annotation codes, hoping to inspire the broader vision community.

Efficient bundle optimization for accurate camera pose estimation in mobile augmented reality systems

Augmented reality has a long research history in computer vision and computer graphics communities. It aims to enhance the user experience for real scenes via overlapping virtual objects. Nowadays, mobile augmented reality has attracted much attention from researchers and developers due to the development of hardware techniques. Modern mobile devices such as mobile phones have a powerful computational ability for augmented reality applications. As a result, many researchers have paid attention to mobile augmented reality. From the technical viewpoint of augmented reality, mobile augmented reality largely depends on camera pose estimation. However, existing methods make it difficult to achieve the best balance between accuracy and efficiency, according to our investigation, and this may handicap the performance of mobile augmented reality systems. To overcome the problem, in this paper, we propose a novel approach to camera pose estimation based on bundle optimization. Our proposed method is evaluated on real-world datasets and is also tested in the mobile augmented reality system. Both experiments demonstrate that our proposed method has fast speed and high accuracy.

Pan-Mamba: Effective pan-sharpening with state space model

Pan-sharpening involves integrating information from low-resolution multi-spectral and high-resolution panchromatic images to generate high-resolution multi-spectral counterparts. While recent advancements in the state space model, particularly the efficient long-range dependency modeling achieved by Mamba, have revolutionized computer vision community, its untapped potential in pan-sharpening motivates our exploration. Our contribution, Pan-Mamba, represents a novel pan-sharpening network that leverages the efficiency of the Mamba model in global information modeling. In Pan-Mamba, we customize two core components: channel swapping Mamba and cross-modal Mamba, strategically designed for efficient cross-modal information exchange and fusion. The former initiates a lightweight cross-modal interaction through the exchange of partial panchromatic and multi-spectral channels, while the latter facilities the information representation capability by exploiting inherent cross-modal relationships. Through extensive experiments across diverse datasets, our proposed approach surpasses state-of-the-art methods, showcasing superior fusion results in pan-sharpening. To the best of our knowledge, this work is the first attempt in exploring the potential of the Mamba model and establishes a new frontier in the pan-sharpening techniques. The source code is available at https://github.com/alexhe101/Pan-Mamba.

Associations between School-Based Vision Program Outcomes and School Characteristics in 410 Schools.

School-based vision programs (SBVPs) deliver vision care services directly to students at school, helping address disparities in access to pediatric vision care. We aim to evaluate the associations between SBVP outcomes and school-level characteristics. Retrospective cross-sectional data analysis PARTICIPANTS: Public schools with at least 50 SBVP-enrolled students 5 to 22 years old with complete demographic data. Schools with fewer than 60% of total grade levels served by the SBVP were excluded, resulting in a sample of 410 schools. Vision screening and eye exam data were extracted from Helen Keller International's United States Vision Program dataset from 2016 through 2022. Individual student data were aggregated to characterize each school's SBVP outcomes and analyzed with schools' publicly available socioeconomic and demographic data (student body race and ethnicity composition, proportion of students qualifying for free and reduced-price meals [FARM%], and proportion of English Language Learners). Descriptive statistics summarized SBVP outcomes. Fractional regression models were utilized to understand associations between SBVP outcomes and school characteristics. SBVP outcomes were rates of vision screening failure, eyeglasses prescription, and community eye care referral among each school's SBVP-enrolled students. We evaluated 151 (36.8%) elementary, 155 (37.8%) middle, and 104 (25.4%) high schools, with a median FARM% of 87.4% and a plurality of Hispanic students in 61.0% of schools. Overall median rates of vision screening failure, eyeglasses prescription, and referral were 38.4%, 25.2%, and 5.4%, respectively. High schools were associated with increased screening failure and eyeglasses prescription rates and a decrease in referral rate, compared with elementary schools. In multivariate analysis, each 10% increase in FARM% was associated with a 2.6% (95% confidence interval [CI]: 1.54-3.65), 1.8% (CI: 0.87-2.74), and 0.86% (CI: 0.36-1.36) increase in screening failure, eyeglasses prescription, and referral rates, respectively. There is significant vision care demand among public schools, especially those with student populations from lower socioeconomic backgrounds. SBVPs are important in bridging the gap in pediatric vision care access in such populations. Our findings demonstrated opportunities to allocate program staffing and equipment resources according to schools' anticipated needs, thus maximizing SBVPs' impact in delivering pediatric vision care.

Highly Efficient Active Learning With Tracklet-Aware Co-Cooperative Annotators for Person Re-Identification.

Supervised person re-identification (ReID) has attracted widespread attentions in the computer vision community due to its great potential in real-world applications. However, the demand of human annotation heavily limits the application as it is costly to annotate identical pedestrians appearing from different cameras. Thus, how to reduce the annotation cost while preserving the performance remains challenging and has been studied extensively. In this article, we propose a tracklet-aware co-cooperative annotators' framework to reduce the demand of human annotation. Specifically, we partition the training samples into different clusters and associate adjacent images in each cluster to produce the robust tracklet which decreases the annotation requirements significantly. Besides, to further reduce the cost, we introduce a powerful teacher model in our framework to implement the active learning strategy and select the most informative tracklets for human annotator, the teacher model itself, in our setting, also acts as an annotator to label the relatively certain tracklets. Thus, our final model could be well-trained with both confident pseudo-labels and human-given annotations. Extensive experiments on three popular person ReID datasets demonstrate that our approach could achieve competitive performance compared with state-of-the-art methods in both active learning and unsupervised learning (USL) settings.

ECTFormer: An efficient Conv-Transformer model design for image recognition

Since the success of Vision Transformers (ViTs), there has been growing interest in combining ConvNets and Transformers in the computer vision community. While the hybrid models have demonstrated state-of-the-art performance, many of these models are too large and complex to be applied to edge devices for real-world applications. To address this challenge, we propose an efficient hybrid network called ECTFormer that leverages the strengths of ConvNets and Transformers while considering both model performance and inference speed. Specifically, our approach involves: (1) optimizing the combination of convolution kernels by dynamically adjusting kernel sizes based on the scale of feature tensors; (2) revisiting existing overlapping patchify to not only reduce the model size but also propagate fine-grained patches for the performance enhancement; and (3) introducing an efficient single-head self-attention mechanism, rather than multi-head self-attention in the base Transformer, to minimize the increase in model size and boost inference speed, overcoming bottlenecks of ViTs. In experimental results on ImageNet-1K, ECTFormer not only demonstrates comparable or higher top-1 accuracy but also faster inference speed on both GPUs and edge devices compared to other efficient networks.

Multi-Camera Lighting Estimation for Mobile Augmented Reality

Lighting estimation is an important and long-standing task in computer vision and graphics communities[1]. Over the past few years, we have observed a rising demand for creating immersive user experiences in mobile augmented reality (AR) applications. In particular, environment lighting estimation is essential to render visually coherent virtual objects that need to be overlaid on physical scenes in mobile AR. In this work, we explore the key research questions in lighting understanding for an emerging application domain, mobile AR on multi-camera handheld devices, particularly try-on applications where end users leverage handheld mobile devices, such as smartphones, to overlay products of interest on their faces. Mobile AR try-on apps promote online shopping and often require photorealism to provide user experiences on par with physical try-on.

DCSS-UNet: UNet based on State Space Model for Polyp Segmentation

Early and accurate segmentation of medical images can provide valuable information for medical treatment. In recent years, the automatic and accurate segmentation of polyps in colonoscopy images has received extensive attention from the research community of artificial intelligence and computer vision. Many researchers have conducted in-depth research on models based on CNN and Transformer. However, CNN have limited ability to model remote dependencies, which makes it challenging to fully utilize semantic information in images. On the other hand, the complexity of the secondary computation poses a challenge to the transformer. Recently, state-space models (SSMS), such as Mamba, have been recognized as a promising approach. They not only show superior performance in remote interaction, but also maintain linear computational complexity. Inspired by Mamba, we propose DCSS-UNet, where we utilize visual state space (VSS) blocks in VMamba to capture a wide range of contextual information. In the Skip connection phase, we propose Skip Connects Feature Attention modules(SFA) to better communicate information from the encoder. In the decoder stage, we innovatively combined the Temporal Fusion Attention Module(TFAM) to effectively fuse the feature information. In addition, we introduced a custom Loss calculation method, Tversky Loss, for the model to achieve faster convergence and improve segmentation along polyp boundaries. Our model was trained on the Kvasir-SEG and CVC-ClinicDB datasets, and validated on datasets Kvasir-SEG, CVC-ColonDB, CVC-300, and ETIS. The results show that the model achieves good segmentation accuracy and generalization performance with a low number of parameters. We are 6.1% ahead in the Kavirs-SEG dataset and 3.1% ahead in the CVC-ClinicDB dataset compared to VM-UNet.

From Simple to Complex Scenes: Learning Robust Feature Representations for Accurate Human Parsing.

Human parsing has attracted considerable research interest due to its broad potential applications in the computer vision community. In this paper, we explore several useful properties, including high-resolution representation, auxiliary guidance, and model robustness, which collectively contribute to a novel method for accurate human parsing in both simple and complex scenes. Starting from simple scenes: we propose the boundary-aware hybrid resolution network (BHRN), an advanced human parsing network. BHRN utilizes deconvolutional layers and multi-scale supervision to generate rich high-resolution representations. Additionally, it includes an edge perceiving branch designed to enhance the fineness of part boundaries. Building on BHRN, we construct a dual-task mutual learning (DTML) framework. It not only provides implicit guidance to assist the parser by incorporating boundary features, but also explicitly maintains the high-order consistency between the parsing prediction and the ground truth. Toward complex scenes: we develop a domain transform method to enhance the model robustness. By transforming the input space from the spatial domain to the polar harmonic Fourier moment domain, the mapping relationship to the output semantic space is highly stable. This transformation yields robust representations for both clean and corrupted data. When evaluated on standard benchmark datasets, our method achieves superior performance compared to state-of-the-art human parsing methods. Furthermore, our domain transform strategy significantly improves the robustness of DTML dramatically in most complex scenes.

A branched Convolutional Neural Network for RGB-D image classification of ceramic pieces

From smart sensors on assembly lines to robots performing complex tasks, the fourth industrial revolution is rapidly transforming manufacturing. The growing prominence of 3D cameras in the industry has led the computer vision community to explore innovative ways of integrating depth and color data to achieve higher precision, essential for ensuring product quality in manufacturing. In this study, we introduce an innovative branched convolutional neural network designed to produce high-speed classification of multimodal images, such as RGB-Depth (RGB-D) images. The fundamental concept underlying the branched approach is the specialization of each branch as a dedicated feature extractor for a single modality, followed by their merge (intermediate fusion) to enable effective classification. Feeding our model is our novel multimodal dataset, named CeramicNet, composed of 8 classes that include RGB, depth, and RGB-D variations to enable extensive experimentation and evaluation of the models which, to the best of our knowledge, has not been previously introduced in the computer vision community. We conducted a series of experiments on the CeramicNet dataset. These experiments aimed at fine-tuning the model, assessing the influence of various depth technologies, exploring individual modalities, examining their collective impact, and performing comprehensive data analysis. Comparing our solution against seven widely used models, we achieved remarkable results, securing the top position with a precision of 99.89, with a lead of over 1% against the nearest competitor. What is more, the proposed solution yields an inference time of 127.6 ms — being nearly three times faster than the second-best performer.