Vision Research Research Articles

Multi-scale image recognition strategy based on convolutional neural network

The accurate recognition and interpretation of multi-scale visual information is a critical focus within contemporary computer vision research. To this end, this study explores and innovatively constructs a multi-scale image recognition strategy based on a Convolutional Neural Network (CNN) with a multi-level and multi-resolution perception domain. This strategy is embedded with an advanced multi-level convolutional operation mechanism, which enables the model to intelligently explore and learn the multi-scale feature representation space of images from tiny texture to grand structure, from shallow simple features to deep semantic abstraction. The core technology path of this paper is to design a deep separable convolutional architecture and combine pyramid pool technology to form a unique network module. This modular design not only ensures the computational efficiency of the model but also improves the ability of extracting and integrating multi-scale image features. Following intensive experimentation on an array of extensively recognized and substantial image datasets, the multi-scale image recognition approach introduced in our study has demonstrated marked enhancements in both recognition capability and stability, manifesting clear superiority compared to conventional, single-scale image recognition methodologies. This research not only enriches the theoretical framework of image recognition, but also provides a new and efficient solution for dealing with complex multi-scale image recognition challenges in practical applications, and further promotes the development of image understanding and recognition technology.

The effect of processing partial information in dynamic face perception

Face perception is a major topic in vision research. Most previous research has concentrated on (holistic) spatial representations of faces, often with static faces as stimuli. However, faces are highly dynamic stimuli containing important temporal information. How sensitive humans are regarding temporal information in dynamic faces is not well understood. Studies investigating temporal information in dynamic faces usually focus on the processing of emotional expressions. However, faces also contain relevant temporal information without any strong emotional expression. To investigate cues that modulate human sensitivity to temporal order, we utilized muted dynamic neutral face videos in two experiments. We varied the orientation of the faces (upright and inverted) and the presence/absence of eye blinks as partial dynamic cues. Participants viewed short, muted, monochromic videos of models vocalizing a widely known text (National Anthem). Videos were played either forward (in the correct temporal order) or backward. Participants were asked to determine the direction of the temporal order for each video, and (at the end of the experiment) whether they had understood the speech. We found that face orientation, and the presence/absence of an eye blink affected sensitivity, criterion (bias) and reaction time: Overall, sensitivity was higher for upright compared to inverted faces, and in the condition where an eye blink was present compared to the condition without an eye blink. Reaction times were mostly faster in the conditions with higher sensitivity. A bias to report inverted faces as ‘backward’ observed in Experiment I, where upright and inverted faces were presented randomly interleaved within each block, was absent when presenting upright and inverted faces in different blocks in Experiment II. Language comprehension results revealed that there was higher sensitivity when understanding the speech compared to not understanding the speech in both experiments. Taken together, our results showed higher sensitivity with upright compared to inverted faces, suggesting that the perception of dynamic, task-relevant information was superior with the canonical orientation of the faces. Furthermore, partial information coming from eye blinks, in addition to mouth movements, seemed to play a significant role in dynamic face perception, both when faces were presented upright and inverted. We suggest that studying the perception of facial dynamics beyond emotional expressions will help us to better understand the mechanisms underlying the temporal integration of facial information from different -partial and holistic- sources, and that our results show how different strategies, depending on the available information, are employed by human observers when judging the temporal order of faces.

Robust pedestrian detection via constructing versatile pedestrian knowledge bank

Pedestrian detection is a crucial field of computer vision research which can be adopted in various real-world applications (e.g., self-driving systems). However, despite noticeable evolution of pedestrian detection, pedestrian representations learned within a detection framework are usually limited to particular scene data in which they were trained. Therefore, in this paper, we propose a novel approach to construct versatile pedestrian knowledge bank containing representative pedestrian knowledge which can be applicable to various detection frameworks and adopted in diverse scenes. We extract generalized pedestrian knowledge from a large-scale pretrained model, and we curate them by quantizing most representative features and guiding them to be distinguishable from background scenes. Finally, we construct versatile pedestrian knowledge bank which is composed of such representations, and then we leverage it to complement and enhance pedestrian features within a pedestrian detection framework. Through comprehensive experiments, we validate the effectiveness of our method, demonstrating its versatility and outperforming state-of-the-art detection performances.

A Deep Dive into Robot Vision - An Integrative Systematic Literature Review Methodologies and Research Endeavor Practices

Novel technological swarm and industry 4.0 mold the recent Robot vision research into innovative discovery. To enhance technological paradigm Deep Learning offers remarkable pace to move towards diversified advancement. This research considers the most topical, recent, related and state-of-the-art research reviews that revolve around Robot vision, and shapes the research into Systematic Literature Survey SLR. The SLR considers a combination of more than 100 reviews and empirical studies to perform a critical categorical study and shapes findings against research questions. The research study contribution spans over multiple categories of Robot vision and is tinted along with technical limitations and future research endeavors. Previously multiple research studies have been observed to leverage Robotic vision techniques. Yet, there is none like SLR summarizing recent vision techniques for all targeted Robotic fields. This research SLR could be a precious milestone in Robot vision for each glimpse of Robotics.

Multimodal Convolutional Neural Networks for Sperm Motility and Concentration Predictions

Semen analysis is an important analysis for male infertility primary investigation and manual semen analysis is a conventional method to assess it. Manual semen analysis has been revealed with accuracy and precision limitations due to noncompliance to guidelines and procedures. Sperm motility and concentration are the main indicators for pregnancy and conception rate hence they were selected for parameters prediction. Convolutional neural network (CNN) has benefited computer vision application industry in recent years and has been widely applied in computer vision research tasks. In this paper, three-dimensional CNN (3DCNN) was designed to extract motion and temporal features, which are vital for sperm motility prediction. For sperm concentration, since two-dimensional CNN (2DCNN) is efficient in recognizing and extracting spatial features, well-established Residual Network (ResNet) architecture was adopted and customized for sperm concentration prediction. Multimodal learning approach is a technique to aggregate learnt features from different deep learning architecture that adopted other forms of modalities, which could provide deep learning model with better insights on their tasks. Hence, a multimodal learning deep learning architecture was designed to receive both image-based (frames extracted from video samples) and video-based (stacked frames pre-processed from video samples) input that could provide well-extracted spatial and temporal features for sperm parameters prediction. The results obtained using the proposed methodology have surpassed other similar research works who used deep learning approach. For sperm motility, its best achieved average mean absolute error (MAE) was 8.048, and sperm concentration obtained a competent Pearson’s correlation coefficient (RP) value of 0.853.

Deep neural network-based emotion recognition using facial landmark features and particle swarm optimization

Relating specifically to human–computer interaction (HCI), computer vision research has placed a substantial emphasis on intelligent emotion recognition in recent years. The primary emphasis lies in investigating speech aspects and bodily motions, while the knowledge of recognizing emotions from facial expressions remains relatively unexplored. Automated facial emotion detection allows a machine to assess and understand a person's emotional state, allowing the system to predict intent by analyzing facial expressions. Therefore, this research provides a novel parameter selection strategy using swarm intelligence and a fitness function for intelligent recognition of micro emotions. This paper presents a novel method based on geometric visual representation obtained from facial landmark points. We employ the Deep Neural Networks (DNN) model to analyze the input features from the normalized angle and distance values derived from these landmarks. The results of the experiments show that Particle Swarm Optimization (PSO) worked very well by using only a few carefully chosen features. The method achieved a recognition success rate of 98.76% on the MUG dataset and 97.79% on the GEMEP datasets.

Understanding of facial features in face perception: insights from deep convolutional neural networks.

Face recognition has been a longstanding subject of interest in the fields of cognitive neuroscience and computer vision research. One key focus has been to understand the relative importance of different facial features in identifying individuals. Previous studies in humans have demonstrated the crucial role of eyebrows in face recognition, potentially even surpassing the importance of the eyes. However, eyebrows are not only vital for face recognition but also play a significant role in recognizing facial expressions and intentions, which might occur simultaneously and influence the face recognition process. To address these challenges, our current study aimed to leverage the power of deep convolutional neural networks (DCNNs), an artificial face recognition system, which can be specifically tailored for face recognition tasks. In this study, we investigated the relative importance of various facial features in face recognition by selectively blocking feature information from the input to the DCNN. Additionally, we conducted experiments in which we systematically blurred the information related to eyebrows to varying degrees. Our findings aligned with previous human research, revealing that eyebrows are the most critical feature for face recognition, followed by eyes, mouth, and nose, in that order. The results demonstrated that the presence of eyebrows was more crucial than their specific high-frequency details, such as edges and textures, compared to other facial features, where the details also played a significant role. Furthermore, our results revealed that, unlike other facial features, the activation map indicated that the significance of eyebrows areas could not be readily adjusted to compensate for the absence of eyebrow information. This finding explains why masking eyebrows led to more significant deficits in face recognition performance. Additionally, we observed a synergistic relationship among facial features, providing evidence for holistic processing of faces within the DCNN. Overall, our study sheds light on the underlying mechanisms of face recognition and underscores the potential of using DCNNs as valuable tools for further exploration in this field.

Data generation using generative adversarial networks to increase data volume

The article is an in-depth analysis of two leading approaches in the field of generative modeling: generative adversarial networks (GANs) and the pixel-to-pixel (Pix2Pix) image translation model. Given the growing interest in automation and improved image processing, the authors focus on the key operating principles of each model, analyzing their unique characteristics and features. The article also explores in detail the various applications of these approaches, highlighting their impact on modern research in computer vision and artificial intelligence. The purpose of the study is to provide readers with a scientific understanding of the effectiveness and potential of each of the models, and to highlight the opportunities and limitations of their application. The authors strive not only to cover the technical aspects of the models, but also to provide a broad overview of their impact on various industries, including medicine, the arts, and solving real-world problems in image processing. In addition, we have identified prospects for the use of these technologies in various fields, such as medicine, design, art, entertainment, and in unmanned aerial vehicle systems. The ability of GANs and Pix2Pix to adapt to a variety of tasks and produce high-quality results opens up broad prospects for industry and research.

BrutNet: A novel approach for violence detection and classification using DCNN with GRU

AbstractAutomatic Violence Detection and Classification (AVDC) with deep learning has garnered significant attention in computer vision research. This paper presents a novel approach for combining a custom Deep Convolutional Neural Network (DCNN) with a Gated Recurrent Unit (GRU) in developing a new AVDC model called BrutNet. Specifically, a time‐distributed DCNN (TD‐DCNN) is developed to generate a compact 2D representation with 512 spatial features per frame from a set of equally‐spaced frames of dimension 16090 in short video segments. Further to leverage the temporal information, a GRU layer is utilised, generating a condensed 1D vector that enables binary classification of violent or non‐violent content through multiple dense layers. Overfitting is addressed by incorporating dropout layers with a rate of 0.5, while the hidden and output layers employ rectified linear unit (ReLU) and sigmoid activations, respectively. The model is trained on the NVIDIA Tesla K80 GPU through Google Colab, demonstrating superior performance compared to existing models across various video datasets, including hockey fights, movie fights, AVD, and RWF‐2000. Notably, the model stands out by requiring only 3.416 million parameters and achieving impressive test accuracies of 97.62%, 100%, 97.22%, and 86.43% on the respective datasets. Thus, BrutNet exhibits the potential to emerge as a highly efficient and robust AVDC model in support of greater public safety, content moderation and censorship, computer‐aided investigations, and law enforcement.

Ultimate pose estimation: A comparative study

AbstractPose estimation is a computer vision task used to detect and estimate the pose of a person or an object in images or videos. It has some challenges that can leverage advances in computer vision research and others that require efficient solutions. In this paper, we provide a preliminary review of the state‐of‐the‐art in pose estimation, including both traditional and deep learning approaches. Also, we implement and compare the performance of Hand Pose Estimation (HandPE), which uses PoseNet architecture for hand sign problems, for an ASL dataset by using different optimizers based on 10 common evaluation metrics on different datasets. Also, we discuss some related future research directions in the field of pose estimation and explore new architectures for pose estimation types. After applying the PoseNet model, the experiment results showed that the accuracy achieved was 99.9%, 89%, 97%, 79%, and 99% for the ASL alphabet, HARPET, Yoga, Animal, and Head datasets, comparing those with common optimizers and evaluation metrics on different dataset.

An Overview of Re-Identification

Pedestrian re-identification (Re-ID) is an emerging cutting-edge technology in the research area of intelligentvideo analysis in recent years, belonging to the category of image processing and analysis in complex video environments. This technology plays a crucial role in numerous monitoring and security applications and has been a hot topic in computer vision research. Pedestrian re-identification is considered an important sub-problem in image retrieval, which involves using computer vision algorithms to match pedestrian images or videos across devices to identify the same pedestrian from image databases of different monitoring devices. Research in this area can be traced back to the 1990s when researchers proposed various methods to address the challenge of pedestrian re-identification. This article summarizes the relevant research on personnel re-identification based on deep learning technology and its applications in different scenarios. Besides, it also identifies existing problems with this technology and its prospects.

MWSIS: Multimodal Weakly Supervised Instance Segmentation with 2D Box Annotations for Autonomous Driving

Instance segmentation is a fundamental research in computer vision, especially in autonomous driving. However, manual mask annotation for instance segmentation is quite time-consuming and costly. To address this problem, some prior works attempt to apply weakly supervised manner by exploring 2D or 3D boxes. However, no one has ever successfully segmented 2D and 3D instances simultaneously by only using 2D box annotations, which could further reduce the annotation cost by an order of magnitude. Thus, we propose a novel framework called Multimodal Weakly Supervised Instance Segmentation (MWSIS), which incorporates various fine-grained label correction modules for both 2D and 3D modalities, along with a new multimodal cross-supervision approach. In the 2D pseudo label generation branch, the Instance-based Pseudo Mask Generation (IPG) module utilizes predictions for self-supervised correction. Similarly, in the 3D pseudo label generation branch, the Spatial-based Pseudo Label Generation (SPG) module generates pseudo labels by incorporating the spatial prior information of the point cloud. To further refine the generated pseudo labels, the Point-based Voting Label Correction (PVC) module utilizes historical predictions for correction. Additionally, a Ring Segment-based Label Correction (RSC) module is proposed to refine the predictions by leveraging the depth prior information from the point cloud. Finally, the Consistency Sparse Cross-modal Supervision (CSCS) module reduces the inconsistency of multimodal predictions by response distillation. Particularly, transferring the 3D backbone to downstream tasks not only improves the performance of the 3D detectors, but also outperforms fully supervised instance segmentation with only 5% fully supervised annotations. On the Waymo dataset, the proposed framework demonstrates significant improvements over the baseline, especially achieving 2.59% mAP and 12.75% mAP increases for 2D and 3D instance segmentation tasks, respectively. The code is available at https://github.com/jiangxb98/mwsis-plugin.

Tools Identification By On-Board Adaptation of Vision-and-Language Models

A robotic workshop assistant has been a long-standing grand challenge for robotics, speech, computer vision, and artificial intelligence (AI) research. We revisit the goal of visual identification of tools from human queries in the current era of Large Vision-and-Language models (like GPT-4). We find that current off-the-shelf models (that are trained on internet images) are unable to overcome the domain shift and unable to identify small, obscure tools in cluttered environments. Furthermore, these models are unable to match tools to their intended purpose or affordances. We present a novel system for online domain adaptation that can be run directly on a small on-board processor. The system uses Hyperdimensional Computing (HD), a fast and efficient neuromorphic method. We adapted CLIP to work with explicit ("I need the hammer") and implicit purpose-driven queries ("Drive these nails"), and even with depth images as input. This demo allows the user to try out various real tools and interact via free-form audio.

Learning from an Infant’s Visual Experience

Infants see a selective view of the world: they see some objects with high frequency and from a wide range of viewpoints (e.g., their toys during playing) while a much larger set of objects are seen much more rarely and from limited viewpoints (e.g., objects they see outdoors). Extensive, repeated visual experiences with a small number of objects during infancy plays a big role in the development of human visual skills. Internet-style datasets that are commonly used in computer vision research do not contain the regularities that result from such repeated, structured experiences with a few objects. This has led to a dearth of models that learn by exploiting these regularities. In my PhD dissertation, I use deep learning models to investigate how regularities in an infant's visual experience can be leveraged for visual representation learning.

Towards Improved Proxy-Based Deep Metric Learning via Data-Augmented Domain Adaptation

Deep Metric Learning (DML) plays an important role in modern computer vision research, where we learn a distance metric for a set of image representations. Recent DML techniques utilize the proxy to interact with the corresponding image samples in the embedding space. However, existing proxy-based DML methods focus on learning individual proxy-to-sample distance, while the overall distribution of samples and proxies lacks attention. In this paper, we present a novel proxy-based DML framework that focuses on aligning the sample and proxy distributions to improve the efficiency of proxy-based DML losses. Specifically, we propose the Data-Augmented Domain Adaptation (DADA) method to adapt the domain gap between the group of samples and proxies. To the best of our knowledge, we are the first to leverage domain adaptation to boost the performance of proxy-based DML. We show that our method can be easily plugged into existing proxy-based DML losses. Our experiments on benchmarks, including the popular CUB-200-2011, CARS196, Stanford Online Products, and In-Shop Clothes Retrieval, show that our learning algorithm significantly improves the existing proxy losses and achieves superior results compared to the existing methods. The code and Appendix are available at: https://github.com/Noahsark/DADA

EarthVQA: Towards Queryable Earth via Relational Reasoning-Based Remote Sensing Visual Question Answering

Earth vision research typically focuses on extracting geospatial object locations and categories but neglects the exploration of relations between objects and comprehensive reasoning. Based on city planning needs, we develop a multi-modal multi-task VQA dataset (EarthVQA) to advance relational reasoning-based judging, counting, and comprehensive analysis. The EarthVQA dataset contains 6000 images, corresponding semantic masks, and 208,593 QA pairs with urban and rural governance requirements embedded. As objects are the basis for complex relational reasoning, we propose a Semantic OBject Awareness framework (SOBA) to advance VQA in an object-centric way. To preserve refined spatial locations and semantics, SOBA leverages a segmentation network for object semantics generation. The object-guided attention aggregates object interior features via pseudo masks, and bidirectional cross-attention further models object external relations hierarchically. To optimize object counting, we propose a numerical difference loss that dynamically adds difference penalties, unifying the classification and regression tasks. Experimental results show that SOBA outperforms both advanced general and remote sensing methods. We believe this dataset and framework provide a strong benchmark for Earth vision's complex analysis. The project page is at https://Junjue-Wang.github.io/homepage/EarthVQA.

PDTE: Pyramidal deep Taylor expansion for optical flow estimation

Optical flow estimation is an important hot research in computer vision. Although existing methods had got a considerable progress in improving their performance, they still have drawbacks, such as heavily computational burden, inaccurate pixel-level offset estimation, and poor interpretability. To address these issues, this letter proposes a pyramidal deep Taylor expansion (PDTE) framework, including: First, we seriously interpreted the relationship between optical flow computation and Taylor expansion. Then, the proposed PDTE is constructed by employing global motion aggregation (GMA) to calculate each derivative part which contributes to the final estimated optical flow. Quantitative and qualitative results on the Sintel and KITTI datasets validate that the proposed PDTE scheme is effective and outperforms the state-of-the-art optical flow estimation methods. The results of extensive experiments in the ablation study demonstrate that PDTE performs well on shape preservation and the accuracy improvement of optical flow estimation, even pixel-level offset calculation.

The Application and Development of Image Recognition Technology in the Field of Computer Vision

Image recognition is an important field of computer vision research. With the emergence of convolutional neural networks, image recognition has rapidly developed in many fields and has entered people's daily lives. This article reviews image recognition applications and future prospects in three representative fields: healthcare, intelligent driving, and security monitoring. In medical image analysis, image recognition technology can improve medical image interpretation speed and accuracy by automatically learning image features, providing doctors with more comprehensive information and decision-making support. Regarding intelligent driving, image recognition can help vehicles perceive and understand the traffic environment in real time, improving driving safety and automation level. In terms of security monitoring, image recognition plays a vital role in facial recognition, object detection, and behavior analysis, providing intelligent monitoring and security guarantees. However, these areas still have challenges, such as large data volumes, labeling difficulties, and privacy issues. In the future, people can look forward to combining image recognition technology with other technologies to improve models' interpretability and decision-making ability and further promote the development of these fields.

Enhancing Few-Shot Learning in Lightweight Models via Dual-Faceted Knowledge Distillation.

In recent computer vision research, the pursuit of improved classification performance often leads to the adoption of complex, large-scale models. However, the actual deployment of such extensive models poses significant challenges in environments constrained by limited computing power and storage capacity. Consequently, this study is dedicated to addressing these challenges by focusing on innovative methods that enhance the classification performance of lightweight models. We propose a novel method to compress the knowledge learned by a large model into a lightweight one so that the latter can also achieve good performance in few-shot classification tasks. Specifically, we propose a dual-faceted knowledge distillation strategy that combines output-based and intermediate feature-based methods. The output-based method concentrates on distilling knowledge related to base class labels, while the intermediate feature-based approach, augmented by feature error distribution calibration, tackles the potential non-Gaussian nature of feature deviations, thereby boosting the effectiveness of knowledge transfer. Experiments conducted on MiniImageNet, CIFAR-FS, and CUB datasets demonstrate the superior performance of our method over state-of-the-art lightweight models, particularly in five-way one-shot and five-way five-shot tasks.

Robust Object Detection Using Fire Hawks Optimizer with Deep Learning Model for Video Surveillance

In recent years, video surveillance has become an integral part of computer vision research, addressing a variety of challenges in security, memory management and content extraction from video sequences. This paper introduces the Robust Object Detection using Fire Hawks Optimizer with Deep Learning (ROD-FHODL) technique, a novel approach designed specifically for video surveillance applications. Combining object detection and classification the proposed technique employs a two-step procedure. Utilizing the power of the Mask Region-based Convolutional Neural Network (Mask-RCNN) for object detection, we optimize its hyperparameters using the Fire Hawks Optimizer (FHO) algorithm to improve its efficacy. Our experimental results on the UCSD dataset demonstrate the significant impact of the proposed work. It achieves an extraordinary RUNNT of 1.34[Formula: see text]s on the pedestrian-1 dataset, significantly outperforming existing models. In addition, the proposed system surpasses in accuracy, with a pedestrian-1 accuracy rate of 97.45% and Area Under the Curve (AUC) values of 98.92%. Comparative analysis demonstrates the superiority of the proposed system in True Positive Rate (TPR) versus False Positive Rate (FPR) across thresholds. In conclusion, the proposed system represents a significant advancement in video surveillance, offering advances in speed, precision and robustness that hold promise for enhancing security, traffic management and public space monitoring in smart city infrastructure and other applications.