Visual Perception Module Research Articles

A Wearable Visually Impaired Assistive System Based on Semantic Vision SLAM for Grasping Operation.

Because of the absence of visual perception, visually impaired individuals encounter various difficulties in their daily lives. This paper proposes a visual aid system designed specifically for visually impaired individuals, aiming to assist and guide them in grasping target objects within a tabletop environment. The system employs a visual perception module that incorporates a semantic visual SLAM algorithm, achieved through the fusion of ORB-SLAM2 and YOLO V5s, enabling the construction of a semantic map of the environment. In the human-machine cooperation module, a depth camera is integrated into a wearable device worn on the hand, while a vibration array feedback device conveys directional information of the target to visually impaired individuals for tactile interaction. To enhance the system's versatility, a Dobot Magician manipulator is also employed to aid visually impaired individuals in grasping tasks. The performance of the semantic visual SLAM algorithm in terms of localization and semantic mapping was thoroughly tested. Additionally, several experiments were conducted to simulate visually impaired individuals' interactions in grasping target objects, effectively verifying the feasibility and effectiveness of the proposed system. Overall, this system demonstrates its capability to assist and guide visually impaired individuals in perceiving and acquiring target objects.

VCAFusion: An infrared and visible image fusion network with visual perception and cross-scale attention

Infrared and visible image fusion methods aim to combine salient target instances and abundant texture details into fused images. However, due to the interference of harsh conditions, such as dense smoke, fog, and intense light, it is feasible for external interference information to be integrated into the fused image, which seriously affects the image quality. To this end, we propose an infrared and visible image fusion network with visual perception and cross-scale attention modules, termed VCAFusion, which integrates critical information from source images in harsh conditions more efficiently. Specifically, considering that the human eye can identify key information under adverse conditions, we design a visual perception module (VPM), guiding information integration from the perspective of human visual perception. In addition, we propose a cross-scale attention module (CSAM) based on shifted window cross-attention, which aims to capture the long-distance correlation between adjacent scale features, providing more accurate image information for image restoration. Experiments with a variety of datasets reveal that the VCAFusion can adaptively retain image information and further improve image generation ability in harsh conditions.

An effective method for small object detection in low-resolution images

Having a tiny scale and few identifiable features, small objects are particularly difficult to detect, especially when the image resolution is not high. Further, large-scale variation across object instances places a burden on small object detection. In view of the insufficient detection ability of YOLOF (You Only Look One-level Feature) detector for small objects at low resolution, we propose an effective method, named YOLOFs, to boost small object detection precision in the case of object instances scale variation. The proposed three key modules (feature fusion module, visual perception module and feature encoding module) expand the receptive field of the network and greatly improve small object detection precision. Extensive experiments on the COCO, VOC, and Fire datasets prove the effectiveness of our method while keeping the detector simple and accurate. Without bells and whistles, our method outperforms YOLOF 5.1 AP (average precision), 6.2 AP, and 4.9 AP on small object detection at 704×704,608×608, and 512 × 512 resolutions, respectively, and it also achieves better performance on the VOC and Fire datasets. Our code will be made publicly available.

Visual Perception Generalization for Vision-and-Language Navigation via Meta-Learning.

Vision-and-language navigation (VLN) is a challenging task that requires an agent to navigate in real-world environments by understanding natural language instructions and visual information received in real time. Prior works have implemented VLN tasks on continuous environments or physical robots, all of which use a fixed-camera configuration due to the limitations of datasets, such as 1.5-m height, 90° horizontal field of view (HFOV), and so on. However, real-life robots with different purposes have multiple camera configurations, and the huge gap in visual information makes it difficult to directly transfer the learned navigation skills between various robots. In this brief, we propose a visual perception generalization strategy based on meta-learning, which enables the agent to fast adapt to a new camera configuration. In the training phase, we first locate the generalization problem to the visual perception module and then compare two meta-learning algorithms for better generalization in seen and unseen environments. One of them uses the model-agnostic meta-learning (MAML) algorithm that requires few-shot adaptation, and the other refers to a metric-based meta-learning method with a feature-wise affine transformation (AT) layer. The experimental results on the VLN-CE dataset demonstrate that our strategy successfully adapts the learned navigation skills to new camera configurations, and the two algorithms show their advantages in seen and unseen environments respectively.

Monocular vision guided deep reinforcement learning UAV systems with representation learning perception

In recent years, numerous studies have applied deep reinforcement learning (DRL) algorithms to vision-guided unmanned aerial systems. However, DRL is not good at training deep networks in an end-to-end manner due to data inefficiency and lack of direct supervision signals. This paper provides a visual information dimension reduction scheme with representation learning as the visual perception module, which reduces the dimensions of high-dimensional visual information and retains its features related to UAV navigation. Combining such state representation learning with the DRL model can effectively reduce the complexity of the neural network required by DRL. Based on this scheme, we design three motion control models with a monocular camera as the main sensor and train them to control UAVs for obstacle avoidance tasks in a simulated environment. Experiments show that all these models achieve high obstacle avoidance ability after a certain period of training. In addition, one of them also enables the monocular vision guidance system to avoid obstacles in the blind spot of side vision.

VG-Swarm: A Vision-Based Gene Regulation Network for UAVs Swarm Behavior Emergence

We present VG-Swarm, a practical and effective method for aerial robots dynamic encirclement, which consists of a vision-based gene regulatory network (V-GRN) and a visual perception module. For each flying robot deployed with the proposed method, the relative spatial positions of the surrounding robots, targets, and obstacles are first obtained by omnidirectional monocular vision. Then the proposed method is used to generate the concentration field within its own perception range according to the obtained position information. The agent individually calculates and selects an optimal moving direction in its concentration field, and finally stays on its selected encirclement pattern (a closed concentration contour around the target). As a result, a swarm of flying robots can emerge adaptive pattern formations to entrap the targets even without any communication and global information. We verify the effectiveness and robustness of the proposed method in various simulations and real-world experiments.

Vision-Enhanced and Consensus-Aware Transformer for Image Captioning

Image captioning generates descriptions in a natural language for a given image. Due to its great potential for a wide range of applications, many deep learning based-methods have been proposed. The co-occurrence of words such as mouse and keyboard, constitutes commonsense knowledge, which is referred to as consensus. However, it is challenging to consider commonsense knowledge in producing captions that have rich, natural, and meaningful semantics. In this paper, a Vision-enhanced and Consensus-aware Transformer (VCT) is proposed to exploit both visual information and consensus knowledge for image captioning with three key components: a vision-enhanced encoder, consensus-aware knowledge representation generator, and consensus-aware decoder. The vision-enhanced encoder extends the vanilla self-attention module with a memory-based attention module and a visual perception module for learning better visual representation of an image. Specifically, the relationships between regions in an image and the image’s global context are leveraged with scene memory in the memory-based attention module. The visual perception module further enhances the correlation among neighboring tokens in both the spatial and channel-wise dimensions. To learn consensus-aware representations, a word correlation graph is constructed by computing the statistical co-occurrence between semantic concepts. Then consensus knowledge can be acquired using a graph convolutional network in the consensus-aware knowledge representation generator. Finally, such consensus knowledge is integrated into the consensus-aware decoder through consensus memory and a knowledge-based control module to produce a caption. Experimental results on two popular benchmark datasets (MSCOCO and Flickr30k) demonstrate that our proposed model achieves state-of-the-art performance. Extensive ablation studies also validate the effectiveness of each component.

Brain over Brawn: Using a Stereo Camera to Detect, Track, and Intercept a Faster UAV by Reconstructing the Intruder’s Trajectory

This paper presents our approach to intercepting a faster intruder UAV, inspired by the MBZIRC 2020 Challenge 1. By utilizing a priori knowledge of the shape of the intruder’s trajectory, we can calculate an interception point. Target tracking is based on image processing by a YOLOv3 Tiny convolutional neural network, combined with depth calculation using a gimbal-mounted ZED Mini stereo camera. We use RGB and depth data from the camera, devising a noise-reducing histogram-filter to extract the target’s 3D position. Obtained 3D measurements of target’s position are used to calculate the position, orientation, and size of a figure-eight shaped trajectory, which we approximate using a Bernoulli lemniscate. Once the approximation is deemed sufficiently precise, as measured by the distance between observations and estimate, we calculate an interception point to position the interceptor UAV directly on the intruder’s path. Our method, which we have significantly improved based on the experience gathered during the MBZIRC competition, has been validated in simulation and through field experiments. Our results confirm that we have developed an efficient, visual-perception module that can extract information describing the intruder UAV’s motion with precision sufficient to support interception planning. In a majority of our simulated encounters, we can track and intercept a target that moves 30% faster than the interceptor. Corresponding tests in an unstructured environment yielded 9 out of 12 successful results.

Real-Time Detection and Spatial Localization of Insulators for UAV Inspection Based on Binocular Stereo Vision

Unmanned aerial vehicles (UAVs) have become important tools for power transmission line inspection. Cameras installed on the platforms can efficiently obtain aerial images containing information about power equipment. However, most of the existing inspection systems cannot perform automatic real-time detection of transmission line components. In this paper, an automatic transmission line inspection system incorporating UAV remote sensing with binocular visual perception technology is developed to accurately detect and locate power equipment in real time. The system consists of a UAV module, embedded industrial computer, binocular visual perception module, and control and observation module. Insulators, which are key components in power transmission lines as well as fault-prone components, are selected as the detection targets. Insulator detection and spatial localization in aerial images with cluttered backgrounds are interesting but challenging tasks for an automatic transmission line inspection system. A two-stage strategy is proposed to achieve precise identification of insulators. First, candidate insulator regions are obtained based on RGB-D saliency detection. Then, the skeleton structure of candidate insulator regions is extracted. We implement a structure search to realize the final accurate detection of insulators. On the basis of insulator detection results, we further propose a real-time object spatial localization method that combines binocular stereo vision and a global positioning system (GPS). The longitude, latitude, and height of insulators are obtained through coordinate conversion based on the UAV’s real-time flight data and equipment parameters. Experiment results in the actual inspection environment (220 kV power transmission line) show that the presented system meets the requirement of robustness and accuracy of insulator detection and spatial localization in practical engineering.

Visual Perception and Modeling for Autonomous Apple Harvesting

Vision perception and modelling are the essential tasks of robotic harvesting in the unstructured orchard. This paper develops a framework of visual perception and modelling for robotic harvesting of fruits in the orchard environments. The developed framework includes visual perception, scenarios mapping, and fruit modelling. The Visual perception module utilises a deep-learning model to perform multi-purpose visual perception task within the working scenarios; The scenarios mapping module applies OctoMap to represent the multiple classes of objects or elements within the environment; The fruit modelling module estimates the geometry property of objects and estimates the proper access pose of each fruit. The developed framework is implemented and evaluated in the apple orchards. The experiment results show that visual perception and modelling algorithm can accurately detect and localise the fruits, and modelling working scenarios in real orchard environments. The $F_{1}$ score and mean intersection of union of visual perception module on fruit detection and segmentation are 0.833 and 0.852, respectively. The accuracy of the fruit modelling in terms of centre localisation and pose estimation are 0.955 and 0.923, respectively. Overall, an accurate visual perception and modelling algorithm are presented in this paper.

Visual Object Recognition and Pose Estimation Based on a Deep Semantic Segmentation Network

In recent years, deep learning-based object recognition algorithms become emerging in robotic vision applications. This paper addresses the design of a novel deep learning-based visual object recognition and pose estimation system for a robot manipulator to handle random object picking tasks. The proposed visual control system consists of a visual perception module, an object pose estimation module, a data argumentation module, and a robot manipulator controller. The visual perception module combines deep convolution neural networks (CNNs) and a fully connected conditional random field layer to realize an image semantic segmentation function, which can provide stable and accurate object classification results in cluttered environments. The object pose estimation module implements a model-based pose estimation method to estimate the 3D pose of the target for picking control. In addition, the proposed data argumentation module automatically generates training data for training the deep CNN. Experimental results show that the proposed scene segmentation method used in the data argumentation module reaches a high accuracy rate of 97.10% on average, which is higher than other state-of-the-art segment methods. Moreover, with the proposed data argumentation module, the visual perception module reaches an accuracy rate over than 80% and 72% in the case of detecting and recognizing one object and three objects, respectively. In addition, the proposed model-based pose estimation method provides accurate 3D pose estimation results. The average translation and rotation errors in the three axes are all smaller than 0.52 cm and 3.95 degrees, respectively. These advantages make the proposed visual control system suitable for applications of random object picking and manipulation.

Support relation analysis and decision making for safe robotic manipulation tasks

In this article, we describe an approach to address the issue of automatically building and using high-level symbolic representations that capture physical interactions between objects in static configurations. Our work targets robotic manipulation systems where objects need to be safely removed from piles that come in random configurations. We assume that a 3D visual perception module exists so that objects in the piles can be completely or partially detected. Depending on the outcome of the perception, we divide the issue into two sub-issues: (1) all objects in the configuration are detected; (2) only a subset of objects are correctly detected. For the first case, we use notions from geometry and static equilibrium in classical mechanics to automatically analyze and extract act and support relations between pairs of objects. For the second case, we use machine learning techniques to estimate the probability of objects supporting each other. Having the support relations extracted, a decision making process is used to identify which object to remove from the configuration so that an expected minimum cost is optimized. The proposed methods have been extensively tested and validated on data sets generated in simulation and from real world configurations for the scenario of unloading goods from shipping containers.

Simulation of Passenger Flows on Urban Rail Transit Platform based on Adaptive Agents

Modeling and simulation of passenger flows on urban rail transit platform is a key issue in improving operation efficiency and service of level of urban rail transit, which should consider architectural environment, facilities implementation, and transportation organization. To simulate this kind of passenger for planning or evaluation, 3-layer architecture adaptive agent model is proposed to simulate passenger microscopic behaviors, which is based on visual perception module, making-decisions module, and action execution module. In respect of perception of agents, we construct a neuron-model-based perception model of environmental crowding to examine how individual URT passengers on the move represent the visual information of environmental crowding. Then, we define rules for behaviors based on cognitive heuristics for making-decisions module, and propose a discrete rule for the updating of passenger movement state for action execution module. Based on modeling passenger behavior dynamics, a microscopic simulation model for complex passenger flows on urban rail transit platform is developed. As a case study, the passenger flows scenarios of an island platform of urban rail transit station are simulated. Simulation results show that boarding and alighting passengers demand and train departure frequency have a remarkable impact on the maximum number of assembling passengers on platform and efficiency of mustering and evacuating under given conditions of building environment and facilities.

Learning Styles Preferences Among Third Year Optometry Students of Universiti Kebangsaan Malaysia

This study is conducted to determine learning style preferences among optometry students from Universiti Kebangsaan Malaysia. In addition, differences in learning style in terms of gender and ethnicity will also be compared. Thirty-five respondents comprising of third year Optometry students were asked to answer the Learning Style Questionnaire by Honey and Mumford. They were asked to recall their learning styles for two modules; visual perception and binocular vision. The findings of this study showed that the students have similar learning styles for both modules. In visual perception module, the most dominant learning style is reflector (42.9%), followed by pragmatist (37.1%) and the least preferred is theorist (20%). Similarly, in binocular vision module, the most dominant style is reflector (54.3%), followed by pragmatist (34.3%) and theorist (11.4%). We found gender differences in learning styles where male students are dominantly pragmatist while female students are reflector. There is no difference in learning style in terms of ethnicity for reflector and pragmatist but theorist were found among Chinese students only. In conclusion, each student has their specific learning style. It is important that they understand the strength and weaknesses of their learning styles in order to achieve academic excellence.