Camera Sensor Data Research Articles

Motion and geometry-related information fusion through a framework for object identification from a moving camera in urban driving scenarios

Accurate and robust semantic scene understanding for urban driving is challenging due to the complex object types, object motion, and as well as ego-motion. Typical approaches to this problem use the fusion of multiple sensors such as camera, IMU, LiDAR, and radar to identify the state of the surrounding objects, including distance, direction, position, and velocity. However, such sensor modalities are very much complex and costly. This paper proposes a new framework for object identification (FOI) from a moving camera in a complex driving environment utilizing only camera sensor data. The framework is capable of detecting objects, extracting their behavioral features in terms of motion, position, velocity, and distance. All of this information (referred to as object-wise semantic information) are fused in order to acquire a better understanding of the driving scenario. The work addresses ego-motion compensation and extraction of accurate motion information of moving objects from a moving camera using image registration and optical flow estimation. A moving object detection model is designed within the framework by integrating an encoder–decoder network with a semantic segmentation network. The approach involves two mutual tasks: semantic segmentation of objects into specific classes and binary pixel classification to predict whether the detected object is moving or static based on temporal information. The work also contributes a novel dataset for moving object detection that covers all types of dynamic objects. The evaluation of FOI has been performed on different sequences of KITTI, EU-life long, and the proposed datasets. The experimental results show that the proposed framework provides accurate object-wise semantic information.

Agricultural Robot-Centered Recognition of Early-Developmental Pest Stage Based on Deep Learning: A Case Study on Fall Armyworm (Spodoptera frugiperda)

Accurately detecting early developmental stages of insect pests (larvae) from off-the-shelf stereo camera sensor data using deep learning holds several benefits for farmers, from simple robot configuration to early neutralization of this less agile but more disastrous stage. Machine vision technology has advanced from bulk spraying to precise dosage to directly rubbing on the infected crops. However, these solutions primarily focus on adult pests and post-infestation stages. This study suggested using a front-pointing red-green-blue (RGB) stereo camera mounted on a robot to identify pest larvae using deep learning. The camera feeds data into our deep-learning algorithms experimented on eight ImageNet pre-trained models. The combination of the insect classifier and the detector replicates the peripheral and foveal line-of-sight vision on our custom pest larvae dataset, respectively. This enables a trade-off between the robot's smooth operation and localization precision in the pest captured, as it first appeared in the farsighted section. Consequently, the nearsighted part utilizes our faster region-based convolutional neural network-based pest detector to localize precisely. Simulating the employed robot dynamics using CoppeliaSim and MATLAB/SIMULINK with the deep-learning toolbox demonstrated the excellent feasibility of the proposed system. Our deep-learning classifier and detector exhibited 99% and 0.84 accuracy and a mean average precision, respectively.

3D Instance Segmentation and Object Detection Framework Based on the Fusion of Lidar Remote Sensing and Optical Image Sensing

Since single sensor and high-density point cloud data processing have certain direct processing limitations in urban traffic scenarios, this paper proposes a 3D instance segmentation and object detection framework for urban transportation scenes based on the fusion of Lidar remote sensing technology and optical image sensing technology. Firstly, multi-source and multi-mode data pre-fusion and alignment of Lidar and camera sensor data are effectively carried out, and then a unique and innovative network of stereo regional proposal selective search-driven DAGNN is constructed. Finally, using the multi-dimensional information interaction, three-dimensional point clouds with multi-features and unique concave-convex geometric characteristics are instance over-segmented and clustered by the hypervoxel storage in the remarkable octree and growing voxels. Finally, the positioning and semantic information of significant 3D object detection in this paper are visualized by multi-dimensional mapping of the boundary box. The experimental results validate the effectiveness of the proposed framework with excellent feedback for small objects, object stacking, and object occlusion. It can be a remediable or alternative plan to a single sensor and provide an essential theoretical and application basis for remote sensing, autonomous driving, environment modeling, autonomous navigation, and path planning under the V2X intelligent network space– ground integration in the future.

Non-textured objects visual SLAM using pola-rization 3D reconstruction

The common visual SLAM technology is to obtain the scene image information through monocular and binocular cameras, and then process the camera sensor data to get the restored map. However, these camera sensors are not sensitive enough to non-textured scenes to restore the map. It is easy to cause large deviation in image acquisition and cannot restore the corresponding shape. Therefore, in order to address these challenges, this paper proposes a SLAM method that uses polarization camera to obtain polarization characteristics of the target and then calculate the depth information to realize image reconstruction. Specifically, in the SLAM process, we use a monocular polarization camera to acquire an image sequence, and the phase angle and normal vector of each pixel obtained by polarization are used to recover the depth of the textured object, then multi-view normal vector constraints are fused to carry out 3D reconstruction, and finally, in the process of depth propagation, polarized light field information is used to constrain the propagation of non-textured regions. We respectively verified the indoor and outdoor shooting data of single and multiple object scenes, and the reconstruction results are compared with the existing DSO, ORB-SLAM and other algorithms, which shows that the effect of our visual SLAM method is better than that of the existing common SLAM method in reconstructing the surface shape of non-textured objects.

Exploit Camera Raw Data for Video Super- Resolution via Hidden Markov Model Inference.

To the best of our knowledge, the existing deep-learning-based Video Super-Resolution (VSR) methods exclusively make use of videos produced by the Image Signal Processor (ISP) of the camera system as inputs. Such methods are 1) inherently suboptimal due to information loss incurred by non-invertible operations in ISP, and 2) inconsistent with the real imaging pipeline where VSR in fact serves as a pre-processing unit of ISP. To address this issue, we propose a new VSR method that can directly exploit camera sensor data, accompanied by a carefully built Raw Video Dataset (RawVD) for training, validation, and testing. This method consists of a Successive Deep Inference (SDI) module and a reconstruction module, among others. The SDI module is designed according to the architectural principle suggested by a canonical decomposition result for Hidden Markov Model (HMM) inference; it estimates the target high-resolution frame by repeatedly performing pairwise feature fusion using deformable convolutions. The reconstruction module, built with elaborately designed Attention-based Residual Dense Blocks (ARDBs), serves the purpose of 1) refining the fused feature and 2) learning the color information needed to generate a spatial-specific transformation for accurate color correction. Extensive experiments demonstrate that owing to the informativeness of the camera raw data, the effectiveness of the network architecture, and the separation of super-resolution and color correction processes, the proposed method achieves superior VSR results compared to the state-of-the-art and can be adapted to any specific camera-ISP. Code and dataset are available at https://github.com/proteus1991/RawVSR.

Procuring cooperative intelligence in autonomous vehicles for object detection through data fusion approach

In an autonomous vehicle (AV), in order to efficiently exploit the acquired resources, big data analyses will be a reliable source for extracting valuable information from various sensors and actuators. The data extracted with the combined ability of telematics and real-time investigation forms the vibrant asset for self-driving cars. To demonstrate the significances of big data analysis, this study proposes a competent architecture for real-time big data analysis for an AV, which indeed keeps pace with the latest trends and advancement concerning an emerging paradigm. There are a massive amount of sensors and independent systems needed to be realised for better competence in an AV, and the proposed model focuses on independent sensors that distinguish objects and handles visual information to decide the path. In order to attain the objective as mentioned above, a sensor fusion mechanism is proposed, which combines 3D camera sensor data and Lidar sensor information to provide an optimised solution for path selection. Furthermore, three algorithms, namely overlapping algorithm, sequential adding algorithm, the distance-focused algorithm is designed for higher efficiency in sensor fusion mechanism. The proposed methodology is for the best exploitation of the enormous dataset, meant for real-time processing for an AV.

Indoor human activity recognition using high-dimensional sensors and deep neural networks

Many smart home applications rely on indoor human activity recognition. This challenge is currently primarily tackled by employing video camera sensors. However, the use of such sensors is characterized by fundamental technical deficiencies in an indoor environment, often also resulting in a breach of privacy. In contrast, a radar sensor resolves most of these flaws and maintains privacy in particular. In this paper, we investigate a novel approach toward automatic indoor human activity recognition, feeding high-dimensional radar and video camera sensor data into several deep neural networks. Furthermore, we explore the efficacy of sensor fusion to provide a solution in less than ideal circumstances. We validate our approach on two newly constructed and published data sets that consist of 2347 and 1505 samples distributed over six different types of gestures and events, respectively. From our analysis, we can conclude that, when considering a radar sensor, it is optimal to make use of a three-dimensional convolutional neural network that takes as input sequential range-Doppler maps. This model achieves 12.22% and 2.97% error rate on the gestures and the events data set, respectively. A pretrained residual network is employed to deal with the video camera sensor data and obtains 1.67% and 3.00% error rate on the same data sets. We show that there exists a clear benefit in combining both sensors to enable activity recognition in the case of less than ideal circumstances.

The Effect of Real-time Headbox Adjustments on Data Quality.

Following a patent owned by Tobii, the framerate of a CMOS camera can be increased by reducing the size of the recording window so that it fits the eyes with minimum room to spare. The position of the recording window can be dynamically adjusted within the camera sensor area to follow the eyes as the participant moves the head. Since only a portion of the camera sensor data is communicated to the computer and processed, much higher framerates can be achieved with the same CPU and camera. Eye trackers can be expected to present data at a high speed, with good accuracy and precision, small latency and with minimal loss of data while allowing participants to behave as normally as possible. In this study, the effect of headbox adjustments in real-time is investigated with respect to the above-mentioned parameters. It was found that, for the specific camera model and tracking algorithm, one or two headbox adjustments per second, as would normally be the case during recording of human participants, could be tolerated in favour of a higher framerate. The effect of adjustment of the recording window can be reduced by using a larger recording window at the cost of the framerate.

Human-like Artificial Intelligent Wheelchair Robot Navigated by Multi-Sensor Models in Indoor Environments and Error Analysis

Intelligent mobile robot navigation in indoor environments is still a challenge. In this paper, we propose a method in which the wheelchair robot imitates human like navigation by interacting with the surrounding environments. Two types of sensor data are used to train neural networks, which are later used to control the robot to reach the goal location in different indoor environments. The robot navigates from the start to the goal location in the environments with obstacles. In first model, we used the Laser Range Finder (LRF) sensor data as input of the neural network. In the second model in addition to the LRF data, the processed camera sensor data are also utilized. We compare the performance of two neural networks models by analyzing the error between the human and neural network based real robot navigations. The experimental results show that our proposed models are efficient for mobile robot navigations. In addition, errors are analyzed in this paper.