Real-time Scene Research Articles

Parametric optimization of unmanned vehicle controller by PSO algorithm

For an unmanned vehicle, in difficult conditions, when spatial constraints seriously narrow the space of admissible states, the strategy of choosing a state space is more effective than sampling in the control space. Although this was obvious, the practical question is how to achieve it while meeting the stringent constraints of the vehicle’s dynamic feasibility.This article presents an unmanned vehicle control system based on the predictive integrated path model (MPPI) controller, deep convolutional neural network (CNN) for real-time scene understanding and particle swarm optimization (PSO) to find the vector of optimal cost function parameters. The method is based on the optimization of the cost function, which determines where the vehicle should move on the surface of the path.

Detection of real-time augmented reality scene light sources and construction of photorealis tic rendering framework

In this paper, the main network of multi-channel light sources is improved, so that multi-channel pictures can be fused for joint training. Secondly, for high-resolution detection pictures, the huge memory consumption leads to a reduction in batches and then affects the model distribution. Group regularization is adopted. We can still train the model normally in small batches; then, combined with the method of the regional candidate network, the final detection accuracy and the accuracy of the candidate frame regression are improved. Finally, through in-depth analysis, based on image lighting technology and physical-based rendering theory, the requirements for lighting effects and performance limitations, combined with a variety of image enhancement technologies, such as gamma correction, HDR, and these technologies used in Java. Real-time lighting algorithms that currently run efficiently on mainstream PCs. The algorithm can be well integrated into the existing rasterization rendering pipeline, while into account better lighting effects and higher operating efficiency. Finally, the lighting effects achieved by the algorithm are tested and compared through experiments. This algorithm not only achieves a very good light and shadow effect when rendering virtual objects with a real scene as the background but also can meet the realistic rendering of picture frames in more complex scenes. Rate requirements. The experimental results show that the virtual light source automatically generated by this algorithm can approximate the lighting of the real scene, and the virtual object and the real object can produce approximately consistent lighting effects in an augmented reality environment with one or more real light sources.

EKENet: Efficient knowledge enhanced network for real-time scene parsing

Scene parsing is essential for many high-level AI applications, such as intelligent vehicles and traffic surveillance. In this work, we propose a highly efficient and powerful deep convolutional neural network, namely Efficient Knowledge Enhanced Network (EKENet), for parsing scenes in real-time. Unlike most existing approaches that compromise efficiency for the sake of high accuracy, EKENet achieves an ideal trade-off between the two. Our EKENet is built upon a novel building block, namely Efficient Dual Abstraction (EDA) block, which employs an efficiently parallel convolution structure for extracting spatial features and modeling cross-channel correlations in a dual fashion. Additionally, a novel light-weight Encoding-Enhancing (EE) module is designed to enhance our EKENet, which can efficiently encode high-level knowledge extracted from top layers to guide the learning of low-level features from bottom layers.Extensive experiments on challenging benchmarks, Cityscapes and CamVid datasets, demonstrate that EKENet achieves the new state-of-the-art performance in terms of speed and accuracy tradeoff.

DefSLAM: Tracking and Mapping of Deforming Scenes From Monocular Sequences

Monocular simultaneous localization and mapping (SLAM) algorithms perform robustly when observing rigid scenes; however, they fail when the observed scene deforms, for example, in medical endoscopy applications. In this article, we present DefSLAM, the first monocular SLAM capable of operating in deforming scenes in real time. Our approach intertwines Shape-from-Template (SfT) and Non-Rigid Structure-from-Motion (NRSfM) techniques to deal with the exploratory sequences typical of SLAM. A deformation tracking thread recovers the pose of the camera and the deformation of the observed map, at frame rate, by means of SfT processing a template that models the scene shape-at-rest. A deformation mapping thread runs in parallel with the tracking to update the template, at keyframe rate, by means of an isometric NRSfM processing a batch of full perspective keyframes. In our experiments, DefSLAM processes close-up sequences of deforming scenes, both in a laboratory-controlled experiment and in medical endoscopy sequences, producing accurate 3-D models of the scene with respect to the moving camera.

Efficient Multi-Object Detection and Smart Navigation Using Artificial Intelligence for Visually Impaired People

Visually impaired people face numerous difficulties in their daily life, and technological interventions may assist them to meet these challenges. This paper proposes an artificial intelligence-based fully automatic assistive technology to recognize different objects, and auditory inputs are provided to the user in real time, which gives better understanding to the visually impaired person about their surroundings. A deep-learning model is trained with multiple images of objects that are highly relevant to the visually impaired person. Training images are augmented and manually annotated to bring more robustness to the trained model. In addition to computer vision-based techniques for object recognition, a distance-measuring sensor is integrated to make the device more comprehensive by recognizing obstacles while navigating from one place to another. The auditory information that is conveyed to the user after scene segmentation and obstacle identification is optimized to obtain more information in less time for faster processing of video frames. The average accuracy of this proposed method is 95.19% and 99.69% for object detection and recognition, respectively. The time complexity is low, allowing a user to perceive the surrounding scene in real time.

MOXA: A Deep Learning Based Unmanned Approach For Real-Time Monitoring of People Wearing Medical Masks.

With 6.93M confirmed cases of COVID-19 worldwide, making individuals aware of their sanitary health and ongoing pandemic remains the only way to prevent the spread of this virus. Wearing masks is an important step in this prevention. Hence, there is a need for monitoring if people are wearing masks or not. Closed circuit television (CCTV) cameras endowed with computer vision function by embedded systems, have become popular in a wide range of applications, and can be used in this case for real time monitoring of people wearing masks or not. In this paper, we propose to model this task of monitoring as a special case of object detection. However, real-time scene parsing through object detection running on edge devices is very challenging, due to limited memory and computing power of embedded devices. To deal with these challenges, we used a few popular object detection algorithms such as YOLOv3, YOLOv3Tiny, SSD and Faster R-CNN and evaluated them on Moxa3K benchmark dataset. The results obtained from these evaluations help us to determine methods that are more efficient, faster, and thus are more suitable for real-time object detection specialized for this task.

Pedestrian detection based on a hybrid Gaussian model and support vector machine

ABSTRACT In order to improve the detection rate of pedestrian, a novel technique that is based on hybrid Gaussian background modeling combined with HOG and SVM is proposed. The random video frame test sample is used to verify the performance of the model. The results show that under the condition of ensuring the detection rate and detection rate, the false detection rate of the hybrid Gauss combined with the HOG + SVM model is only 5.2% in comparison with that of HOG + AdaBoost at 7.1%, which confirms that the model can accurately detect pedestrians in complex scenes in real time.

A Comparative Study in Real-Time Scene Sonification for Visually Impaired People.

In recent years, with the development of depth cameras and scene detection algorithms, a wide variety of electronic travel aids for visually impaired people have been proposed. However, it is still challenging to convey scene information to visually impaired people efficiently. In this paper, we propose three different auditory-based interaction methods, i.e., depth image sonification, obstacle sonification as well as path sonification, which convey raw depth images, obstacle information and path information respectively to visually impaired people. Three sonification methods are compared comprehensively through a field experiment attended by twelve visually impaired participants. The results show that the sonification of high-level scene information, such as the direction of pathway, is easier to learn and adapt, and is more suitable for point-to-point navigation. In contrast, through the sonification of low-level scene information, such as raw depth images, visually impaired people can understand the surrounding environment more comprehensively. Furthermore, there is no interaction method that is best suited for all participants in the experiment, and visually impaired individuals need a period of time to find the most suitable interaction method. Our findings highlight the features and the differences of three scene detection algorithms and the corresponding sonification methods. The results provide insights into the design of electronic travel aids, and the conclusions can also be applied in other fields, such as the sound feedback of virtual reality applications.

May a Pair of ‘Eyes’ Be Optimal for Vehicles Too?

Following a very brief look at the human vision system, an extended summary of our own elemental steps towards future vision systems for ground vehicles is given, leading to the proposal made in the main part. The question is raised of why the predominant solution in biological vision systems, namely pairs of eyes (very often multi-focal and gaze-controllable), has not been found in technical systems up to now, though it may be a useful or even optimal solution for vehicles too. Two potential candidates with perception capabilities closer to the human sense of vision are discussed in some detail: one with all cameras mounted in a fixed way onto the body of the vehicle, and one with a multi-focal gaze-controllable set of cameras. Such compact systems are considered advantageous for many types of vehicles if a human level of performance in dynamic real-time vision and detailed scene understanding is the goal. Increasingly general realizations of these types of vision systems may take all of the 21st century to be developed. The big challenge for such systems with the capability of learning while seeing will be more on the software side than on the hardware side required for sensing and computing.

Real-Time Scene Text Detection with Differentiable Binarization

Recently, segmentation-based methods are quite popular in scene text detection, as the segmentation results can more accurately describe scene text of various shapes such as curve text. However, the post-processing of binarization is essential for segmentation-based detection, which converts probability maps produced by a segmentation method into bounding boxes/regions of text. In this paper, we propose a module named Differentiable Binarization (DB), which can perform the binarization process in a segmentation network. Optimized along with a DB module, a segmentation network can adaptively set the thresholds for binarization, which not only simplifies the post-processing but also enhances the performance of text detection. Based on a simple segmentation network, we validate the performance improvements of DB on five benchmark datasets, which consistently achieves state-of-the-art results, in terms of both detection accuracy and speed. In particular, with a light-weight backbone, the performance improvements by DB are significant so that we can look for an ideal tradeoff between detection accuracy and efficiency. Specifically, with a backbone of ResNet-18, our detector achieves an F-measure of 82.8, running at 62 FPS, on the MSRA-TD500 dataset. Code is available at: https://github.com/MhLiao/DB.

Research on Fault Diagnosis Training System Based on Virtual Space Launch Site

Aiming at the problem that the ground equipment system of space launch range is complex and it is difficult to implement real fault diagnosis training, this paper analyses the requirements of fault diagnosis training, puts forward a design scheme of fault diagnosis system based on virtual range, and expounds the basic workflow of virtual fault diagnosis training system. The fault tree and expert system are combined to complete the fault diagnosis training process design. The real-time scheduling method of virtual scene based on multi-threaded entrance is used to load and render the virtual scene in real time according to the movement trend of the virtual avatar, which ensures the fluency of interoperability in terms of function and performance, so as to achieve the purpose of improving the training effect.

Boosting Real-Time Driving Scene Parsing With Shared Semantics

Real-time scene parsing is a fundamental feature for autonomous driving vehicles with multiple cameras. In this letter we demonstrate that sharing semantics between cameras with different perspectives and overlapped views can boost the parsing performance when compared with traditional methods, which individually process the frames from each camera. Our framework is based on a deep neural network for semantic segmentation but with two kinds of additional modules for sharing and fusing semantics. On the one hand, a semantics sharing module is designed to establish the pixel-wise mapping between the input images. Features as well as semantics are shared by the map to reduce duplicated workload which leads to more efficient computation. On the other hand, feature fusion modules are designed to combine different modal of semantic features, which leverage the information from both inputs for better accuracy. To evaluate the effectiveness of the proposed framework, we have applied our network to a dual-camera vision system for driving scene parsing. Experimental results show that our network outperforms the baseline method on the parsing accuracy with comparable computations.

Joint Anchor-Feature Refinement for Real-Time Accurate Object Detection in Images and Videos

Object detection has been vigorously investigated for years but fast accurate detection for real-world scenes remains a very challenging problem. Overcoming drawbacks of single-stage detectors, we take aim at precisely detecting objects for static and temporal scenes in real time. Firstly, as a dual refinement mechanism, a novel anchor-offset detection is designed, which includes an anchor refinement, a feature location refinement, and a deformable detection head. This new detection mode is able to simultaneously perform two-step regression and capture accurate object features. Based on the anchor-offset detection, a dual refinement network (DRNet) is developed for high-performance static detection, where a multi-deformable head is further designed to leverage contextual information for describing objects. As for temporal detection in videos, temporal refinement networks (TRNet) and temporal dual refinement networks (TDRNet) are developed by propagating the refinement information across time. We also propose a soft refinement strategy to temporally match object motion with the previous refinement. Our proposed methods are evaluated on PASCAL VOC, COCO, and ImageNet VID datasets. Extensive comparisons on static and temporal detection verify the superiority of DRNet, TRNet, and TDRNet. Consequently, our developed approaches run in a fairly fast speed, and in the meantime achieve a significantly enhanced detection accuracy, i.e., 84.4% mAP on VOC 2007, 83.6% mAP on VOC 2012, 69.4% mAP on VID 2017, and 42.4% AP on COCO. Ultimately, producing encouraging results, our methods are applied to online underwater object detection and grasping with an autonomous system. Codes are publicly available at https://github.com/SeanChenxy/TDRN.

Snapshot hyperspectral light field imaging using image mapping spectrometry.

In this Letter, we present a snapshot hyperspectral light field imaging system using a single camera. By integrating an unfocused light field camera with a snapshot hyperspectral imager, the image mapping spectrometer, we captured a five-dimensional (5D) ($x,y,u,v,\lambda $x,y,u,v,λ) ($x,y,$x,y, spatial coordinates; $u,v,$u,v, emittance angles; $\lambda ,$λ, wavelength) datacube in a single camera exposure. The corresponding volumetric image ($x,y,z$x,y,z) at each wavelength is then computed through a scale-depth space transform. We demonstrated the snapshot advantage of our system by imaging the spectral-volumetric scenes in real time.

Application of Internet of Things and Virtual Reality Technology in College Physical Education

College physical education is an important component of the higher education system and national health plans. Promoting the scientific and modern construction of the physical education teaching system in colleges and universities is conducive to enhancing the science and effectiveness of higher education. Aiming at the problems of single teaching methods and insufficient long-distance teaching ability in the current physical education teaching process of colleges and universities, based on virtual reality technology, this paper designs and proposes a college physical education virtual reality system consisting of the Internet of Things, cloud platform and mobile client. This system collects relevant data from the Internet of Things and interacts with the virtual reality scene in real time, rendering the scene through the cloud and experiencing virtual reality through the mobile terminal. After the requirements analysis and system framework design of the virtual reality system for physical education in colleges and universities, the system's main functions and data warehouse, software architecture design and system testing process were carried out. Through the analysis of specific trial cases and user feedback information of a college, it shows that the designed virtual reality system of college physical education has good application and promotion effect, and provides a scientific reference for deepening reform of college physical education Suggest.

An Approach to Global Illumination Calculation Based on Hybrid Cone Tracing

For 3D geographic information systems (GIS) or video game systems, global illumination (GI) effect can greatly improve the understanding of the volumetric structure and the spatial relationships of objects in the scene. However, GI effect is computationally expensive. It requires not only complex visibility computations between arbitrary points but also the integration computation over a large number of directions. These two computations become extremely difficult tasks for the real-time or interactive algorithms of the dynamic scene. This paper proposes a lightweight GI calculation approach built upon a hybrid cone tracing algorithm with which to approximate the GI effect of the open world scene in real time. First, the 3D scene is divided into two types: complex meshes and height-field meshes. Second, the corresponding lightmaps of the two meshes are generated and mipmaped, respectively. GPU hardware acceleration technology is used to do the calculation efficiently. Finally, a hybrid cone tracing method is performed on the GPU to gather the indirect lighting information for each shaded point. The method first carries out the cone-scene intersection test and then carries out the light sampling and accumulation calculation. All tracing calculations are based on the hybrid lightmap representation. In addition, the experiment results show the effectiveness of our method. Compared with the 3D texture VCT method, memory consumption can be reduced by up to 80% in our experiment.

Change Detection in Multitemporal Monitoring Images Under Low Illumination

Video surveillance may involve the simultaneous monitoring of a large number of areas. Real-time automatic change detection of a monitoring area (such as involving the movement of people or vehicles) can reduce risks incurred in negligent manual observation. However, the low signal-to-noise ratio (SNR) of dark environments can significantly corrupt camera images, making it difficult for machine learning surveillance systems to detect small changes in monitored images. In addition, in the absence of changes between two multitemporal monitoring images, sensor noise can lead to false alarms. The objective of this paper is to reduce the effect of sensor noise on change detection of monitored images and the run time of change detection algorithms. For these purposes, we proposed a novel multitemporal monitoring image change detection algorithm based on morphological structure filtering and normalized fusion difference image. First, the random noise in two surveillance images was removed using a multidirectional weighted multiscale series of a morphological filter. Next, two difference images were obtained by using the compression log-ratio operator and the mean ratio operator, and a fusion difference image was obtained by equal-weight fusion of the two difference images. Then, the sigmoid function was used to compress the fusion difference map to obtain a normalized fusion difference image, and a median filter was used to obtain a final difference image. Finally, the k-means clustering algorithm was utilized to obtain the change detection results. The experimental results demonstrate that the proposed method can accurately detect changes in a night monitoring scene in real time. Subjective and objective evaluation of the experimental results demonstrate that the proposed method is superior to reference algorithms in terms of change detection accuracy, time and robustness.

Grey Weighted QoS Evaluation based on Real-Time Scene

Grey Weighted QoS Evaluation based on Real-Time Scene

The Aircraft Pose Estimation Based on a Convolutional Neural Network

The pose estimation of the aircraft in the airport plays an important role in preventing collisions and constructing the real-time scene of the airport. However, current airport target surveillance methods regard the aircraft as a point, neglecting the importance of pose estimation. Inspired by human pose estimation, this paper presents an aircraft pose estimation method based on a convolutional neural network through reconstructing the two-dimensional skeleton of an aircraft. Firstly, the key points of an aircraft and the matching relationship are defined to design a 2D skeleton of an aircraft. Secondly, a convolutional neural network is designed to predict all key points and components of the aircraft kept in the confidence maps and the Correlation Fields, respectively. Thirdly, all key points are coarsely matched based on the matching relationship and then refined through the Correlation Fields. Finally, the 2D skeleton of an aircraft is reconstructed. To overcome the lack of benchmark dataset, the airport surveillance video and Autodesk 3ds Max are utilized to build two datasets. Experiment results show that the proposed method get better performance in terms of accuracy and efficiency compared with other related methods.

Combining Direct 3D Volume Rendering and Magnetic Particle Imaging to Advance Radiation-Free Real-Time 3D Guidance of Vascular Interventions.

Magnetic particle imaging (MPI) is a novel tomographic radiation-free imaging technique that combines high spatial resolution and real-time capabilities, making it a promising tool to guide vascular interventions. Immediate availability of 3D image data is a major advantage over the presently used digital subtraction angiography (DSA), but new methods for real-time image analysis and visualization are also required to take full advantage of the MPI properties. This laboratory study illustrates respective techniques by means of three different patient-specific 3D vascular flow models. The selected models corresponded to typical anatomical intervention sites. Routine patient cases and image data were selected, relevant vascular territories segmented, 3D models generated and then 3D-printed. Printed models were used to perform case-specific MPI imaging. The resulting MPI images, direct volume rendering (DVR)-based fast 3D visualization options, and their suitability to advance vascular interventions were evaluated and compared to conventional DSA. The experiments illustrated the feasibility and potential to enhance image interpretation during interventions by using MPI real-time volumetric imaging and problem-tailored DVR-based fast (approximately 30 frames/s) 3D visualization options. These options included automated viewpoint selection and cutaway views. The image enhancement potential is especially relevant for complex geometries (e.g., in the presence of superposed vessels). The unique features of the as-yet preclinical imaging modality MPI render it promising for guidance of vascular interventions. Advanced fast DVR could help to fulfill this promise by intuitive visualization of the 3D intervention scene in real time.