Real-time Scene Research Articles

A Convolutional Recurrent Neural-Network-Based Machine Learning for Scene Text Recognition Application

Optical character recognition (OCR) is the process of acquiring text and layout information through analysis and recognition of text data image files. It is also a process to identify the geometric location and orientation of the texts and their symmetrical behavior. It usually consists of two steps: text detection and text recognition. Scene text recognition is a subfield of OCR that focuses on processing text in natural scenes, such as streets, billboards, license plates, etc. Unlike traditional document category photographs, it is a challenging task to use computer technology to locate and read text information in natural scenes. Imaging sequence recognition is a longstanding subject of research in the field of computer vision. Great progress has been made in this field; however, most models struggled to recognize text in images of complex scenes with high accuracy. This paper proposes a new pattern of text recognition based on the convolutional recurrent neural network (CRNN) as a solution to address this issue. It combines real-time scene text detection with differentiable binarization (DBNet) for text detection and segmentation, text direction classifier, and the Retinex algorithm for image enhancement. To evaluate the effectiveness of the proposed method, we performed experimental analysis of the proposed algorithm, and carried out simulation on complex scene image data based on existing literature data and also on several real datasets designed for a variety of nonstationary environments. Experimental results demonstrated that our proposed model performed better than the baseline methods on three benchmark datasets and achieved on-par performance with other approaches on existing datasets. This model can solve the problem that CRNN cannot identify text in complex and multi-oriented text scenes. Furthermore, it outperforms the original CRNN model with higher accuracy across a wider variety of application scenarios.

Dual-Space Graph-Based Interaction Network for RGB-Thermal Semantic Segmentation in Electric Power Scene

Real-time scene comprehension is the basis for automatic electric power inspection. However, existing RGB-based scene comprehension methods may achieve unsatisfied performance when dealing with complex scenarios, insufficient illumination or occluded appearances. To solve this problem, by cooperating visual and thermal images, the Dual-Space Graph-based Interaction Network (DSGBINet) is proposed to achieve all-day time semantic segmentation of power equipment in high-voltage power transmission line and electric transformer substation scenes. Specifically, modality-specific features are first extracted via two separate backbone networks with the same architecture. Multi-modality high-level features are first fused via long-range relationship in coordinate space. Then, multi-modality features from regular grids are further clustered and assigned to vertices in feature space. Cross-graph and inner-graph regional relations are utilized for reasoning and enhancement, which could exploit the mutual benefits and extract rich contextual information in a semantic view. Furthermore, to overcome the huge scale difference and the inherent characteristic of thermal images, the idea of multi-task learning is integrated into the decoding process. The edge detection and semantic segmentation are achieved collaboratively, which could segment the different power equipment more accurately and completely. The comparative and ablation experiments on the proposed two RGB-T semantic segmentation datasets evaluate the effectiveness and robustness of the proposed network compared with existing state-of-the-art methods. The extended experiments on the public datasets further demonstrate the superiority of the proposed method. Our dataset and code will be released at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/hhujiang/DSGBINet</uri> .

Object Sub-Categorization and Common Framework&lt;br /&gt; Method using Iterative AdaBoost for Rapid Detection of Multiple Objects

Object detection and tracking in real time has numerous applications and benefits in various fields like survey, crime detection etc. The idea of gaining useful information from real time scenes on the roads is called as Traffic Scene Perception (TSP). TSP actually consists of three subtasks namely, detecting things of interest, recognizing the discovered objects and tracking of the moving objects. Normally the results obtained could be of value in object recognition and tracking, however the detection of a particular object of interest is of higher value in any real time scenario. The prevalent systems focus on developing unique detectors for each of the above-mentioned subtasks and they work upon utilizing different features. This obviously is time consuming and involves multiple redundant operations. Hence in this paper a common framework using the enhanced AdaBoost algorithm is proposed which will examine all dense characteristics only once thereby increasing the detection speed substantially. An object sub-categorization strategy is proposed to capture the intra-class variance of objects in order to boost generalisation performance even more. We use three detection applications to demonstrate the efficiency of the proposed framework: traffic sign detection, car detection, and bike detection. On numerous benchmark data sets, the proposed framework delivers competitive performance using state-of-the-art techniques.

Real-Time Scene Text Detection With Differentiable Binarization and Adaptive Scale Fusion.

Recently, segmentation-based scene text detection methods have drawn extensive attention in the scene text detection field, because of their superiority in detecting the text instances of arbitrary shapes and extreme aspect ratios, profiting from the pixel-level descriptions. However, the vast majority of the existing segmentation-based approaches are limited to their complex post-processing algorithms and the scale robustness of their segmentation models, where the post-processing algorithms are not only isolated to the model optimization but also time-consuming and the scale robustness is usually strengthened by fusing multi-scale feature maps directly. In this paper, we propose a Differentiable Binarization (DB) module that integrates the binarization process, one of the most important steps in the post-processing procedure, into a segmentation network. Optimized along with the proposed DB module, the segmentation network can produce more accurate results, which enhances the accuracy of text detection with a simple pipeline. Furthermore, an efficient Adaptive Scale Fusion (ASF) module is proposed to improve the scale robustness by fusing features of different scales adaptively. By incorporating the proposed DB and ASF with the segmentation network, our proposed scene text detector consistently achieves state-of-the-art results, in terms of both detection accuracy and speed, on five standard benchmarks.

Ambiguity-Aware Multi-Object Pose Optimization for Visually-Assisted Robot Manipulation

6D object pose estimation aims to infer the relative pose between the object and the camera using a single image or multiple images. Most works have focused on predicting the object pose without associated uncertainty under occlusion and structural ambiguity (symmetricity). However, these works demand prior information about shape attributes, and this condition is hardly satisfied in reality; even asymmetric objects may be symmetric under the viewpoint change. In addition, acquiring and fusing diverse sensor data is challenging when extending them to robotics applications. Tackling these limitations, we present an ambiguity-aware 6D object pose estimation network, PrimA6D++, as a generic uncertainty prediction method. The major challenges in pose estimation, such as occlusion and symmetry, can be handled in a generic manner based on the measured ambiguity of the prediction. Specifically, we devise a network to reconstruct the three rotation axis primitive images of a target object and predict the underlying uncertainty along each primitive axis. Leveraging the estimated uncertainty, we then optimize multi-object poses using visual measurements and camera poses by treating it as an object SLAM problem. The proposed method shows a significant performance improvement in T-LESS and YCB-Video datasets. We further demonstrate real-time scene recognition capability for visually-assisted robot manipulation.

A Real-Time Scene Parsing Network for Autonomous Maritime Transportation

On the sea, scene parsing, one of the most basic environmental perception methods for autonomous tasks, helps autonomous devices safely navigate and avoid obstacles. However, most of the existing methods lack real-time performance and have difficulty accurately detecting small obstacles and changeable textures. We, therefore, propose a real-time scene-parsing network for autonomous maritime transportation. Specifically, we first design a lightweight model framework and then explore three efficient loss functions, such that a balance between accuracy and real-time performance can be achieved. The first loss function is an obstacle-weighted loss, which improves the extraction of small obstacles by analyzing the distribution law of obstacles at sea. The second one is a detail loss, which optimizes the detail segmentation at complex contours by emphasizing edge features. The last one is an affinity loss, utilizing the context dependency between features to accurately detect reflections, ripples, and other changeable textures. In addition, a new maritime sense-parsing dataset called Greater Bay Area (GBA) dataset is proposed and made publicly available. We tested the proposed model on the GBA dataset and MaSTr1325 dataset, and the experimental results show that the proposed method achieves superior performance in both segmentation and speed, with mean intersection over union (mIOU) of 94.59% and an FPS of 39.44.

A Geometry-Aware Consistent Constraint for Height Estimation From a Single SAR Imagery in Mountain Areas

Height estimation from a single synthetic aperture radar(SAR) imagery has shown a great potential in real-time scene understanding and environment detection. It is a mathematical ill-posed problem for that a single 2D image may be projected from multiple 3D scenes. Then, the problem is that the accuracy of height estimation from a single SAR image is not high enough without prior knowledge, especially in mountain areas. Thus, we proposes a geometry-aware consistent constraint(GACC) to improve the performance of single SAR height estimation. The simulated SAR images generated from height maps are used to establish high precision transformation between map and radar coordinates in mountain areas. The main idea of GACC is that the simulated SAR image generated from estimated height map should be consistent with the ground truth simulated SAR image. A sparse height map is included as the supplementary inputs to further improve the accuracy of estimated height maps. Comparison experiments are performed in Geermu, Huangshan and Guiyang datasets and the results show that the RMSE of the estimated height map by Unet with GACC and 0.0434% sparse height gets reduced by about 98%, 78% and 94% compared with that by Unet in the three datasets.

Indoor 3D Point Cloud Segmentation Based on Multi-Constraint Graph Clustering

Indoor scene point cloud segmentation plays an essential role in 3D reconstruction and scene classification. This paper proposes a multi-constraint graph clustering method (MCGC) for indoor scene segmentation. The MCGC method considers multi-constraints, including extracted structural planes, local surface convexity, and color information of objects for indoor segmentation. Firstly, the raw point cloud is partitioned into surface patches, and we propose a robust plane extraction method to extract the main structural planes of the indoor scene. Then, the match between the surface patches and the structural planes is achieved by global energy optimization. Next, we closely integrate multiple constraints mentioned above to design a graph clustering algorithm to partition cluttered indoor scenes into object parts. Finally, we present a post-refinement step to filter outliers. We conducted experiments on a benchmark RGB-D dataset and a real indoor laser-scanned dataset to perform numerous qualitative and quantitative evaluation experiments, the results of which have verified the effectiveness of the MCGC method. Compared with state-of-the-art methods, MCGC can deal with the segmentation of indoor scenes more efficiently and restore more details of indoor structures. The segment precision and the segment recall of experimental results reach 70% on average. In addition, a great advantage of the MCGC method is that the speed of processing point clouds is very fast; it takes about 1.38 s to segment scene data of 1 million points. It significantly reduces the computation overhead of scene point cloud data and achieves real-time scene segmentation.

PlaneFusion: Real-Time Indoor Scene Reconstruction With Planar Prior

Real-time dense SLAM techniques aim to reconstruct the dense three-dimensional geometry of a scene in real time with an RGB or RGB-D sensor. An indoor scene is an important type of working environment for these techniques. The planar prior can be used in this scenario to improve the reconstruction quality, especially for large low-texture regions that commonly occur in an indoor scene. This article fully explores the planar prior in a dense SLAM pipeline. First, we propose a novel plane detection and segmentation method that runs at 200 Hz on a modern graphics processing unit. Our algorithm for constructing global plane constraints is very efficient; hence, we use it in the process of each input frame for the camera pose estimation while maintaining the real-time performance. Second, we propose herein a plane-based map representation that greatly reduces the memory footprint of plane regions while keeping the geometric details on planes. The experiments reveal that our system yields superior reconstruction results with planar information running at more than 30 fps. Aside from speed and storage improvements, our technique also handles the low-texture problem in plane regions.

Motion characteristics simulations of supercavitating vehicle based on a three-dimensional cavity topology algorithm

The maneuvering motion of a supercavitating vehicle pitching and yawing inevitably occurs when navigating in three-dimensional space. In order to give some insights into design of underwater vehicles, the maneuvering motion in the longitudinal and horizontal planes is investigated in this paper. Considering the definition of multi-body coordinate frames, a three-dimensional cavity topology model is proposed to predict the development of the shape of cavity and interact with the supercavitating vehicle in the simulations. Based on the cavity model, an improved planning model is established to calculate the necessary critical parameters of the planning force. According to the mathematical model above, a numerical algorithm is developed to simulate the maneuvering motion of the supercavitating vehicle. The higher algorithm efficiency is validated by the relative error of 10% order of magnitude and the more stable performance. The real-time scene figures and time-varying curves of the dynamic characteristics are obtained through the developed algorithm to study the influence of fins deflection on the maneuvering motion. The developed algorithm is validated to some extent to be a better alternative than the empirical one and provides a new method for analyses of simulation results of supercavitating vehicles.

An Object Detection and Localization Method Based on Improved YOLOv5 for the Teleoperated Robot

In the traditional teleoperation system, the operator locates the object using the real-time scene information sent back from the robot terminal; however, the localization accuracy is poor and the execution efficiency is low. To address the issues, we propose an object detection and localization method for the teleoperated robot. First, we improved the classic YOLOv5 network model to produce superior object detection performance and named the improved model YOLOv5_Tel. On the basis of the classic YOLOv5 network model, the feature pyramid network was changed to a bidirectional feature pyramid network (BiFPN) network module to achieve the weighted feature fusion mechanism. The coordinate attention (CA) module was added to make the model pay more attention to the features of interest. Furthermore, we pruned the model from the depth and width to make it more lightweight and changed the bounding box regression loss function GIOU to SIOU to speed up model convergence. Then, the YOLOv5_Tel model and ZED2 depth camera were used to achieve object localization based on the binocular stereo vision ranging principle. Finally, we established an object detection platform for the teleoperated robot and created a small dataset to validate the proposed method. The experiment shows that compared with the classic YOLOv5 series network model, the YOLOv5_Tel is higher in accuracy, lighter in weight, and faster in detection speed. The mean average precision (mAP) value of the YOLOv5_Tel increased by 0.8%, 0.9%, and 1.0%, respectively. The model size decreased by 11.1%, 70.0%, and 86.4%, respectively. The inference time decreased by 9.1%, 42.9%, and 58.3%, respectively. The proposed object localization method has a high localization accuracy with an average relative error of only 1.12%.

A SMART DATA APPROACH TO ANALYZE VEHICLE FLOWS

Abstract. In the logic of Smart Cities it is of fundamental importance to analyze the traffic situation through dedicated sensors and networks. According to this approach and through the potential of smart data is based this study. Improve prediction of traffic patterns by analyzing and counting vehicles in a virtualized scene in real time. In the past, the technique of hardware inductive coils was used that were dropped in the asphalt to exploit the principle of magnetic induction in order to verify the transit of vehicles. This technique is not able to classify vehicles or estimate their speed, unless using multiple inductive coils. The proposed system provides for the virtualization of an area of interest which requires a selection and mapping of the areas where the control areas are to be included. The “image detection” techniques allow us to classify the vehicles in transit. With the techniques of “machine learning” can to able to verify the flow, count the vehicles present in the scene and classify them by vehicle type in real time. The vehicle counting and classification data available in the cloud platform allow to model and update the main nodes of the network in order to improve the prediction and estimates of the best routes of the road network according to the degree of saturation of the flows and the length of the line of the graph. The model can also indicate additional information of an environmental nature in an ITS system present in the cloud.

Real-Time Scene Monitoring for Deaf-Blind People.

It is estimated that at least 15 million people worldwide live with severe deaf-blindness, with many more experiencing varying degrees of deaf-blindness. The existing options of assistance are mostly limited to walking canes, guide dogs and human care. We propose a wearable device which harnesses a multi-antenna mmWave radar transceiver and a haptic feedback array for real time detection of a person moving within a scene. We present our findings from a series of workshops with participants classed with multi-sensory impairments (MSI), to demonstrate the relative success of this approach and its potential for integration into existing assistance for the MSI of the future.

Multi-Agent Multi-View Collaborative Perception Based on Semi-Supervised Online Evolutive Learning.

In the edge intelligence environment, multiple sensing devices perceive and recognize the current scene in real time to provide specific user services. However, the generalizability of the fixed recognition model will gradually weaken due to the time-varying perception scene. To ensure the stability of the perception and recognition service, each edge model/agent needs to continuously learn from the new perception data unassisted to adapt to the perception environment changes and jointly build the online evolutive learning (OEL) system. The generalization degradation problem can be addressed by deploying the semi-supervised learning (SSL) method on multi-view agents and continuously tuning each discriminative model by collaborative perception. This paper proposes a multi-view agent’s collaborative perception (MACP) semi-supervised online evolutive learning method. First, each view model will be initialized based on self-supervised learning methods, and each initialized model can learn differentiated feature-extraction patterns with certain discriminative independence. Then, through the discriminative information fusion of multi-view model predictions on the unlabeled perceptual data, reliable pseudo-labels are obtained for the consistency regularization process of SSL. Moreover, we introduce additional critical parameter constraints to continuously improve the discriminative independence of each view model during training. We compare our method with multiple representative multi-model and single-model SSL methods on various benchmarks. Experimental results show the superiority of the MACP in terms of convergence efficiency and performance. Meanwhile, we construct an ideal multi-view experiment to demonstrate the application potential of MACP in practical perception scenarios.

An Illumination-Invariant Shadow-Based Scene Matching Navigation Approach in Low-Altitude Flight

Differences in acquisition time, light conditions, and viewing angle create significant differences among the airborne remote sensing images from Unmanned Aerial Vehicles (UAVs). Real-time scene matching navigation applications based on fixed reference maps are error-prone and have poor robustness. This paper presents a novel shadow-based matching method for the localization of low-altitude flight UAVs. A reference shadow map is generated from an accurate (0.5 m spatial resolution) Digital Surface Model (DSM) with the known date and time information; a robust shadow detection algorithm is employed to detect shadows in aerial images; the shadows can then be used as a stable feature for scene matching navigation. Combining the conventional intensity-based matching method, a fusion scene navigation scheme that is more robust to illumination variations is proposed. Experiments were performed with Google satellite maps, DSM data, and real aerial images of the Zurich region. The radial localization error of the Shadow-based Matching (SbM) is less than 7.3 m at flight height below 1200 m. The fusion navigation approach also achieves an optimal combination of shadow-based matching and intensity-based matching. This study shows the solution to the inconsistencies caused by changes in light, viewing angle, and acquisition time for accurate and effective scene matching navigation.

Spatial maps with working area limit line from images of crane's top-view camera

Due to the complex working environment, limited vision, and complicated crane operations, the crane operator faces two major problems when constructing with a crane: safety and working efficiency. Every year, numerous accidents occur, the majority of which are the result of the crane operator's inability to acknowledge hidden hazards. Some of these accidents can be effectively avoided through a comprehensive and intuitive display of important visual information to the crane operator. In this research, one of the most important safety-related visual information, namely the working area limit line, is provided on the acquisition of a crane's 3D spatial map based on a top-view camera. The purpose is achieved firstly by a real-time scene reconstruction of a visual simultaneously location and mapping approach semi-direct visual odometry and regularized monocular depth estimation. Then the working area limit line is shown on the top-view image according to the reconstructed 3D spatial map. The working area limit line can be successfully and precisely shown in the top-view image in both simulation and field experiments. The augmented top-view image with a working area limit line can help the crane operator notice the hidden hazards of outreaching the crane's working limit and improve the operation efficiency. The hazard of crane overturn can be effectively avoided with the displayed result. Based on the proposed approach, much more safety and operation related visual information can be displayed according to the 3D spatial map and classified operation stages in our future work.

Real-Time Construction of Thermal Model Based on Multimodal Scene Data

In commercial buildings, the total consumption of central air conditioning accounts for about 40%–50%. However, at present, the initial design value of building Heating Ventilation and Air Conditioning (HVAC) is usually far greater than the actual refrigeration value of refrigeration demand, which will lead to great energy consumption waste. Moreover, the operation of HVAC affects the thermal comfort of users, so it is necessary to establish a thermal model for the scene to control. The thermal model describes the temperature of the scene in different environments. So it is very important to design a thermal model to calculate the scene in real time. Because the flow of people, the opening of windows, the ventilation of the scene and other parameters influence the change of thermal state in the scene environment, these parameters are complicated to model. Human disturbance will lead to the instability of the state of the scene environment. The inconsistency of its thermal model will lead to energy allocation tracking strategies in different regions. To solve this problem, We propose a thermal model for building thermal comfort using a multimodal analysis framework. This paper analyzes multiple temperature and humidity sensors and area image by multimodal combination and processes the image and sensor data by combining CNN and LSTM. Our results show that when the thermal model analyzed by this method is deployed in a building in the south of China, the MSE accuracy of the local effect of temperature field prediction reaches 99%, and its AMAX reaches 94%, so the running stability of the model in the scene is high. In addition, the research shows that the thermal model analysis framework can make the Internet of Things (IoT) in buildings more intelligent, and it can be combined with this thermal model to improve human comfort, make it easier to deploy in each hot zone, and have a better overall energy-saving effect.

Real-Time Monitoring of College Sports Dance Competition Scenes Using Deep Learning Algorithms

In order to improve the real-time detection effect, therefore, a research on real-time scene detection of sports dance competition based on deep learning is proposed. The collected scene image is grayed by using the weighted average method, and the best image interpolation is calculated by using the deep learning method, so as to realize the smooth processing of sawtooth and mosaic information generated by panoramic mapping. After selecting the cube model, the processed scene information is projected to the visual plane to construct the panorama of the competition scene. Finally, combined with the three-frame difference, the changes between adjacent image frames are calculated to obtain the moving target. The test results show that the motion detection accuracy of professional dancers can reach more than 75.0% and that of amateur dancer can reach more than 64.2%.

Edge artificial intelligence camera network: an efficient object detection and tracking framework

Computer-intensive tasks can be distributed to edge devices in edge computing, which has many benefits, including distributed processing, low latency, better security, privacy, etc. In traditional video surveillance, video is cached in the data center for post analysis and processing, but it is impossible to analyze the specific behaviors of specific objects intelligently in real-time scene. Taking this into consideration, we propose an efficient object detection and tracking framework via edge computing, which distributes the object detection and tracking tasks to edge cameras within the network instead of sending all the data to a centralized server. Its advantages can be summarized as follows: (1) instead of analyzing and processing video in the cloud, we analyze them locally, which greatly reduces the bandwidth and delay caused by the transmission of large amounts of data and ensures the security of the data. (2) we propose a neural network model compression framework for object detection with less storage and computing cost, which reduces the number of parameters and floating point operations by 39.3% and 17.28%. We use the detection results for tracking, combined with a simple tracking method, we can achieve a real-time object detection and tracking framework and more quickly determine the type and trajectory of moving objects. (3) We propose a communication module, which enables data interaction among cameras, and makes it easier for us to find the moving range of the object in the cross-camera area. In the experiment, network cameras and edge devices were used to build a framework, and an offline inference method was used to complete the detection and tracking of objects and judge the motion trajectory to verify the feasibility of the framework.

A Novel Method for Tunnel Digital Twin Construction and Virtual-Real Fusion Application

Tunnels play important roles in integrated transport infrastructure. A digital twin reproduces a real tunnel scene in virtual space and provides new means for tunnel digital maintenance. Aiming at the existing problems of video fragmentation, separation of video and business data, and lack of two- and three-dimensional linkage response methods in tunnel digital operation, in this paper, we propose a novel method for tunnel digital twin construction and virtual-real integration operation. Firstly, the digital management requirements of tunnel operations are systematically analyzed to clarify the purpose of digital twin construction. Secondly, BIM technology is used to construct a static model of the tunnel scene that conforms to the real tunnel main structure. Thirdly, a three-dimensional registration and projection calculation method is proposed to integrate tunnel surveillance video into a three-dimensional virtual scene in real time. Fourthly, multi-source sensing data are gathered and fused to form a digital twin scene that is basically the same as the real tunnel traffic operations scene. Finally, a management model suitable for digital twins is discussed to improve the efficiency of tunnel operations and management, and a tunnel in China is selected to verify this method. The results show that the proposed method is helpful to realize the application of two- and three-dimensional linkages of tunnel traffic smooth, accident rescue, facility management, and emergency response, and to improve the efficiency of tunnel digital management.