Traffic Video Analysis Research Articles

Attention Mechanism and Detection Box Information Based Real-time Multi-Object Vehicle Detection

Ensuring both the accuracy of vehicle target detection and meeting real-time requirements is crucial in traffic videos. The YOLOv5s target detection framework, known for its accuracy and efficiency, has attracted attention in academic circles. However, there are still some features that can be optimized. First of all, the detection subnet of the YOLOv5s framework cannot smoothly convert complex feature maps into relatively sparse target prediction boxes. To solve this, we integrate a self-attention-based gating mechanism into the detection subnet, forming the YOLOv5s-SAG network. Secondly, the loss function of CIoU used by YOLOv5s pays insufficient attention to the overlapping area of the detection frame, which can be used as metric for measuring target detection effectiveness. We add the loss term of area ratio to CIoU to further improve the modeling ability. Finally, the current multi-class Non-Maximum Suppression algorithm can cause high overlap of multi-class detection frames. To improve it, we propose a multi-class CS-NMS algorithm based on category suppression. Experimental results show an approximately 8% improvement in the mAP50 index on the UA-DETRAC dataset compared with YOLOv5s. The proposed algorithm also achieves better detection results compared to mainstream target detection algorithms and meets the real-time requirements of traffic video analysis.

Deep learning serves traffic safety analysis: A forward‐looking review

This paper explores deep learning (DL) methods that are used or have the potential to be used for traffic video analysis, emphasising driving safety for both autonomous vehicles and human-operated vehicles. A typical processing pipeline is presented, which can be used to understand and interpret traffic videos by extracting operational safety metrics and providing general hints and guidelines to improve traffic safety. This processing framework includes several steps, including video enhancement, video stabilisation, semantic and incident segmentation, object detection and classification, trajectory extraction, speed estimation, event analysis, modelling, and anomaly detection. The main goal is to guide traffic analysts to develop their own custom-built processing frameworks by selecting the best choices for each step and offering new designs for the lacking modules by providing a comparative analysis of the most successful conventional and DL-based algorithms proposed for each step. Existing open-source tools and public datasets that can help train DL models are also reviewed. To be more specific, exemplary traffic problems are reviewed and required steps are mentioned for each problem. Besides, connections to the closely related research areas of drivers' cognition evaluation, crowd-sourcing-based monitoring systems, edge computing in roadside infrastructures, automated driving systems-equipped vehicles are investigated, and the missing gaps are highlighted. Finally, commercial implementations of traffic monitoring systems, their future outlook, and open problems and remaining challenges for widespread use of such systems are reviewed.

Application Research of Key Frames Extraction Technology Combined with Optimized Faster R-CNN Algorithm in Traffic Video Analysis

The intelligent transportation system under the big data environment is the development direction of the future transportation system. It effectively integrates advanced information technology, data communication transmission technology, electronic sensing technology, control technology, and computer technology and applies them to the entire ground transportation management system to establish a real-time, accurate, and efficient comprehensive transportation management system that works on a large scale and in all directions. Intelligent video analysis is an important part of smart transportation. In order to improve the accuracy and time efficiency of video retrieval schemes and recognition schemes, this article firstly proposes a segmentation and key frame extraction method for video behavior recognition, using a multi-time scale dual-stream network to extract video features, improving the efficiency and efficiency of video behavior detection. On this basis, an improved algorithm for vehicle detection based on Faster R-CNN is proposed, and the Faster R-CNN network feature extraction layer is improved by using the principle of residual network, and a hole convolution is added to the network to filter out the redundant features of high-resolution video images to improve the problem of vehicle missed detection in the original algorithm. The experimental results show that the key frame extraction technology combined with the optimized Faster R-CNN algorithm model greatly improves the accuracy of detection and reduces the leakage. The detection rate is satisfactory.

DeepDeSpy: A Deep Learning-Based Wireless Spy Camera Detection System

Spy cameras planted in various private places, such as motels, hotels, homestays ( i.e., Airbnb), and restrooms, have raised immense privacy concerns. Wi-Fi spy cameras are used extensively by various adversaries because of easy installability, followed by size reduction. To prevent invasions of privacy, most studies have detected wireless cameras based on video traffic analysis and require additional synchronous data from external sensors or stimulus hardware to confirm the user’s motion. Such supplements make the users uncomfortable, requiring extra effort and time for setting. This paper proposes an effective spy camera detection system called DeepDeSpy to detect the recording of a spy camera with no effort from the user. The core idea is using the channel state information (CSI) and the network traffic from the camera to detect whether the wireless camera records the movements of the user. The CSI signal is prone to motion, and detecting motion from an enormous amount of CSI data in real-time is challenging. This was handled by leveraging the convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM) deep learning methods. Such synergistic CNN and BiLSTM deep learning models enable instant and accurate detection by automatically extracting meaningful features from the sequential raw CSI data. The feasibility of DeepDeSpy was verified by implementing it on both a PC and a smartphone and evaluating it in real-life scenarios (e.g., various room sizes and user physical activities). The average accuracy achieved in different real-life settings was approximately 96%, reaching 98.9% with intensive physical activity in the large-size room. Moreover, the ability to achieve instant detection on a smartphone within only a one-second response time makes it workable for real-time applications.

Technical and Economic Feasibility Assessment of a Cloud-Enabled Traffic Video Analysis Framework

Computer vision techniques are expected to significantly improve the development of intelligent transportation systems (ITS), which are anticipated to be a key component of future Smart City frameworks. Powered by computer vision techniques, the conversion of existing traffic cameras into connected “smart sensors” called intelligent video analysis (IVA) systems has shown the great capability of producing insightful data to support ITS applications. However, developing such IVA systems for large-scale, real-time application deserves further study, as the current research efforts are focused more on model effectiveness instead of model efficiency. Therefore, we have introduced a real-time, large-scale, cloud-enabled traffic video analysis framework using the NVIDIA DeepStream and NVIDIA Metropolis. In this study, we have evaluated the technical and economic feasibility of our proposed framework to help traffic agency to build IVA systems more efficiently. Our study shows that the daily operating cost for our proposed framework on Google Cloud Platform is less than $0.14 per camera, and that, compared with manual inspections, proposed framework achieves an average vehicle-counting accuracy of 83.7% on sunny days.

An IoT approach for efficient overtake detection of vehicles using H264/AVC video data

In this paper, a low-cost embedded autonomous system is presented. Its main device is a Raspberry Pi 2, which allows real-time analysis of traffic videos. However, the previously developed computer vision algorithms do not work at the pixel-level, meaning that they do not analyze each pixel on every frame of the video because this would entail working with such a large amount of information that it would not be possible to obtain real-time processing. Therefore, as an alternative, our system works on compressed video. In particular, our system takes as an input the H.264/AVC motion vector data produced in the encoder of the GPU embedded in Broadcom’s BCM2836 SoC. Furthermore, apart from working with very little information, the algorithms that are employed for video analysis must be efficient on their own. Thus, we propose algorithms that are capable of obtaining results directly from the analysis of a group of statistic measures, which are calculated based on a combination of input vectors. Experimentally, our system applies on the identification of overtakes, particularly those exceeding the maximum permitted speed, generating at the same time a report that is sent to a complaints e-mail account.

Joint 3D Reconstruction and Object Tracking for Traffic Video Analysis Under IoV Environment

Benefits from artificial intelligence and the Internet of Vehicles (IoV), Management of modern transportation have great progress, especially in urban areas. However, traditional traffic video analysis and visualization are usually conducted in offsite and textural environments, i.e., text and number, which do not promote user's sensorial perception and interaction. Thus, the problem that how to use modern novel techniques to analyze traffic video for improving intelligent transportation is so emergency. In this paper, we introduce a joint 3D reconstruction and object tracking approach to traffic video analysis under the IoV environment, which is an integrative framework and consists of 3D reconstruction, object detection, and visual tracking. The 3D reconstruction system is connected to the Internet of Vehicles and integrated into the system to retrieve image data for recovering the 3D model of vehicles, and then, visualizing vehicle trajectories in real-time by augmented reality. And the system can also locate the vehicle's position in real-time. The experiments in both laboratory and practice show great feedback, which will effectively contribute to intelligent transportation.

A new cast shadow detection method for traffic surveillance video analysis using color and statistical modeling

In traffic surveillance video analysis systems, the cast shadows of vehicles often have a negative effect on video analysis results. A novel cast shadow detection framework, which consists of a new foreground detection method and a cast shadow detection method, is presented in this paper to detect and remove the cast shadows from the foreground. The new foreground detection method applies an innovative Global Foreground Modeling (GFM) method, a Gaussian mixture model or GMM, and the Bayes classifier for foreground and background classification. While the GFM method is for global foreground modeling, the GMM is for local background modeling, and the Bayes classifier applies both the foreground and the background models for foreground detection. The rationale of the GFM method stems from the observation that the foreground objects often appear in recent frames and their trajectories often lead them to different locations in these frames. As a result, the statistical models used to characterize the foreground objects should not be pixel based or locally defined. The cast shadow detection method contains four hierarchical steps. First, a set of new chromatic criteria is presented to detect the candidate shadow pixels in the HSV color space. Second, a new shadow region detection method is proposed to cluster the candidate shadow pixels into shadow regions. Third, a statistical shadow model, which uses a single Gaussian distribution to model the shadow class, is presented for classifying shadow pixels. Fourth, an aggregated shadow detection method is presented for final shadow detection. Experiments using the public video data ‘Highway-1’ and ‘Highway-3’, and the New Jersey Department of Transportation (NJDOT) real traffic video sequences show the feasibility of the proposed method. In particular, the proposed method achieves better shadow detection performance than the popular shadow detection methods, and is able to improve the traffic video analysis results.

An efficient multiple access control protocol for directional dense urban traffic surveillance system

Artificial intelligence (AI) methods for traffic video analysis have been widely identified as potential solutions for solving hard problems in intelligent transport systems (ITS). To exploit the advantages of AI, dense cameras to monitor the traffic are required to be deployed along the road and at the intersections. The captured videos of these cameras should be back-hauled to the control center, acting as the inputs of the AI methods. To bear such large data traffic load and to cover long transmission range, directional communication technology can be employed, which concentrate the energy of the wireless signal in a specified direction to provide high data rate and long transmission range (up to hundreds of kilometers). In this paper, the communication time extension problem (CTEP) is identified when directional transmission is applied to the dense urban traffic surveillance system, where the wireless signal propagation time approximates the data transmission time. A link distance division-based time division multiple access (LDD-TDMA) protocol is proposed to address the identified CTEP. Firstly, the directional wireless communication links are classified into categories according to the link distance, where nodes located in the same communication ring belong to the same category. Then a link distance aware (LDA)-based slot allocation algorithm is proposed to assign the time slots to the links. The optimal communication rings’ radius is derived in closed form formula, and the minimum average links’ distance is derived. Simulation results show that LDD-TDMA outperforms TDMA by 13.37% when the ring number is 4.

Video Traffic Analysis over LEACH-GA routing protocol in a WSN

Abstract One of the most advanced evolutions of Wireless communication is Wireless Sensor Network (WSN), which plays a vital role in many applications. The growth and attention towards the designing of many new QoS aware routing protocol for WSN has been increased for the past few decades. Energy awareness has a greater significance while developing a routing protocol in order to increase network lifetime. In this paper, a traditional LEACH-GA routing protocol has been used for analyzing the video traffic over the sensor network.

Employing Topical Relations in Semantic Analysis of Traffic Videos

Motion patterns in traffic video can be directly exploited to generate high-level descriptions of video content, which can be used for rule mining and abnormal event detection. The most recent and successful unsupervised methods for complex traffic scene analysis are based on topic models. In this paper, a topic related sparse topical coding framework is proposed for more effectively discovering motion patterns in traffic videos.

Urban Traffic Density Estimation Based on Ultrahigh-Resolution UAV Video and Deep Neural Network

This paper presents an advanced urban traffic density estimation solution using the latest deep learning techniques to intelligently process ultrahigh-resolution traffic videos taken from an unmanned aerial vehicle (UAV). We first capture nearly an hour-long ultrahigh-resolution traffic video at five busy road intersections of a modern megacity by flying a UAV during the rush hours. We then randomly sampled over 17 K 512×512 pixel image patches from the video frames and manually annotated over 64 K vehicles to form a dataset for this paper, which will also be made available to the research community for research purposes. Our innovative urban traffics analysis solution consists of an advanced deep neural network (DNN) based vehicle detection and localization, type (car, bus, and truck) recognition, tracking, and vehicle counting over time. We will present extensive experimental results to demonstrate the effectiveness of our solution. We will show that our enhanced single shot multibox detector (Enhanced-SSD) outperforms other DNN-based techniques and that deep learning techniques are more effective than traditional computer vision techniques in traffic video analysis. We will also show that ultrahigh-resolution video provides more information that enables more accurate vehicle detection and recognition than lower resolution contents. This paper not only demonstrates the advantages of using the latest technological advancements (ultrahigh-resolution video and UAV), but also provides an advanced DNN-based solution for exploiting these technological advancements for urban traffic density estimation.

A New Accurate and Fast Homography Computation Algorithm for Sports and Traffic Video Analysis

Homography has wide applications in aerial photographic surveys, camera calibration, traffic scene analysis, and sports science, such as player and team performance evaluation. Unlike the mainstream homography that utilizes points as matching features, homography estimation for sports and traffic video can achieve higher accuracy and speed by utilizing straight lines in the scenes, which convey more information than points. Owing to the more stringent requirement of accuracy and computational speed for advanced video analysis, this paper presents a novel homography computational algorithm. Three major novelties are proposed and validated, which are multiple points Hough transform for straight line extraction, correspondence initialization by angle to estimate a set of quasi-optimal solutions, and the feature correspondences optimization to achieve a minimized error using genetic algorithm. With these contributions, the experiments have shown that the proposed algorithm can improve the homography computational accuracy by up to 130% and reduce the processing time by up to 96% over the state-of-the-art algorithms for the same purposes.

Detection and classification of vehicles for traffic video analytics

We present a traffic video analysis system based on computer vision techniques. The system is designed to automatically gather important statistics for policy makers and regulators in an automated fashion. These statistics include vehicle counting, vehicle type classification, estimation of vehicle speed from video and lane usage monitoring. The core of such system is the detection and classification of vehicles in traffic videos. We implement two models for this purpose, first is a MoG + SVM system and the second is based on Faster RCNN, a recently popular deep learning architecture for detection of objects in images. We show in our experiments that Faster RCNN outperforms MoG in detection of vehicles that are static, overlapping or in night time conditions. Faster RCNN also outperforms SVM for the task of classifying vehicle types based on appearances.

交通视频分析与跟踪系统设计

交通视频分析与跟踪系统设计

Video traffic analytics for large scale surveillance

The video traffic analysis is the most important issue for large scale surveillance. In the large scale surveillance system, huge amount of live digital video data is submitted to the storage servers through the number of externally connected scalable components. The system also contains huge amount of popular and unpopular old videos in the archived storage servers. The video data is delivered to the viewers, partly or completely on demand through a compact system. In real time, huge amount of video data is imported to the viewer's node for various analysis purposes. The viewers use a number of interactive operations during the real time tracking suspect. The compact video on demand system is used in peer to peer mesh type hybrid architecture. The chunk of video objects move fast through the real time generated compact topological space. Video traffic analytics is required to transfer compressed multimedia data efficiently. In this work, we present a dynamically developed topological space, using mixed strategy by game approach to move the video traffic faster. The simulation results are well addressed in real life scenario.

Video traffic analysis using scene and vehicle models

This paper presents a complete system for analyzing a vehicle׳s behavior in the context of real-time traffic video surveillance applications. To obtain the best possible results, it is fundamental to exploit the scene characteristics and the predefined traffic rules. For that purpose, an initial training step is performed that involves estimating the geometrical structure of the road, i.e., the depth relative to the camera, the vanishing points, the road areas itself, the road decomposition (into normal and forbidden traffic lanes or areas), the traffic rules, the typical vehicle trajectories and speeds, and the lane-changing rules. This process leads to a scene model, which is used together with a simple vehicle geometrical model during the vehicle detection, tracking and trajectory estimation phases to improve the robustness against the perspective and occlusion effects. Shadow effects are also accounted for during the moving object detection phase. Finally, this spatio-temporal analysis is used to obtain information that concerns the vehicles׳ behaviors. Experiments show that the information obtained is reliable and can be computed in real time.

Learning-based spatio-temporal vehicle tracking and indexing for transportation multimedia database systems

One key technology of intelligent transportation systems is the use of advanced sensor systems for on-line surveillance to gather detailed information on traffic conditions. Traffic video analysis can provide a wide range of useful information to traffic planners. In this context, the object-level indexing of video data can enable vehicle classification, traffic flow analysis, incident detection and analysis at intersections, vehicle tracking for traffic operations, and update of design warrants. In this paper, a learning-based automatic framework is proposed to support the multimedia data indexing and querying of spatio-temporal relationships of vehicle objects in a traffic video sequence. The spatio-temporal relationships of vehicle objects are captured via the proposed unsupervised image/video segmentation method and object tracking algorithm, and modeled using a multimedia augmented transition network model and multimedia input strings. An efficient and effective background learning and subtraction technique is employed to eliminate the complex background details in the traffic video frames. It substantially enhances the efficiency of the segmentation process and the accuracy of the segmentation results to enable more accurate video indexing and annotation. The paper uses four real-life traffic video sequences from several road intersections under different weather conditions in the study experiments. The results show that the proposed framework is effective in automating data collection and access for complex traffic situations.