Real-time Video Streaming Research Articles

Edge Coordinated Query Configuration for Low-Latency and Accurate Video Analytics

To develop smart city and intelligent manufacturing, video cameras are being increasingly deployed. In order to achieve fast and accurate response to live video queries (e.g., license plate recording and object tracking), the real-time high-volume video streams should be delivered and analyzed efficiently. In this article, we introduce an end-edge-cloud coordination framework for low-latency and accurate live video analytics. Considering the locality of video queries, edge platform is designated as the system coordinator. It accepts live video queries and configures the related end cameras to generate video frames that meet quality requirements. By taking into account the latency constraint, edge computing resources are subtly distributed to process the live video frames from different sources such that the analytic accuracy of the accepted video queries can be maximized. Since the amount of required edge computing resource and video quality to accurately address different video queries are unknown in advance, we propose an online video quality and computing resource configuration algorithm to gradually learn the optimal configuration strategy. Extensive simulation results show that as compared to other benchmarks, the proposed configuration algorithm can effectively improve the analytic accuracy, while providing low-latency response.

SNS 실시간 동영상 서비스 이용의 만족도와 지속사용의도: 개인적 특성과 사회적 현존감의 영향

본 연구는 기존의 소셜미디어에서 새롭게 제공되는 SNS 실시간 동영상 서비스이용의 이용동기를 밝히고 이용동기와 이용자 개인적 특성이 이용 만족도와 지속사용의도에 미치는 영향을 살펴보았다. 또한, 개인적 특성과 이용 만족도와의 관계를 사회적 현존감이 매개하는지 검증하고자 하였다. 이를 위해 온라인 서베이를 실시하였으며(N = 400), 이용동기로는 정보추구, 사회적 상호작용, 오락/즐거움, 현실도피가 도출되었다. 이들 이용동기 중 정보추구, 오락/즐거움, 현실도피는 이용 만족도와 지속사용의도에 유의미한 영향을 미치는 것으로 나타났다. 또한 SNS 실시간 동영상 서비스 이용자 특성의 개혁성, 자기표현, 자기노출은 모두 이용 만족도와 지속사용의도에 유의미한 영향을 미치는 것으로 나타났다. 이와 함께, 사회적 현존감은 개혁성, 자기표현, 자기노출 성향이 이용 만족도에 미치는 영향을 매개하는 것으로 확인되었다. 본 연구는 앞으로의 뉴미디어 동향에 중요한 역할을 할 것으로 여겨지는 SNS 실시간 동영상 서비스와 관련하여, 이용동기와 개인적 특성, 이용 만족도와 지속사용의사와의 관계를 살펴보았다는데 의의가 있으며, 기존의 소셜미디어 이용동기 연구에서 중요한 요인으로 다뤄졌던 사회적 상호작용 요인이 SNS 실시간 동영상이용 만족도에 유의미한 영향을 미치지 않는다는 점을 발견하였다. 이는 기존의 SNS 실시간 동영상 서비스가 필요로 하는 기능과 관련하여 이론적, 실무적 함의를 제공하는데 기여할 수 있다고 하겠다.

Joint Rate and Power Optimization for Multimedia Streaming in Wireless Fading Channels via Parametric Policy Gradient

In this paper, we consider the two-timescale queueaware resource control for wireless video streaming applications. By applying a singular perturbation approach, we study a twotimescale joint optimization of transmission power and streaming rate control for real-time video streaming. The timescale parameter e is introduced to separate the state variables into slow and fast varying groups, and the power control is adaptive to both the channel state information and the playback queue state information (QSI) at the fast timescale, while the streaming rate representing the video quality is only adaptive to the QSI at the slow timescale. We first derive the averaged system by taking the perturbation parameter e → 0, and the limiting problem is a firm approximation for the formal problem with small e. Based on the averaged system, we derive the Hamilton-Jacobi-Bellman equation and show that the optimal control policy has a structural property which can further help us to simplify the problem. We use the parametric policy reinforcement learning algorithm to obtain the optimal parameterized policy, and show the algorithm convergence result. Finally, the proposed scheme's performance is compared with various baselines through simulations and it is shown that significant gain can be achieved.

Performance Evaluation of Multimedia Content over MPLS e-Learning Network

The growth in the use of multimedia in the educational sector is accelerating rapidly. Multimedia contents are very sensitive in terms of link capacity and other parameters affecting its quality. Due to the lack of existing of a network dedicated for testing and evaluating the e-learning network, there is a need to propose and build a testing environment. In this paper, we present a practical implementation of a testing environment that provide multimedia transmission to aid e-learning over a virtualized Multi-Protocol Label Switching (MPLS) network. The designed network combines hardware and software resources where the source is a real-time video streaming using a professional video camera connected to a virtual network of routers to deliver the multimedia content wirelessly to the intended classroom. The network is successfully implemented and evaluated with acceptable quality according to cisco reference for Quality of Service (QoS) Network Design Guide.

Edge Video Analytics for Public Safety: A Review

With the installation of enormous public safety and transportation infrastructure cameras, video analytics has come to play an essential part in public safety. Typically, video analytics is to collectively leverage the advanced computer vision (CV) and artificial intelligence (AI) to solve the four-W problem. That is to identify Who has done something (What) at a specific place (Where) at some time (When). According to the difference of latency requirements, video analytics can be applied to postevent retrospective analysis, such as archive management, search, forensic investigation and real-time live video stream analysis, such as situation awareness, alerting, and interested object (criminal suspect/missing vehicle) detection. The latter is characterized as having higher requirements on hardware resources as the sophisticated image processing algorithms under the hood. However, analyzing large-scale live video streams on the Cloud is impractical as the edge solution that conducts the video analytics on (or close to) the camera provides a silvering light. Analyzing live video streams on the edge is not trivial due to the constrained hardware resources on edge. The AI-dominated video analytics requires higher bandwidth, consumes considerable CPU/GPU resources for processing, and demands larger memory for caching. In this paper, we review the applications, algorithms, and solutions that have been proposed recently to facilitate edge video analytics for public safety.

Remote control of a robotic prosthesis arm with six-degree-of-freedom for ultrasonic scanning and three-dimensional imaging

Robotic ultrasound systems have turned into clinical use over the past decades, increasing precision and quality of medical operations. In this paper, we propose a robot-assisted and tele-controlled ultrasound scanning system for three-dimensional imaging. A six degree-of-freedom (DOF) robot arm acting as an upper-limb prosthesis remotely controlled by an operator drove the ultrasound probe to scan the skin surfaces. Via internet, the operator in a distant place monitored the robotic prosthesis and the patient through four video cameras on-site. According to the real-time video streams, the operator could manipulate a joystick for sending instructions from the remote operating terminal to the system on the patient side to adjust the probe motions. In this way, the safety and accuracy of the proposed system could be guaranteed. During the probe scanning, the acquired B-scans and their positional data were transferred to the remote workstation for the subsequent volume reconstruction and visualization. In-vitro and in-vivo experiments were conducted to validate the feasibility and performance of the proposed system. The quantitative experimental results show that the error for volume measurement was less than 1.1%, indicating that our system could accurately and flexibly control the probe scanning and produce 3D ultrasound images with high quality.

SIAT: A Distributed Video Analytics Framework for Intelligent Video Surveillance

In recent years, the amount of intelligent CCTV cameras installed in public places for surveillance has increased enormously and as a result, a large amount of video data is produced every moment. Due to this situation, there is an increasing request for the distributed processing of large-scale video data. In an intelligent video analytics platform, a submitted unstructured video undergoes through several multidisciplinary algorithms with the aim of extracting insights and making them searchable and understandable for both human and machine. Video analytics have applications ranging from surveillance to video content management. In this context, various industrial and scholarly solutions exist. However, most of the existing solutions rely on a traditional client/server framework to perform face and object recognition while lacking the support for more complex application scenarios. Furthermore, these frameworks are rarely handled in a scalable manner using distributed computing. Besides, existing works do not provide any support for low-level distributed video processing APIs (Application Programming Interfaces). They also failed to address a complete service-oriented ecosystem to meet the growing demands of consumers, researchers and developers. In order to overcome these issues, in this paper, we propose a distributed video analytics framework for intelligent video surveillance known as SIAT. The proposed framework is able to process both the real-time video streams and batch video analytics. Each real-time stream also corresponds to batch processing data. Hence, this work correlates with the symmetry concept. Furthermore, we introduce a distributed video processing library on top of Spark. SIAT exploits state-of-the-art distributed computing technologies with the aim to ensure scalability, effectiveness and fault-tolerance. Lastly, we implant and evaluate our proposed framework with the goal to authenticate our claims.

A high-dynamic range CMOS camera based on dual-gain channels

To overcome the ghosting phenomenon of multi-exposure technology, a new high-dynamic range (HDR) image processing method is proposed in this paper, which combines the features from dual-gain channel images. Further, a complete CMOS camera based on the HDR method is implemented, which produces a real-time HDR live video streams. This camera can capture the details of bright and dark areas in the scene completely with an extended dynamic range up to 95 dB. An ALTERA FPGA is the core processing unit of the entire camera, and it completes all the functional modules of the camera efficiently, including dual-channel video capture, image caching, HDR synthesis and tone mapping. Finally, the real-time HDR video flow has a display resolution of 1920 × 1080 and a frame rate of 60 fps.

A Hardware–Software Co-Design Framework for Real-Time Video Stabilization

Live digital video is a valuable source of information in security, broadcast and industrial quality control applications. Motion jitter due to camera and platform instability is a common artefact found in captured video which renders it less effective for subsequent computer vision tasks such as detection and tracking of objects, background modeling, mosaicking, etc. The process of algorithmically compensating for the motion jitter is hence a mandatory pre-processing step in many applications. This process, called video stabilization, requires estimation of global motion from consecutive video frames and is constrainted by additional challenges such as preservation of intentional motion and native frame resolution. The problem is exacerbated in the presence of local motion of foreground objects and requires robust compensation of the same. As such achieving real-time performance for this computationally intensive operation is a difficult task for embedded processors with limited computational and memory resources. In this work, development of an optimized hardware–software co-design framework for video stabilization has been investigated. Efficient video stabilization depends on the identification of key points in the frame which in turn requires dense feature calculation at the pixel level. This task has been identified to be most suitable for offloading the pipelined hardware implemented in the FPGA fabric due to the involvement of complex memory and computation operations. Subsequent tasks to be performed for the overall stabilization algorithm utilize these sparse key points and have been found to be efficiently handled in the software. The proposed Hardware–Software (HW–SW) co-design framework has been implemented on Zedboard FPGA platform which houses Xilinx Zynq SOC equipped with ARM A9 processor. The proposed implementation scheme can process real-time video stream input at 28 frames per second and is at least twice faster than the corresponding software-only approach. Two different hardware accelerator designs have been implemented using different high-level synthesis tools using rapid prototyping principle and consume less than 50% of logic resources available on the host FPGA while being at least 30% faster than contemporary designs.

Enabling real-time applications in Wi-Fi networks

Being of high importance, real-time applications, such as online gaming, real-time video streaming, virtual reality, and remote-control drone and robots, introduce many challenges to the developers of wireless networks. Such applications pose strict requirements on the delay and packet loss ratio, and it is hardly possible to satisfy them in Wi-Fi networks that use random channel access. The article presents a novel approach to enable real-time communications by exploiting an additional radio. This approach was recently proposed by us in the IEEE 802.11 Working Group and attracted much attention. To evaluate its gain and to study how real-time traffic coexists with the usual one, a mathematical model is designed. The numerical results show that the proposed approach allows decreasing the losses and delays for the real-time traffic by orders of magnitude, while the throughput for the usual traffic is reduced insignificantly in comparison to existing networks.

Cross-layer scheme for low latency multiple description video streaming over Vehicular Ad-hoc NETworks (VANETs)

There is nowadays a growing demand in vehicular communications for real-time applications requiring video assistance. The new state-of-the-art high-efficiency video coding (HEVC) standard is very promising for real-time video streaming. It offers high coding efficiency, as well as dedicated low delay coding structures. Among these, the all intra (AI) coding structure guarantees minimal coding time at the expense of higher video bitrates, which therefore penalizes transmission performances. In this work, we propose an original cross-layer system in order to enhance received video quality in vehicular communications. The system is low complex and relies on a multiple description coding (MDC) approach. It is based on an adaptive mapping mechanism applied at the IEEE 802.11p standard medium access control (MAC) layer. Simulation results in a realistic vehicular environment demonstrate that for low delay video communications, the proposed method provides significant video quality improvements on the receiver side.

Rearranging Online Tubes for Streaming Video Synopsis: A Dynamic Graph Coloring Approach.

To efficiently browse long surveillance videos, the video synopsis technique is often used to rearrange tubes (i.e., tracks of moving objects) along the temporal axis to form a much shorter video. In this process, two key issues need to be addressed, i.e., the minimization of spatial tube collision and the maximization of temporal video condensation. In addition, when a surveillance video comes as a stream, an online algorithm with the capability of dynamically rearranging tubes is also required. Toward this end, this paper proposes a novel graph-based tube rearrangement approach for online video synopsis. The relationships among tubes are modeled with a dynamic graph, whose nodes (i.e., object masks of tubes) and edges (i.e., relationships) can be progressively inserted and updated. Based on this graph, we propose a dynamic graph coloring algorithm to efficiently rearrange all tubes by determining when they should appear. Extensive experimental results show that our approach can condense online surveillance video streams in real time with less tube collision and high compact ratio.

Real-Time Eyeblink Detector and Eye State Classifier for Virtual Reality (VR) Headsets (Head-Mounted Displays, HMDs).

The aim of the study is to develop a real-time eyeblink detection algorithm that can detect eyeblinks during the closing phase for a virtual reality headset (VR headset) and accordingly classify the eye’s current state (open or closed). The proposed method utilises analysis of a motion vector for detecting eyelid closure, and a Haar cascade classifier (HCC) for localising the eye in the captured frame. When the downward motion vector (DMV) is detected, a cross-correlation between the current region of interest (eye in the current frame) and a template image for an open eye is used for verifying eyelid closure. A finite state machine is used for decision making regarding eyeblink occurrence and tracking the eye state in a real-time video stream. The main contributions of this study are, first, the ability of the proposed algorithm to detect eyeblinks during the closing or the pause phases before the occurrence of the reopening phase of the eyeblink. Second, realising the proposed approach by implementing a valid real-time eyeblink detection sensor for a VR headset based on a real case scenario. The sensor is used in the ongoing study that we are conducting. The performance of the proposed method was 83.9% for accuracy, 91.8% for precision and 90.40% for the recall. The processing time for each frame took approximately 11 milliseconds. Additionally, we present a new dataset for non-frontal eye monitoring configuration for eyeblink tracking inside a VR headset. The data annotations are also included, such that the dataset can be used for method validation and performance evaluation in future studies.

Streaming High-Definition Real-Time Video to Mobile Devices with Partially Reliable Transfer

Delivering High-Definition (HD) real-time video to mobile devices is challenged with stringent constraints in delay and reliability. In the presence of network dynamics (e.g., channel errors and bandwidth fluctuations), the existing error-control mechanisms [e.g., Automatic Repeat reQuest (ARQ) and Forward Error Correction (FEC)] frequently induce deadline violations and quality degradations. To strike an effective balance between delay and reliability in real-time video transmission, this research presents an application-layer solution dubbed P artial r E liability based R eal-tim E S treaming (PERES) to perform partially reliable transfer. First, we develop an analytical framework to model the delay-constrained partial reliability for acknowledgement (ACK) and negative acknowledgement (NAK) based real-time video streaming. Second, we propose scheduling algorithms for video-aware reliability adaptation and network-adaptive buffer control. PERES is able to maximize the transmission reliability of high-priority video frames within stringent delay constraint . We implement the proposed transmission scheme in embedded video monitoring systems and evaluate the efficacy over different wireless network environments. Evaluation results demonstrate PERES achieves appreciable improvements over the reference schemes in perceptual video quality, delay performance, and bandwidth efficiency.

Real time violence detection framework for football stadium comprising of big data analysis and deep learning through bidirectional LSTM

Football is the most popular sport in the world with four billion fans all over the world. Reportedly, the violence incidence rates are high during or after the matches. The violent or destructive behavior carried out by a person or player, who watches or plays the game in the stadium is known as football hooliganism. To prevent or control the violence, a real time violence detection system is exclusively needed to monitor the behavior of the crowd and players to take necessary action before the violence is about to happen. Even it is necessary for the system to find whether the attack is non-intentional or intentional in the game. In this paper, a real time violence detection system is proposed which processes the huge input streaming data and recognize the violence with human intelligence simulation. The input to the system is the enormous amount of real time video streams from different sources which is processed in Spark framework. In the Spark framework, the frames are separated and the features of individual frames are extracted by using HOG (Histogram of Oriented Gradients) function. Then the frames are labeled based on features as violence model, human part model and negative model, which are used to train the Bidirectional Long Short-Term Memory (BDLSTM) network for recognition of violence scenes. The bidirectional LSTM can access the information both in forward and reverse direction. Thus the output is generated in context to both past and future information. The network is trained with the violent interaction dataset (VID), containing 2314 videos with 1077 fight ones and 1237 no-fight ones. Moreover to make the model robust to violence detection, we have created a dataset with 410 video clips having non-violence scenes and 409 video clips having violence scenes, acquired from the football stadium. The performance of this model is validated and it proves the sturdiness of the system with an accuracy of 94.5 percentage in recognizing the violent action.

A fault tolerant proxy protocol preserving both data and timing and its usage in industrial tele maintenance

This paper shows the special needs of tele maintenance regarding Quality of Service (QoS). Then measurements of real-world communication links which are used by industry are presented and analyzed. Based on these results and the requirements of services which are needed for an efficient tele maintenance like real time video streaming, a novel data and time preserving error tolerant proxy (DaTPET proxy) was developed, which is explained in this paper. It finishes with the results and a demonstration in a real scenario.

Communication-Aware UAV Path Planning

Autonomous air drones, known as Unmanned Aerial Vehicles (UAVs), often carry out missions with real-time video streaming. In this work, we address the off-line mission plan problem of computing the optimal path from an origin to a final destination in two-dimensional space in order to maximize the quality of the communication, given a cellular coverage, and thus to provide throughput guarantees to the video streaming. In addressing the problem, we consider the energy budget constraint, the presence of wind in the area and the path smoothing problem. We propose novel path planning algorithms that are shown to outperform classical approaches that are oblivious of communication network coverage.

Distributed Deep Deterministic Policy Gradient for Power Allocation Control in D2D-Based V2V Communications

Device-to-device (D2D) communication is an emerging technology in the evolution of the 5G network enabled vehicle-to-vehicle (V2V) communications. It is a core technique for the next generation of many platforms and applications, e.g. real-time high-quality video streaming, virtual reality game, and smart city operation. However, the rapid proliferation of user devices and sensors leads to the need for more efficient resource allocation algorithms to enhance network performance while still capable of guaranteeing the quality-of-service. Currently, deep reinforcement learning is rising as a powerful tool to enable each node in the network to have a real-time self-organising ability. In this paper, we present two novel approaches based on deep deterministic policy gradient algorithm, namely “distributed deep deterministic policy gradient” and “sharing deep deterministic policy gradient”, for the multi-agent power allocation problem in D2D-based V2V communications. Numerical results show that our proposed models outperform other deep reinforcement learning approaches in terms of the network's energy efficiency and flexibility.

Congestion Control and Packet Scheduling for Multipath Real Time Video Streaming

Real time video traffic has stringent delay requirement and is bandwidth demanding when streaming over the internet. Single path transmission may provide unsatisfactory picture quality to users due to packet loss, jitter, and low bandwidth. Multipath transmission is another option to achieve better service by aggregating bandwidth and exploiting path diversity. As utility maximization theory shows, to achieve better satisfaction is to provide higher throughput and lower frame delivery delay for real time video traffic. The congestion control is responsible to probe maximum available bandwidth and packet scheduling is to take advantages of path diversity, which are two key research points in multipath context and are also the main concern in this paper. Given the BBR congestion control algorithm causes high-packet loss rate and large transmission delay when BBR flows competing for bandwidth resource, a delay response BBR (Delay-BBR) algorithm is proposed for real time video transmission. The main idea is that the sender will actively reduce the rate to let intermediate routers drain the excess occupied buffer once round trip delay signal exceeds a certain threshold. The results show that the delay-BBR can achieve lower transmission delay and lower packet loss rate when compared with QUIC-BBR and WebRTC-BBR. The rate stable feature makes delay-BBR appropriate for real time video transmission. Moreover, a packet scheduling algorithm works in coordination with the rate control algorithm achieves lower frame delivery delay further compared with the benchmark algorithms.

Dynamic Soft Real-Time Scheduling with Preemption Threshold for Streaming Media

Over the last decades, the advancements in networking technology and new multimedia devices have motivated the research on efficient video streaming mechanisms under wireless. We consider combing soft real-time video streaming scheduling with threshold to minimize the ineffective preemption. Based on the value density and urgency of soft real-time task, the dynamic scheduling with preemption threshold strategy (DSPT) is proposed in the paper. By analyzing the response time and preemption relationship of tasks, the preemption thresholds are assigned. Simulation results show that the DSPT strategy achieves improvements about success rate, delay time, and benefit of the system.