Real-time Video Streaming Research Articles

实时视频流缩放系统设计

实时视频流缩放系统设计

Autonomous flying blimp interaction with human in an indoor space

We present the Georgia Tech Miniature Autonomous Blimp (GT-MAB), which is designed to support human-robot interaction experiments in an indoor space for up to two hours. GT-MAB is safe while flying in close proximity to humans. It is able to detect the face of a human subject, follow the human, and recognize hand gestures. GT-MAB employs a deep neural network based on the single shot multibox detector to jointly detect a human user’s face and hands in a real-time video stream collected by the onboard camera. A human-robot interaction procedure is designed and tested with various human users. The learning algorithms recognize two hand waving gestures. The human user does not need to wear any additional tracking device when interacting with the flying blimp. Vision-based feedback controllers are designed to control the blimp to follow the human and fly in one of two distinguishable patterns in response to each of the two hand gestures. The blimp communicates its intentions to the human user by displaying visual symbols. The collected experimental data show that the visual feedback from the blimp in reaction to the human user significantly improves the interactive experience between blimp and human. The demonstrated success of this procedure indicates that GT-MAB could serve as a flying robot that is able to collect human data safely in an indoor environment.

Development of LTE based Real-time Video Streaming Drone System

Development of LTE based Real-time Video Streaming Drone System

A Video Streaming Transmission Scheme Based on Frame Priority in Device-to-Device Multicast Networks

Owing to a dramatic increase of traffic, high demand for quality of service (QoS), and insufficient radio resources, real-time video streaming transmission with stringent delay constraints has been intensely concerned. By exploiting the device-to-device (D2D) multicast communications, this paper proposes a video streaming transmission scheme based on the frame priority (FP) to improve the QoS perceived by users and the users’ satisfaction about the video quality. First, the FP strategy is proposed, which mainly considers the encoding characteristics of video streaming and users’ feedback to ensure that the frames to be retransmitted are valuable for decoding. Then, in order to transmit a sufficient number of valuable frames within the delay constraints, the optimization of time consumption of the retransmitted frame is formulated. Based on this, the physical-layer resource allocation in the D2D multicast networks is discussed, where the relay selection, D2D subgroup forming, and channel allocation are jointly investigated with the aid of three-dimensional channel quality matrix. Furthermore, a heuristic algorithm is proposed to obtain a near-optimal performance with low complexity. The simulation results verify the advantages of our proposed transmission scheme in users’ video reception quality and the satisfaction of all users.

Design and Performance Evaluation of Content-Oriented Communication System for IoT Network: A Case Study of Named Node Networking for Real-Time Video Streaming System

Information-Centric Networking (ICN) was born in an era that more and more users are shifting their interests to the content itself rather than the location where contents are stored. This paradigm shift in the usage patterns of the Internet, along with the urgent needs for content naming, pervasive caching, better security, and mobility support, has prompted researchers to consider a radical change to the Internet architecture. However, ICN is still in its infancy and development stage, and many issues still exist and need to be addressed. The common ICN architecture is lacking the host-centric communication ability and difficult to provide seamless mobility in current solutions. To solve this problem, our team had proposed the Named-Node Networking (3N) concept, which not only naming the content but also naming the node and it proved to have better performance of providing seamless mobility in the simulation. However, the previous contributions were limited in the 3N namespace for seamless mobility support in both producer and consumer. In this paper, we have proposed a 3N system which includes 3N naming, data delivery, mobility support, and data security. Moreover, we have created a 3N-based real-time video streaming system to evaluate data delivery performance and mobility handoff performance. The evaluation result proofs that our system performs better than TCP video streaming in a multi-client situation and a WiFi-based handoff was demonstrated.

Personalized Real-Time Virtual Tours In Places With Cultural Interest

Virtual tours using drones enhance the experience the users perceive from a place with cultural interest. Drones equipped with 360o cameras perform real-time video streaming of the cultural sites. The user preferences about each monument type should be considered in order to decide the appropriate flying route for the drone. This article describes a scheme for supporting personalized real-time virtual tours at sites with cultural interest using drones. The user preferences are modeled using the MPEG-21 and the MPEG-7 standards, while Web Ontology Language (OWL) ontologies are used for metadata structure and semantics. The Metadata-Aware Analytic Network Process (MANP) algorithm is proposed in order to weigh the user preferences for each monument type. Subsequently, the Trapezoidal Fuzzy Topsis for Heritage Route Selection (TFT-HRS) algorithm ranks the candidate heritage routes. Finally, after each virtual tour, the user preferences metadata are updated in order in order the scheme to continuously learn about the user preferences.

The Extended Multidimensional Neo-Fuzzy System and Its Fast Learning in Pattern Recognition Tasks

Methods of machine learning and data mining are becoming the cornerstone in information technologies with real-time image and video recognition methods getting more and more attention. While computational system architectures are getting larger and more complex, their learning methods call for changes, as training datasets often reach tens and hundreds of thousands of samples, therefore increasing the learning time of such systems. It is possible to reduce computational costs by tuning the system structure to allow fast, high accuracy learning algorithms to be applied. This paper proposes a system based on extended multidimensional neo-fuzzy units and its learning algorithm designed for data streams processing tasks. The proposed learning algorithm, based on the information entropy criterion, has significantly improved the system approximating capabilities. Experiments have confirmed the efficiency of the proposed system in solving real-time video stream recognition tasks.

ROS-based Multiple Cameras Video Streaming for a Teleoperation Interface of a Crawler Robot

Digital cameras are being widely used in robotics in 3D scene reconstruction, simultaneous localization and mapping (SLAM), and other applications, which often require real-time video stream from multiple cameras. In a case of a mobile robotic system with limited hardware capabilities performance issues may arise. In this work we present a real-time video streaming robot operating system (ROS) package for our crawler-type mobile robot Servosila Engineer, which has four on-board cameras. Compared to OpenCV based solution our package demonstrates less resource consumption.

Energy-Efficient Traffic Rate Adaptation for Wireless Streaming Media Transmission

Energy-efficient streaming of real-time scalable video over wireless fading channels with imperfect channel state information is considered in this paper. A traffic rate adaptation (TRA) scheme is presented to minimize the transmission energy consumption while satisfying the quality of experience (QoE) requirement by dynamically selecting the delivered encoding layers of streaming media depending on the channel condition. A partially observable Markov decision process model is introduced to formulate the energy-efficient QoE-guaranteed TRA problem. Based on the Lagrange method of multipliers, an online policy iteration algorithm is proposed to solve the constrained optimization problem. This algorithm performs policy evaluation and improvement without intensive computations and prior knowledge of channel statistics. The adaptability to unknown environments and high computation efficiency make the proposed scheme attractive for implementation in real-life mobile systems. Simulation results demonstrate the effectiveness of the proposed method.

Fast and Resource-Efficient Hardware Implementation of Modified Line Segment Detector

Lines are significant features enclosing high-level information in an image. The line segment Detector (LSD) Algorithm with low error rate is a widely used method to extract lines in images effectively and accurately. However, the algorithm on PC performs too costly both in time and resources for the real-time video processing. This paper provides a fast and resource-efficient hardware implementation solution for a modified LSD algorithm on Field Programmable Gate Arrays (FPGA) for real-time line detection. The task-level pipeline structures are exploited fully in a stream process mapped to the hardware architecture free of frame buffer. Our proposed hardware implementation processes in a stream-in–stream-out manner with little consumption of the on-chip block RAM to store intermediate values. We first employ hardware Gaussian filter and adjust Canny edge detection to obtain an edge map at single-pixel width. Then, a novel structure of region growing model based on dynamic rooted tree is used to detect line segment regions accurately with a latency of only a few rows of pixels. The low cost in time, on-chip resources, and power consumption makes our proposed algorithm suitable for portable real-time streaming video processing applications using line segment features, such as Lane departure warning systems. It can also be applied in real-time machine vision systems that use line segments information for further recognition or stereo correspondence and many others. The proposed algorithm is synthesized and tested on XC7Z020 FPGA with high reliability, accuracy speed, and low cost in both resources and energy.

Modeling and Analysis of FPGA Accelerators for Real-Time Streaming Video Processing in the Healthcare Domain

Complex real-time video processing applications with strict throughput constraints are commonly found in a typical healthcare application. The video processing chain is implemented as Field-Programmable Gate Array (FPGA) accelerators (processing blocks) communicating through a number of First-In First-Out (FIFO) buffers. The FIFO buffers are made out of Block RAM (BRAM) and limited in availability. Therefore, a key design question is the optimal sizes of the FIFO buffers with respect to the throughput constraint. In this paper, we use model-driven analysis and detailed hardware level simulation to address the question of buffer dimensioning in an efficient way. Using a Cyclo-Static Dataflow (CSDF) model and an optimization method, we identify and optimize the FIFO buffers. The results are confirmed using a detailed hardware level simulation and validated by comparison with VHDL simulations. The technique is illustrated on a use case from Philips Healthcare Image Guided Therapy (IGT) on the imaging pipeline of an Interventional X-Ray (i XR) system.

Real-time video stream image tracking

The problem of counting cars on a highway using standard video surveillance equipment was set and solved with the OpenCV computer vision library. The library made possible to grab each video stream frame in real time and process incoming information. Data processing was performed using a segmented neural network. Accuracy in 70% of correct definition of cars was reached.

A Low-Cost Pupil Center Localization Algorithm Based on Maximized Integral Voting of Circular Hollow Kernels

Abstract Pupil center localization is an essential requirement for robust eye gaze tracking systems. In this paper, a low-cost pupil center localization algorithm is presented. The aim is to propose a computationally inexpensive algorithm with high accuracy in terms of performance and processing speed. Hence, a computationally inexpensive pupil center localization algorithm based on maximized integral voting of candidate kernels is presented. As the kernel type, a novel circular hollow kernel (CHK) is used. Estimation of pupil center is employed by applying a rule-based schema for each pixel over the eye sockets. Additionally, several features of CHK are proposed for maximizing the probability of voting for each kernel. Experimental results show promising results with 96.94% overall accuracy with around 13.89 ms computational time (71.99 fps) for a single image as an average time by using a standard PC. An extensive benchmarking study indicates that this method outperforms the learning-free algorithms and it competes with the other methods having a learning phase while their processing speed is much higher. Furthermore, this proposed learning-free system is fast enough to run on an average PC and also applicable to work with even a low-resolution webcam on a real-time video stream.

Optical-acoustic hybrid network toward real-time video streaming for mobile underwater sensors

Underwater sensor networking is generally regarded as an emerging technology in conducting oceanic exploration and research in an automated and effective manner. As underwater operations become more sophisticated and Autonomous Underwater Vehicles (AUVs) become more advanced, there is an increasing demand for real-time video streaming from AUVs to remotely steer them and to probe the environment. However, real-time video streaming requires high bandwidth. To help overcome this obstacle, we propose a hybrid solution that combines acoustic and optical communications. In our hybrid solution, optics provide good quality real-time video streaming. Acoustic maintains a “thin” channel for the network topology and transmission control. The acoustic channel is also used for still frame video delivery when the optical channel fails. In particular, we enable optical communications by acoustic-assisted alignment and use acoustic communications as a backup when the optical signal is interrupted. The main contribution of this research is to enable reliable, real-time video streaming without underwater optical cables. Another important contribution is the smooth transition between the acoustic and optical video delivery mode, by leveraging image processing algorithms to compress the key frames before transmitting them on the acoustic channel.

Performance evaluation of support vector machine and convolutional neural network algorithms in real-time vehicle type and color classification

In order for traffic management and information systems to provide proper traffic flow, it is necessary to obtain information about traffic with the help of various sensors. In this context, in recent years the use of video cameras in traffic observation and control has become very widespread and actively used. Numerous studies such as license plate recognition, vehicle number finding, traffic intensity determination, vehicle speed calculation, band violation and vehicle classification can be done with the help of video processing based video monitoring systems. Traffic surveillance videos are very actively used for this purpose. In this paper, we have developed a system that classifies vehicles according to their type. Firstly we create a vehicle dataset from an uncalibrated camera. Then, we test Tiny-YOLO real-time object detection and classification system and support vector machine (SVM) classifier model on our dataset and well-known public BIT-Vehicle dataset in terms of recall, precision, and intersection over union performance metrics. Experimental results show that two methods can be used to classify real-time streaming traffic video data.

Detection of abnormal behavior in narrow scene with perspective distortion

Automatically and efficiently detecting abnormal events that occur in dynamic surveillance video is one of the important tasks in real-time video streaming analysis. However, in some typical application scenarios with narrow areas, the perspective distortion caused by the large depth-of-field has a tremendous negative impact on the accuracy of detection, thereby increasing the difficulty of identifying abnormal behavior. Taking the real-time violence occurring in the metro platform as an example, the article introduces a more effective algorithm for detecting abnormal behaviors in narrow areas with perspective distortion. The algorithm firstly uses the adaptive transformation mechanism to make up for the distorting effect in the region of interest extraction. Then, an improved pyramid L–K optical flow method with perspective weight and disorder coefficient is proposed to extract the abnormal behavior feature occurred in historical moving images. The side-by-side comparison of the experimental results proves that this algorithm can effectively compensate for the distortion effect and obviously improve the accuracy of abnormal behavior detection in narrow area scenes.

FAST STEREO IMAGE DEPTH ESTIMATION ALGORITHM BASED ON CHANGE DETECTION METHODS IN CAMERAS FIELD OF VIEW

Stereo image depth estimation algorithms have not been widely used in CCTV systems yet, since either their accuracy is low or their computational complexity is high, which prevents implementation of these algorithms due to economic limitations on the cost of equipment used in security systems. In this paper, a new fast algorithm is proposed for reconstructing the depth of stereo images, which is used for reliable event identification in the CCTV systems cameras field of view. This real-time video stream processing method is based on the use of a fast algorithm for detecting changes in the cameras scene in the combination with SGBM depth estimation algorithm, which processes only image areas containing scene changes. The proposed method of significant computational complexity reduction of the depth estimation algorithm makes possible to acquire information about the distance from the cameras to moving objects in the field of view and use this information as an additional feature, that helps to reduce the operational cost and to improve the reliability of CCTV systems.

A templated programmable architecture for highly constrained embedded HD video processing

The implementation of a video reconstruction pipeline is required to improve the quality of images delivered by highly constrained devices. These algorithms require high computing capacities—several dozens of GOPs for real-time HD 1080p video streams. Today’s embedded design constraints impose limitations both in terms of silicon budget and power consumption—usually 2 mm\(^2\) for half a Watt. This paper presents the eISP architecture that is able to reach 188 MOPs/mW with 94 GOPs/mm\(^2\) and 378 GOPs/mW using TSMC 65-nm integration technology. This fully programmable and modular architecture, is based on an analysis of video-processing algorithms. Synthesizable VHDL is generated taking into account different parameters, which simplify the architecture sizing and characterization.

Design and development of an aircraft type portable drone for surveillance and disaster management

PurposeThe purpose of this paper is to demonstrate the development of an aircraft-type autonomous portable drone suitable for surveillance and disaster management. The drone is capable of flying at a maximum speed of 76 km/h. This portable drone comprises five distinct parts those are easily installable within several minutes and can be fit in a small portable kit. The drone consists of a ballistic recovery system, allowing the drone landing vertically. The integrated high-definition camera sends real-time video stream of desired area to the ground control station. In addition, the drone is capable of carrying ~1.8 kg of payload.Design/methodology/approachIn order to design and develop the portable drone, the authors sub-divided the research activities in six fundamental steps: survey of the current drone technologies, design the system architecture of the drone, simulation and modeling of various modules of the drone, development of various modules of the drone and their performance analysis, integration of various modules of the drone, and real-life performance analysis and finalization.FindingsExperimental results: the cruise speed of the drone was in the range between 45 and 62 km/h. The drone was capable of landing vertically using the ballistic recovery system attached with it. On the contrary, the drone can transmit real-time video to the ground control station and, thus, suitable for surveillance. The audio system of the drone can be used for announcement of emergency messages. The drone can carry 1.8 kg of payload and can be used during disaster management. The drone parts are installed within 10 min and fit in a small carrying box.Practical implicationsThe autonomous aircraft-type portable drone has a wide range of applications including surveillance, traffic jam monitoring and disaster management.Social implicationsThe cost of the cost-effective drone is within $700 and creates opportunities for the deployment in the least developed countries.Originality/valueThe autonomous aircraft-type portable drone along with the ballistic recovery system were designed and developed by the authors using their won technology.

Network Coding Aided Collaborative Real-Time Scalable Video Transmission in D2D Communications

There has been an increasing demand for providing real-time video streaming services in the next-generation cellular networks. To improve the quality of such services without additional infrastructure, neighboring devices can recover missing packets by using network coding aided collaborative transmission via device-to-device (D2D) communication. However, most existing work in this area had not considered the issue of how to schedule such coding aided collaborative transmissions effectively for supporting real - time scalable video applications in such environment. In this paper, we study how to improve the quality of real-time scalable video services by efficiently scheduling coding aided collaborative transmissions. We first formulate the problem of optimal collaborative transmission scheduling that determines the optimal transmitting sequence and coding pattern at each transmitting device, which is shown to be NP-hard. To address this problem, we propose a new weight function for measuring the quality of a coding pattern by considering packet recovery gain and potential video decoding gain at receivers. Based on this new weight function, we propose a low complexity centralized algorithm using global state information and an efficient distributed mechanism supporting localized operations in dynamic environment. We deduce their computational complexities. Simulation results verify that the proposed solution outperforms the representative work in the literature.