Multiple Video Streams Research Articles

Designing transport scheme of 3D naked-eye system

3D naked-eye is constructed from multi-stream as a typical representation of stereoscopic video. Its enormous data volume and stringent low-delay transport requirements pose significant challenges for high-quality real-time transport. Through analysis and experimental verification that current streaming media transport frameworks using the server–client or peer-to-peer scheme face difficulties when transmitting 3D naked-eye in a one-to-one format. Besides, the existing bandwidth estimation algorithms cannot achieve the expected performance when dealing with delay-sensitive traffic. This results in low bandwidth utilization and slow bandwidth estimation, rendering it unfeasible to deliver multi-stream on time. We propose an effective transport framework with different modules for real-time multi-stream and introduce an Agent-to-Agent transport scheme that provides many-to-one connection as the main implementation way of 3D naked-eye transport framework. Additionally, we propose a direct bandwidth estimation algorithm to quickly match network bandwidth for low-delay transport. The Agent terminal centrally processes consolidates transports, and provides macro-level management of multiple video streams. The algorithm directly detects the available bandwidth using packet interval and packet rate models. Finally, using rate decision algorithm arbitrates the results to directly measure the maximum available bandwidth of the link. The Agent-to-Agent achieves 99% bandwidth utilization, addressing the limitations of existing streaming schemes in handling concurrent data streams. Our algorithm provides precise bandwidth estimates with minimal time overhead, meeting the requirements of a delay-sensitive 3D naked-eye system across diverse environments.

A Liquidity Analysis System for Large-scale Video Streams in the Oilfield

This article introduces LinkStream, a liquidity analysis system based on multiple video streams designed and implemented for oilfield. LinkStream combines a variety of technologies to solve several problems in computing power and network latency. First, the system adopts an edge-central architecture and tailoring based on spatio-temporal correlation, which greatly reduces computing power requirements and network costs, and enables real-time analysis of large-scale video stream on limited edge devices. Second, it designed a set of liquidity information to describe the liquidity status in the oilfield. Finally, it uses object tracking technology to design a counting algorithm for the unique tubing object in the oilfield. We have deployed LinkStream in an oilfield in Iraq. LinkStream can perform real-time inference on over 200 video streams with acceptable resource overhead.

Enabling lightweight immersive user interaction in smart buildings through learning-based mobile panorama streaming

Smart buildings integrate users’ digitized wearables with their physical surroundings, creating a seamless and interactive user experience. This is achieved through the utilization of multiple sensors, video streaming, artificial intelligence, and edge computing. These technologies gather extensive data and provide users with a wide range of applications, such as 3D audio/video in AR/VR, localization, virtual tours, and vigilant monitoring. Nevertheless, the current AR/VR devices face limitations due to the bulkiness and discomfort of the hardware used for on-body sensing, such as headsets and specialized glasses. These components often become uncomfortable during prolonged usage, posing a challenge for creating an immersive system that combines lightweight interaction with high-quality presentation.This paper presents a comprehensive system designed to enable immersive interaction in smart buildings with a focus on lightweight solutions. The system consists of the following components: (1). A lightweight panoramic imaging framework to address the challenges related to hardware size and functionality. (2). A learning-based video transcoding cost prediction framework for efficient load balancing. (3). A layered networking architecture designed to facilitate high-quality mobile panorama live streaming. Collectively, these components offer lightweight interaction paired with enhanced presentation quality. Our experimental results demonstrate the effectiveness of the system design, showcasing its seamless operation across different times, geographical locations, and heterogeneous wireless networks.

Enhancing Crowd-Sourced Video Sharing through P2P-Assisted HTTP Video Streaming

This paper introduces a decentralized architecture designed for the sharing and distribution of user-generated video streams. The proposed system employs HTTP Live Streaming (HLS) as the delivery method for these video streams. In the architecture, a creator who captures a video stream using a smartphone camera subsequently transcodes it into a sequence of video chunks called HLS segments. These chunks are then stored in a distributed manner across the worker network, forming the core of the proposed architecture. Despite the presence of a coordinator for bootstrapping within the worker network, the selection of worker nodes for storing generated video chunks and autonomous load balancing among worker nodes are conducted in a decentralized fashion, eliminating the need for central servers. The worker network is implemented using the Golang-based IPFS (InterPlanetary File System) client, called kubo, leveraging essential IPFS functionalities such as node identification through Kademlia-DHT and message exchange using Bitswap. Beyond merely delivering stored video streams, the worker network can also amalgamate multiple streams to create a new composite stream. This bundling of multiple video streams into a unified video stream is executed on the worker nodes, making effective use of the FFmpeg library. To enhance download efficiency, parallel downloading with multiple threads is employed for retrieving the video stream from the worker network to the requester, thereby reducing download time. The result of the experiments conducted on the prototype system indicates that those concerned with the transmission time of the requested video streams compared with a server-based system using AWS exhibit a significant advantage, particularly evident in the case of low-resolution video streams, and this advantage becomes more pronounced as the stream length increases. Furthermore, it demonstrates a clear advantage in scenarios characterized by a substantial volume of viewing requests.

High speed human action recognition using a photonic reservoir computer

The recognition of human actions in videos is one of the most active research fields in computer vision. The canonical approach consists in a more or less complex preprocessing stages of the raw video data, followed by a relatively simple classification algorithm. Here we address recognition of human actions using the reservoir computing algorithm, which allows us to focus on the classifier stage. We introduce a new training method for the reservoir computer, based on “Timesteps Of Interest”, which combines in a simple way short and long time scales. We study the performance of this algorithm using both numerical simulations and a photonic implementation based on a single non-linear node and a delay line on the well known KTH dataset. We solve the task with high accuracy and speed, to the point of allowing for processing multiple video streams in real time. The present work is thus an important step towards developing efficient dedicated hardware for video processing.

EdgeLeague: Camera Network Configuration With Dynamic Edge Grouping for Industrial Surveillance

Object detection is crucial for surveillance in edge-enabled industrial Internet-of-Things (IIoT). Massive high-dimensional video streams without considering priority differences connect to edges via narrow and time-varying uplink channels, which should be analyzed efficiently for accurate and fast surveillance responses. However, time-varying network environments and constrained edge resources degrade surveillance's accuracy and real-time performance. This paper proposes EdgeLeague for multiple video streams with different quality of service, which maintains high surveillance performance under edge resource limitations and uplink bandwidth dynamics by edge collaboration and camera network configuration. The EdgeLeague scheme is formulated by an NP-hard integer non-linear problem to dynamically configure camera network resolutions and detection models on cooperative edges. To accelerate configuration responses, the formulated problem is decomposed into edge league grouping, video-league matching, and video configuration, solved by low-complexity algorithms. Theoretical analysis is provided for optimal video-league matching. Simulations show EdgeLeague achieves 0.312 s latency and 86.3% surveillance accuracy.

Edge computing-based unified condition monitoring system for process manufacturing

Condition monitoring is of great practical significance in modern manufacturing. This paper presents the implementation of an edge computing-based unified condition monitoring system, which provides all-round services for condition monitoring. These services include monitoring algorithm design and generation, monitoring approaches, and monitoring interfaces. With a visualized interactive interface, the monitoring algorithms can be designed online, generated into executable programs, and then implemented into corresponding edge nodes. The edge computing nodes directly act on the real-time data from production equipment which can improve response time for efficient condition monitoring. During the monitoring process, the parameters of the monitoring algorithms can be adjusted and applied in real time. Besides, the monitoring interface can be configured freely with multiple widgets, including charts and video streaming. The feasibility and practicability of the proposed monitoring system have been demonstrated through a real aluminum cold rolling case in process manufacturing.

Intelligent Caching for Mobile Video Streaming in Vehicular Networks with Deep Reinforcement Learning

Caching-enabled multi-access edge computing (MEC) has attracted wide attention to support future intelligent vehicular networks, especially for delivering high-definition videos in the internet of vehicles with limited backhaul capacity. However, factors such as the constrained storage capacity of MEC servers and the mobility of vehicles pose challenges to caching reliability, particularly for supporting multiple bitrate video streaming caching while achieving considerable quality of experience (QoE). Motivated by the above challenges, in this paper, we propose an intelligent caching strategy that takes into account vehicle mobility, time-varying content popularity, and backhaul capability to improve the QoE of vehicle users effectively. First, based on the mobile video mean opinion score (MV-MOS), we designed an average download percentage (ADP) weighted QoE evaluation model. Then, the video content caching problem is formulated as a Markov decision process (MDP) to maximize the ADP weighted MV-MOS. Owing to the prior knowledge of video content popularity and channel state information that may not be available at the road side unit in practical scenarios, we propose a deep reinforcement learning (DRL)-based caching strategy to solve the problem while achieving a maximum ADP weighted MV-MOS. To accelerate its convergence speed, we further integrate the prioritized experience replay, dueling, and double deep Q-network technologies, which improve the performance of DRL algorithm. Numerical results demonstrate that the proposed DRL-based caching strategy significantly improves QoE, and achieves better video delivery reliability compared to existing non-learning approaches.

A Scalable Analytics Pipeline for COVID-19 Face Mask Surveillance

The COVID-19 coronavirus pandemic still causes a global health crisis. An effective protection method is using a face mask in public areas, according to the World Health Organization (WHO). Computer vision systems can be allies in monitoring public areas where the face mask is mandatory. However, face mask detection is challenging due to many factors, including diversity of people, facial features, head accessories, mask design, image position, and lighting changes. To tackle these issues, we present the following contributions: a new balanced face mask dataset named UTFPR-FMD1, consisting of 61,430 images splitted into “face” and “mask” classes; a transfer learning classification model for computer vision tasks, trained with our dataset; a new processing pipeline that allows face mask detection in video streams. Unlike available public datasets with imbalanced class distributions, the UTFPR-FMD1 contains images from different people, gender, and ages to minimize the training difficulty of deep learning models. We introduced a new measure to select valid images to perform inferences. Experimental results show the effectiveness of our model, outperforming the state-of-art methods for face mask detection tasks. Additionally, and different from other authors, we also present qualitative results. The system can detect heads with up to 60 degrees of rotation and process up to 10 FPS. In future work, we will deploy the current framework into production, perform tests in a near real-time environment, and extend it to process multiple video streams.

Distributed Bandwidth Allocation Strategy for QoE Fairness of Multiple Video Streams in Bottleneck Links

In QoE fairness optimization of multiple video streams, a distributed video stream fairness scheduling strategy based on federated deep reinforcement learning is designed to address the problem of low bandwidth utilization due to unfair bandwidth allocation and the problematic convergence of distributed algorithms in cooperative control of multiple video streams. The proposed strategy predicts a reasonable bandwidth allocation weight for the current video stream according to its player state and the global characteristics provided by the server. Then the congestion control protocol allocates the proportion of available bandwidth, matching its bandwidth allocation weight to each video stream in the bottleneck link. The strategy trains a local predictive model on each client and periodically performs federated aggregation to generate the optimal global scheme. In addition, the proposed strategy constructs global parameters containing information about the overall state of the video system to improve the performance of the distributed scheduling algorithm. The experimental results show that the introduction of global parameters can improve the algorithm’s QoE fairness and overall QoE efficiency by 10% and 8%, respectively. The QoE fairness and overall QoE efficiency are improved by 8% and 7%, respectively, compared with the latest scheme.

A novel utilization-aware and power-delay-aware intelligent DMA controller for video streaming used in AI applications

PurposeIn real-time entertainment processing applications, processing of the multiple data streams demands high efficient multiple transfers, which leads to the computational overhead for system-on-chip (SoC), which runs the artificial intelligence algorithms. High-performance direct memory access controller (DMAC) is incorporated in SoC to perform the multiple data transfers without the participation of main processors. But achieving the area-efficient and power-aware DMAC suitable for streaming the multiple data remains to be a daunting challenge among the researchers.Design/methodology/approachThe purpose of this paper to provide the DMA operations without intervention of central processing unit (CPU) for bulk video data transmissions.FindingsThe proposed DMAC has been developed based on the hybrid advanced extensible interface (AXI)-PCI bus subsystem to handle the multiple data streams from the video sources. The proposed model consists of bus selector module, user control signal, status register, DMA-supported address and AXI-PCI subsystems to achieve better performance in analysing the video frames.Originality/valueThe extensive experimentation is carried out with Xilinx Zynq SoC architecture using Very High Speed integrated circuit hardware description language (VHDL) programming, and performance metrics such as utilization area and power are calculated and compared with the other existing DMA controllers such as Scatter-DMA, Gather-DMA and Enhanced DMA. Simulation results demonstrate that the proposed DMAC has outperformed other existing DMAC in terms of less area, less delay and power, which makes the proposed model suitable for streaming multiple video streams.

Adaptive Configuration Selection and Bandwidth Allocation for Edge-Based Video Analytics

Major cities worldwide have millions of cameras deployed for surveillance, business intelligence, traffic control, crime prevention, etc. Real-time analytics on video data demands intensive computation resources and high energy consumption. Traditional cloud-based video analytics relies on large centralized clusters to ingest video streams. With edge computing, we can offload compute-intensive analysis tasks to nearby servers, thus mitigating long latency incurred by data transmission via wide area networks. When offloading video frames from the front-end device to an edge server, the application configuration (i.e., frame sampling rate and frame resolution) will impact several metrics, such as energy consumption, analytics accuracy and user-perceived latency. In this paper, we study the configuration selection and bandwidth allocation for multiple video streams, which are connected to the same edge node sharing an upload link. We propose an efficient online algorithm, called JCAB, which jointly optimizes configuration adaption and bandwidth allocation to address a number of key challenges in edge-based video analytics systems, including edge capacity limitation, unknown network variation, intrusive dynamics of video contents. Our algorithm is developed based on Lyapunov optimization and Markov approximation, works online without requiring future information, and achieves a provable performance bound. We also extend the proposed algorithms to the multi-edge scenario in which each user or video stream has an additional choice about which edge server to connect. Extensive evaluation results show that the proposed solutions can effectively balance the analytics accuracy and energy consumption while keeping low system latency in a variety of settings.

Methodology to integrate ergonomics information in contextualized digital work instructions

Musculoskeletal disorders (MSDs), often caused by working in non-ergonomic circumstances, are the most important category of work-related diseases. Different methods exist for examining ergonomics through a variety of postural evaluation techniques (e.g. Rapid Entire Body Assessment (REBA)), which are mostly performed through manual survey-based observations. Consequently, measuring ergonomics of operators is time-consuming and happens only sporadically, often after problems occur. Furthermore, as an ergonomist must assign a score to the performed activity, the ergonomics evaluation is subjective. These problems can be resolved by continuous and automated ergonomic load monitoring methods that directly provide feedback to operators. In order to monitor the operator without interfering the task at hand, a vision-based ergonomics monitoring system is developed. Here, ergonomic features (e.g. joint angles) are estimated based on multiple video streams, which are used to calculate an objective ergonomic score using a standard ergonomics evaluation technique. Although cognitive operator support systems (i.e. digital work instructions) for manual assembly environments are widely used, they do typically not include ergonomics information. The goal of the methodology presented in this work is to a) formulate or update digital work instructions, b) provide the operator feedback on its ergonomic soundness via work instructions and c) generate ergonomic risk level reports in an automated way based on the outcomes of the vision-based ergonomics evaluator. Making the ergonomic guidelines context-aware ensures that interventions occur at the right time. The methodology is integrated in a framework for context-aware work instructions and is validated via a proof of concept, based on an actual industrial case. Results show that context-awareness enhances user acceptance and ergonomics of the operator.

Group-Based Adaptive Rendering System for 6DoF Immersive Video Streaming

The Moving Picture Experts Group (MPEG) has started an immersive media standard project to enable multi-view video and depth representation in three-dimensional (3D) scenes. The MPEG Immersive Video (MIV) standard technology is intended to provide a limited 6 degrees of freedom (DoF) based on depth map-based image rendering (DIBR). The 6DoF immersive video system is still challenging because multiple high-quality video streams require high bandwidth and computing resources. This paper proposes a group-based adaptive rendering method for 6DoF immersive video streaming.With group-based MIV, each group can be transmitted independently, which enables adaptive transmission depending on the user’s viewport. The proposed method derives weights from groups for view synthesis and allocates high-quality bitstreams according to a given viewport. This paper also discussed the results of the group-based approach in the MIV, and the advantages and drawbacks of this approach are detailed. In addition, pixel rate constraint analysis has been introduced to facilitate deployment with existing video codecs. On end-to-end evaluation metrics with TMIV anchor, the proposed method saves average 37.26% Bjontegaard-delta rate (BD-rate) on the peak signal-to-noise ratio (PSNR).

Context-based camera selection from multiple video streams

Context-based camera selection from multiple video streams

APPROACHES TO PROVIDING DATA VISUALIZATION ON DEVICES USING MODERN REAL-TIME OPERATING SYSTEMS

The paper considers various approaches to developing display controller driver software for embedded systems, which usually use System-on-chip (SOC) solutions and real-time operating systems (RTOS). It also describes the main principles and design decisions of the chosen RTOS, such as portability, flexible scheduling, responsiveness, etc., as well as used standards (C language, POSIX 1003.1, ARINC 653). The authors list general operating principles of the display controller hardware including support for several displays and overlays that can be used for displaying multiple video streams on one screen or achieving the effect similar to chroma keying. The proposed method for developing display controller drivers defines the main parameters of the display controller hardware and the steps necessary to show an image on a screen correctly. The meth-od also takes into account the features related to using hardware interrupts and estimating the frequen-cy required for display controller to show an image on a screen correctly with the defined screen mode. Contrary to the known methods, the proposed method takes into account various features of the many real-time operating systems: lack of dedicated API to interact with graphics hardware and resources, seamless access to hardware registers from user space and so on. The paper considers several ap-proaches to debugging software on the target hardware as well as using prototyping systems based on FPGA. Prototyping systems usually introduce additional challenges to debugging, such as low simula-tion speed. The proposed method was tested during the development of the display controller driver for the home made RTOS..

Implementation of Simultaneous Multi-Streaming of Live Solar Eclipse Video via 5.8 GHz AirMax

For the 2017 solar eclipse ballooning, we have developed a video payload that can simultaneously live-stream multiple videos via a single 5.8 GHz wireless link through a Ubiquiti Rocket M5 modem. In this paper, we describe our approach to multiplex multiple video streams into a single data stream that a 5.8 GHz wireless modem can transport to the ground station. Various key factors are described to properly configure the Raspberry Pis and optimize the transmission from an M5 modem on the video payload over the 5.8 GHz link while ensuring adequate range and acceptable video quality received at the ground station. A screenshot of the multi-video streaming is provided as an example to justify a successful operation of our video payload.

P‐5.5: An Adaptive Video Scaling System Implementation Based on FPGA

We propose an adaptive video scaling system based on bilinear interpolation implemented on FPGA. Different from the previous system for converting from a specific input resolution to a specified output resolution, we have realized adaptive output for multiple resolution video stream inputs. At the same time, the system's resource occupancy rate is small, which facilitates integration with other modules. Experimental results show that the system can achieve real—time low‐latency and high‐quality scaling for low‐resolution video streams, and theoretically can support 4K video super‐resolution output.

Real-Time Multiobject Tracking Based on Multiway Concurrency.

This paper explored a pragmatic approach to research the real-time performance of a multiway concurrent multiobject tracking (MOT) system. At present, most research has focused on the tracking of single-image sequences, but in practical applications, multiway video streams need to be processed in parallel by MOT systems. There have been few studies on the real-time performance of multiway concurrent MOT systems. In this paper, we proposed a new MOT framework to solve multiway concurrency scenario based on a tracking-by-detection (TBD) model. The new framework mainly focuses on concurrency and real-time based on limited computing and storage resources, while considering the algorithm performance. For the former, three aspects were studied: (1) Expanded width and depth of tracking-by-detection model. In terms of width, the MOT system can support the process of multiway video sequence at the same time; in terms of depth, image collectors and bounding box collectors were introduced to support batch processing. (2) Considering the real-time performance and multiway concurrency ability, we proposed one kind of real-time MOT algorithm based on directly driven detection. (3) Optimization of system level—we also utilized the inference optimization features of NVIDIA TensorRT to accelerate the deep neural network (DNN) in the tracking algorithm. To trade off the performance of the algorithm, a negative sample (false detection sample) filter was designed to ensure tracking accuracy. Meanwhile, the factors that affect the system real-time performance and concurrency were studied. The experiment results showed that our method has a good performance in processing multiple concurrent real-time video streams.

Toward Scalable Video Analytics Using Compressed-Domain Features at the Edge

Intelligent video analytics systems have come to play an essential role in many fields, including public safety, transportation safety, and many other industrial areas, such as automated tools for data extraction, and analyzing huge datasets, such as multiple live video streams transmitted from a large number of cameras. A key characteristic of such systems is that it is critical to perform real-time analytics so as to provide timely actionable alerts on various tasks, activities, and conditions. Due to the computation-intensive and bandwidth-intensive nature of these operations, however, video analytics servers may not fulfill the requirements when serving a large number of cameras simultaneously. To handle these challenges, we present an edge computing-based system that minimizes the transfer of video data from the surveillance camera feeds on a cloud video analytics server. Based on a novel approach of utilizing the information from the encoded bitstream, the edge can achieve low processing complexity of object tracking in surveillance videos and filter non-motion frames from the list of data that will be forwarded to the cloud server. To demonstrate the effectiveness of our approach, we implemented a video surveillance prototype consisting of edge devices with low computational capacity and a GPU-enabled server. The evaluation results show that our method can efficiently catch the characteristics of the frame and is compatible with the edge-to-cloud platform in terms of accuracy and delay sensitivity. The average processing time of this method is approximately 39 ms/frame with high definition resolution video, which outperforms most of the state-of-the-art methods. In addition to the scenario implementation of the proposed system, the method helps the cloud server reduce 49% of the load of the GPU, 49% that of the CPU, and 55% of the network traffic while maintaining the accuracy of video analytics event detection.