High-resolution Video Streams Research Articles

Semantic guidance incremental network for efficiency video super-resolution

AbstractIn video streaming, bandwidth constraints significantly affect client-side video quality. Addressing this, deep neural networks offer a promising avenue for implementing video super-resolution (VSR) at the user end, leveraging advancements in modern hardware, including mobile devices. The principal challenge in VSR is the computational intensity involved in processing temporal/spatial video data. Conventional methods, uniformly processing entire scenes, often result in inefficient resource allocation. This is evident in the over-processing of simpler regions and insufficient attention to complex regions, leading to edge artifacts in merged regions. Our innovative approach employs semantic segmentation and spatial frequency-based categorization to divide each video frame into regions of varying complexity: simple, medium, and complex. These are then processed through an efficient incremental model, optimizing computational resources. A key innovation is the sparse temporal/spatial feature transformation layer, which mitigates edge artifacts and ensures seamless integration of regional features, enhancing the naturalness of the super-resolution outcome. Experimental results demonstrate that our method significantly boosts VSR efficiency while maintaining effectiveness. This marks a notable advancement in streaming video technology, optimizing video quality with reduced computational demands. This approach, featuring semantic segmentation, spatial frequency analysis, and an incremental network structure, represents a substantial improvement over traditional VSR methodologies, addressing the core challenges of efficiency and quality in high-resolution video streaming.

Using Physical Dynamics: Accurate and Real-Time Object Detection for High-Resolution Video Streaming on Internet of Things Devices

Using Physical Dynamics: Accurate and Real-Time Object Detection for High-Resolution Video Streaming on Internet of Things Devices

A Smart System for Future Generation based on the Internet of Things Employing Machine Learning, Deep Learning, and Artificial Intelligence : Comprehensive Survey

The Internet of Things (IoT) is a networked system including interconnected things, devices, and networks that utilize the internet for communication and data exchange. The entity engages in interactions with both its internal and external surroundings. The IoT is capable of seeing the surrounding environment and responding in a way that is appropriate and adaptive. The utilization of advanced technology in this context enhances the environment and thus enhances the overall well-being of humanity. The IoT facilitates inter-device communication, whether through physical or virtual means. The IoT facilitates the enhancement of environmental intelligence, enabling seamless connectivity across many devices at any given moment. The concepts centred on the IoT, such as augmented reality, high-resolution video streaming, autonomous vehicles, intelligent environments, and electronic healthcare, have become pervasive in contemporary society. These applications have requirements for faster data rates, larger bandwidths, enhanced capacities, decreased latencies, and increased throughputs. IoT and Machine learning (ML) are among the fields of research that have shown significant potential for advancement. ML and IoT are used to build intelligent systems. Those networks will modify the ways in which worldwide entities exchange information. This article gives a comprehensive survey of the upcoming 5G-IoT situation, as well as a study of IoT smart system applications and usages. In addition to covering the latest developments in ML and deep learning (DL) and their impact on 5G-IoT, this article describes a comprehensive study of these important enabling technologies and the developing use cases of 5G-IoT.

Reconstruction of the Instantaneous Images Distorted by Surface Waves via Helmholtz–Hodge Decomposition

Imaging through water waves will cause complex geometric distortions and motion blur, which seriously affect the correct identification of an airborne scene. The current methods main rely on high-resolution video streams or a template image, which limits their applicability in real-time observation scenarios. In this paper, a novel recovery method for the instantaneous images distorted by surface waves is proposed. The method first actively projects an adaptive and adjustable structured light pattern onto the water surface for which random fluctuation will cause the image to degrade. Then, the displacement field of the feature points in the structured light image is used to estimate the motion vector field of the corresponding sampling points in the scene image. Finally, from the perspective of fluid mechanics, the distortion-free scene image is reconstructed based on the Helmholtz-Hodge Decomposition (HHD) theory. Experimental results show that our method not only effectively reduces the distortion to the image, but also significantly outperforms state-of-the-art methods in terms of computational efficiency. Moreover, we tested the real-scene sequences of a certain length to verify the stability of the algorithm.

Real-Time Efficient FPGA Implementation of the Multi-Scale Lucas-Kanade and Horn-Schunck Optical Flow Algorithms for a 4K Video Stream.

The information about optical flow, i.e., the movement of pixels between two consecutive images from a video sequence, is used in many vision systems, both classical and those based on deep neural networks. In some robotic applications, e.g., in autonomous vehicles, it is necessary to calculate the flow in real time. This represents a challenging task, especially for high-resolution video streams. In this work, two gradient-based algorithms—Lucas–Kanade and Horn–Schunck—were implemented on a ZCU 104 platform with Xilinx Zynq UltraScale+ MPSoC FPGA. A vector data format was used to enable flow calculation for a 4K (Ultra HD, 3840 × 2160 pixels) video stream at 60 fps. In order to detect larger pixel displacements, a multi-scale approach was used in both algorithms. Depending on the scale, the calculations were performed for different data formats, allowing for more efficient processing by reducing resource utilisation. The presented solution allows real-time optical flow determination in multiple scales for a 4K resolution with estimated energy consumption below 6 W. The algorithms realised in this work can be a component of a larger vision system in advanced surveillance systems or autonomous vehicles.

The Impact of Cloud-Based Live Streaming Technologies on Mobile Applications: Development and Future Trends

The proliferation of cloud-based live streaming technologies has significantly impacted the development and functionality of mobile applications. These technologies, leveraging scalable cloud infrastructure, offer developers a robust platform for delivering real-time streaming content, improving user engagement, and expanding service capabilities. This paper explores the transformative effects of cloud-based live streaming on mobile application development, emphasizing key innovations, current challenges, and future trends. Cloud-based live streaming technologies enable mobile applications to handle high-resolution video streams with minimal latency, thanks to advanced cloud infrastructure and Content Delivery Networks (CDNs). The elasticity and scalability of cloud services allow for dynamic adjustment of resources based on user demand, ensuring a seamless viewing experience across diverse network conditions. This has led to significant improvements in video quality and accessibility, making live streaming more reliable and engaging for end-users.

Design and Implementation of a UAV-Based Airborne Computing Platform for Computer Vision and Machine Learning Applications.

Visual sensing of the environment is crucial for flying an unmanned aerial vehicle (UAV) and is a centerpiece of many related applications. The ability to run computer vision and machine learning algorithms onboard an unmanned aerial system (UAS) is becoming more of a necessity in an effort to alleviate the communication burden of high-resolution video streaming, to provide flying aids, such as obstacle avoidance and automated landing, and to create autonomous machines. Thus, there is a growing interest on the part of many researchers in developing and validating solutions that are suitable for deployment on a UAV system by following the general trend of edge processing and airborne computing, which transforms UAVs from moving sensors into intelligent nodes that are capable of local processing. In this paper, we present, in a rigorous way, the design and implementation of a kg UAV system equipped with the necessary computational power and sensors to serve as a testbed for image processing and machine learning applications, explain the rationale behind our decisions, highlight selected implementation details, and showcase the usefulness of our system by providing an example of how a sample computer vision application can be deployed on our platform.

TRex, a fast multi-animal tracking system with markerless identification, and 2D estimation of posture and visual fields.

Automated visual tracking of animals is rapidly becoming an indispensable tool for the study of behavior. It offers a quantitative methodology by which organisms' sensing and decision-making can be studied in a wide range of ecological contexts. Despite this, existing solutions tend to be challenging to deploy in practice, especially when considering long and/or high-resolution video-streams. Here, we present TRex, a fast and easy-to-use solution for tracking a large number of individuals simultaneously using background-subtraction with real-time (60 Hz) tracking performance for up to approximately 256 individuals and estimates 2D visual-fields, outlines, and head/rear of bilateral animals, both in open and closed-loop contexts. Additionally, TRex offers highly accurate, deep-learning-based visual identification of up to approximately 100 unmarked individuals, where it is between 2.5 and 46.7 times faster, and requires 2-10 times less memory, than comparable software (with relative performance increasing for more organisms/longer videos) and provides interactive data-exploration within an intuitive, platform-independent graphical user-interface.

Object Detection-Tracking Algorithm for Unmanned Surface Vehicles Based on a Radar-Photoelectric System

Object tracking is an important basis for the autonomous navigation of unmanned surface vehicles. However, several problems still must be addressed for a wide applicating of object tracking in unmanned surface vehicles. First, if multiple objects of the same classification exist in the same field of view, then stable extraction of an object is difficult. Second, in an environment with a complex background and large changes in object shape, the tracking accuracy is low, and object tracking errors and tracking loss can easily occur. Third, much time is required to detect a high-resolution real-time video stream, not meeting the delay requirement of the photoelectric servo stable tracking. To resolve these problems, this paper proposes an object detection-tracking algorithm based on a radar-photoelectric system. The algorithm combines an object detection algorithm with an object tracking algorithm and involves the following steps. First, a first-frame object extraction algorithm is used to extract the tracking object from the first frame. Second, a region of interest (ROI)-prediction algorithm is used to predict ROIs and detect objects in these ROIs. This algorithm can effectively solve the above problems in marine tests. When multiple objects of the same classification exist in the same field of view, the algorithm can extract the radar-guided object stably. When faced with a complex background and a large change in object shape, the algorithm substantially improves the accuracy and robustness of object tracking. Compared with the conventional object detection algorithm, the time consumption of this algorithm is reduced by 25.8%.

Internet of Things (IoT) for Next-Generation Smart Systems: A Review of Current Challenges, Future Trends and Prospects for Emerging 5G-IoT Scenarios

The Internet of Things (IoT)-centric concepts like augmented reality, high-resolution video streaming, self-driven cars, smart environment, e-health care, etc. have a ubiquitous presence now. These applications require higher data-rates, large bandwidth, increased capacity, low latency and high throughput. In light of these emerging concepts, IoT has revolutionized the world by providing seamless connectivity between heterogeneous networks (HetNets). The eventual aim of IoT is to introduce the plug and play technology providing the end-user, ease of operation, remotely access control and configurability. This paper presents the IoT technology from a bird’s eye view covering its statistical/architectural trends, use cases, challenges and future prospects. The paper also presents a detailed and extensive overview of the emerging 5G-IoT scenario. Fifth Generation (5G) cellular networks provide key enabling technologies for ubiquitous deployment of the IoT technology. These include carrier aggregation, multiple-input multiple-output (MIMO), massive-MIMO (M-MIMO), coordinated multipoint processing (CoMP), device-to-device (D2D) communications, centralized radio access network (CRAN), software-defined wireless sensor networking (SD-WSN), network function virtualization (NFV) and cognitive radios (CRs). This paper presents an exhaustive review for these key enabling technologies and also discusses the new emerging use cases of 5G-IoT driven by the advances in artificial intelligence, machine and deep learning, ongoing 5G initiatives, quality of service (QoS) requirements in 5G and its standardization issues. Finally, the paper discusses challenges in the implementation of 5G-IoT due to high data-rates requiring both cloud-based platforms and IoT devices based edge computing.

Deep Video Precoding

Several groups worldwide are currently investigating how deep learning may advance the state-of-the-art in image and video coding. An open question is how to make deep neural networks work in conjunction with existing (and upcoming) video codecs, such as MPEG H.264/AVC, H.265/HEVC, VVC, Google VP9 and AOMedia AV1, AV2, as well as existing container and transport formats, without imposing any changes at the client side. Such compatibility is a crucial aspect when it comes to practical deployment, especially when considering the fact that the video content industry and hardware manufacturers are expected to remain committed to supporting these standards for the foreseeable future. We propose to use deep neural networks as precoders for current and future video codecs and adaptive video streaming systems. In our current design, the core precoding component comprises a cascaded structure of downscaling neural networks that operates during video encoding, prior to transmission. This is coupled with a precoding mode selection algorithm for each independently-decodable stream segment, which adjusts the downscaling factor according to scene characteristics, the utilized encoder, and the desired bitrate and encoding configuration. Our framework is compatible with all current and future codec and transport standards, as our deep precoding network structure is trained in conjunction with linear upscaling filters (e.g., the bilinear filter), which are supported by all web video players. Extensive evaluation on FHD (1080p) and UHD (2160p) content and with widely-used H.264/AVC, H.265/HEVC and VP9 encoders, as well as a preliminary evaluation with the current test model of VVC (v.6.2rc1), shows that coupling such standards with the proposed deep video precoding allows for 8% to 52% rate reduction under encoding configurations and bitrates suitable for video-on-demand adaptive streaming systems. The use of precoding can also lead to encoding complexity reduction, which is essential for cost-effective cloud deployment of complex encoders like H.265/HEVC, VP9 and VVC, especially when considering the prominence of high-resolution adaptive video streaming.

High-Throughput and Low-Latency Digital Baseband Architecture for Energy-Efficient Wireless VR Systems

This paper presents a novel baseband architecture that supports high-speed wireless VR solutions using 60 GHz RF circuits. Based on the experimental observations by our previous 60 GHz transceiver circuits, the efficient baseband architecture is proposed to enhance the quality of transmission. To achieve a zero-latency transmission, we define an (106,920, 95,040) interleaved-BCH error-correction code (ECC), which removes iterative processing steps in the previous LDPC ECC standardized for the near-field wireless communication. Introducing the block-level interleaving, the proposed baseband processing successfully scatters the existing burst errors to the small-sized component codes, and recovers up to 1080 consecutive bit errors in a data frame of 106,920 bits. To support the high-speed wireless VR system, we also design the massive-parallel BCH encoder and decoder, which is tightly connected to the block-level interleaver and de-interleaver. Including the high-speed analog interfaces for the external devices, the proposed baseband architecture is designed in 65 nm CMOS, supporting a data rate of up to 12.8 Gbps. Experimental results show that the proposed wireless VR solution can transfer up to 4 K high-resolution video streams without using time-consuming compression and decompression, successfully achieving a transfer latency of 1 ms.

Canny edge detection and Hough transform for high resolution video streams using Hadoop and Spark

Nowadays, video cameras are increasingly used for surveillance, monitoring, and activity recording. These cameras generate high resolution image and video data at large scale. Processing such large scale video streams to extract useful information with time constraints is challenging. Traditional methods do not offer scalability to process large scale data. In this paper, we propose and evaluate cloud services for high resolution video streams in order to perform line detection using Canny edge detection followed by Hough transform. These algorithms are often used as preprocessing steps for various high level tasks including object, anomaly, and activity recognition. We implement and evaluate both Canny edge detector and Hough transform algorithms in Hadoop and Spark. Our experimental evaluation using Spark shows an excellent scalability and performance compared to Hadoop and standalone implementations for both Canny edge detection and Hough transform. We obtained a speedup of 10.8$$\times$$ and 9.3$$\times$$ for Canny edge detection and Hough transform respectively using Spark. These results demonstrate the effectiveness of parallel implementation of computer vision algorithms to achieve good scalability for real-world applications.

Nature Inspired Power Optimization in smartphones

Good battery backup has always been one of the most desirable features of smartphones since inception. Key subsystems of smartphones include powerful processors (application processors - CPU, communication processors - modem and graphics processors - GPU), large displays and high-speed networks. All these subsystems are very demanding on smartphone battery. Popular smartphone use cases, such as high-resolution video streaming, networked gaming, video calling etc., are power hungry. Due to increased security concerns, smartphone manufacturers have added biometric security features such as fingerprint, IRIS and face recognition etc. in smartphones. There is additional current consumption ranging from 300 mA (mA) to 400 mA (mA) each time smartphone user unlocks the device using any one of these security features. Therefore, it will be in the fitness of the things that smartphone battery lifetime is increased as much as possible. Due to irregular nature of mobile network traffic, it is very difficult to accurately predict battery lifetime or set a deadline up to which battery should last. In this paper, we have addressed both these problems. Here, we present a novel power optimization system called NIPO ( N ature I nspired P ower O ptimization). Our contributions are – i) Present a novel client server based power optimization System, called NIPO. ii) Use nature inspired algorithms - particle swarm optimization (PSO), ant colony optimization (ACO), and cuckoo search algorithm (CSA) with Levy Flights, for power optimization in smartphone displays for the first time. iii) Develop a mathematical model of power consumption for OLED (Organic Light Emitting Diode) displays in smartphones. iv) Implement NIPO system on Android based smartphone and Windows server to validate the effectiveness of proposed system. For experimental purposes, the proposed NIPO system was implemented on smartphone running Android OS for client side and a Windows server for server side. The experimental results validate the effectiveness of the proposed NIPO system. NIPO system successfully meets user defined time-to-last requirement of battery lifetime.

Efficient high-resolution video delivery over VANETs

The adoption of video-equipped vehicles in Vehicular ad-hoc networks (VANETs) is experiencing a rapid growth. It is also anticipated a substantial increase in the video content distribution with the arrival of self-driving cars as both passengers and vehicles will be able to produce and consume this type of media. This unveils a set of challenges, especially in VANETs where the network resources tend to be scarce and the connections suffer from time-varying error conditions. Taking everything into consideration, a Quality of Experience (QoE)-driven mechanism is desirable to enhance the video delivery over error-prone networks. To this end, the combined use of forward error correction and unequal error protection has proven its efficiency in delivering high-quality videos with low network overhead. The proposed intelligent quality-driven and network-aware mechanism (AntArmour) uses an ant colony optimization scheme to dynamically allocate a precise amount of redundancy. This allows AntArmour to safeguard, in real-time, the live transmission of high-resolution video streams. This operation is performed according to specific high efficiency video coding details and the actual network conditions such as the signal-to-noise ratio, the network density, the vehicle’s position, and the current packet loss rate (PLR) as well as the prediction of future PLR. The experiments were performed using real map’s clippings and actual video footage. The assessment was performed with the aid of two well-known objective QoE metrics, as well as the measure of the network overhead. The results showed that the proposed mechanism outperformed all its competitors in both video quality improvement and network overhead decrement.

A Graph-Based QoS-Aware Resource Management Scheme for OFDMA Femtocell Networks

Recently, femtocells are expected to be densely deployed to meet the booming demands for wireless data traffic, which inevitably causes serious co-layer interference due to the unplanned deployment. Hence, how to efficiently combat interference while furthest guaranteeing the quality of service (QoS) becomes an open question. To address this issue, in this paper, we present a joint admission control and a resource allocation strategy for an orthogonal frequency-division multiple access-based femtocell network. Particularly, we consider the scenario, where users are classified into two types, of which high priority ones have priority to access the network and enjoy high-resolution video streams. To perfectly eliminate the co-layer interference with limited spectrum while maximizing the number of users whose QoS requirements can be guaranteed, we formulate an integer nonlinear programming problem, followed by a graph-based algorithm with polynomial computational complexity to solve it. The basic principle behind the proposed algorithm is to first chordalize the given conflict graph, and then perform admission control with priority differentiation-based admission control sub-algorithm, followed by the final resource block assignment with maximum effect rank allocation sub-algorithm. Furthermore, we illustrate how our proposal could be implemented in the long-term evolution system. Finally, through extensive simulations, we show that our proposal outperforms other two schemes in terms of guaranteeing QoS requirements.

GPU-Based Realtime System for Cinematic Virtual Reality Production

Relatively inexpensive head mounted display (HMD) devices made possible by advances in digital imaging technology are moving the immersive virtual and augmented reality mediums into the entertainment mainstream. This departure from the traditional art of linear storytelling requires a richer production workflow and toolset capable of seamlessly integrating numerous potential narratives. An effective virtual reality (VR) workflow requires low-latency realtime synchronized capture of multiple live high-resolution video streams, realtime stitching and viewing of the captured assets, and realtime composition with computer generated elements optimally rendered for the HMD. This paper presents a realtime system for VR production that encompasses optimal high-bandwidth data transfers, graphics processor unit-based video stitching, and techniques for optimally rendering to the individual display characteristics of specific VR HMDs for on-set preview.

Joint Transmission and Coding Scheme for High-Resolution Video Streams over Multiuser MIMO-OFDM Systems

Joint Transmission and Coding Scheme for High-Resolution Video Streams over Multiuser MIMO-OFDM Systems

Hands-on Analysis of 802.11ac Modulation and Coding Scheme

Current communication networks are full of mobile devices that are capable of performing data transfers at high data rates or access high resolution video streams. For such requirements, in the data communications there is a permanent concern of wireless communications development to keep up with the wired communication networks. Because the 802.11n technology is not fast enough for increasingly more users willing games and online broadcasts, the 802.11ac technology was developed. This new IEEE standard, along with the future technology 802.11ad, is aiming to achieve a new level of performance, called VHT (Very High Throughput). The goal is to reach transfer rates (over 1Gbps for now) comparable to those establish in wired networks. This article is proposing a study over 802.11ac technology by exploring the performances of the specific MCS (Modulation and Coding Scheme) with a handy digitally modulated signals investigation method, CCDF (Complementary Cumulative Distribution Function).

New interaction modes for rich panoramic live video experiences

The possibilities of panoramic video are based on the capabilities of high-resolution digital video streams and higher bandwidth's opportunities to broadcast, stream and transfer large content across platforms. With these opportunities also come challenges such as how to focus on sub-parts of the video stream and interact with the content shown on a large screen. In this paper, we present studies of two different interaction modes with a large-scale panoramic video for live experiences; we focus on interactional challenges and explore if it is (1) possible to develop new interactional methods/ways of approaching this type of high-resolution content and (2) feasible for users to interact with the content in these new ways. We developed prototypes for two different interaction modes: an individual system on a mobile device, either a tablet or a mobile phone, for interacting with the content on the same and a non-touch gesture-based system for the home or small group interaction. We present pilot studies where we explore the possibilities and challenges with these two interaction modes for panoramic content.