Compressed Video Streams Research Articles

End-to-End Traffic Flow Rate Estimation From MPEG4 part-2 Compressed Video Streams

End-to-End Traffic Flow Rate Estimation From MPEG4 part-2 Compressed Video Streams

ДО ПИТАННЯ ПОБУДОВИ АЛГОРИТМІВ ПЕРЕДАЧІ ПОТОКОВОГО ВІДЕО ДЛЯ МЕРЕЖ З НЕСТАБІЛЬНОЮ ПРОПУСКНОЮ ЗДАТНІСТЮ

Some questions of development of advanced video transmission algorithms that could robustly operate in the electronic warfare conditions are considered. The technology of adaptation of MJPEG compressed video stream for the networks with variable and unstable bandwidth is proposed. The new approach where connection control is separated from encoder basing on man-in-the-middle scenario is developed. In this case it became possible to preserve not only standard decoders on the client side, but the encoder code on the server side as well. The performance and reliability of the proposed MJPEG transmission algorithm was demonstrated in the field tests. The MJPEG transcoder based on developed algorithm demonstrated sufficiently stable video stream from the drone via WiFi network in the presence of impulse signal jamming.

THE STUDY OF THE VIDEO TRANSCODING EFFICIENCY DEPENDING ON THE NUMBER OF DECODERS AND ENCODERS WORKER THREADS

In present study the efficiency of transcoding H.264 compressed video stream depending on the number of decoders and encoders worker threads was investigated. Tests were done using FFmpeg open source libraries based on Intel and ARM platforms. Both CPUs used were quad core and could run up to 4 threads. The effect of the number of decoders and encoder worker threads was studied upon the same input test video. The obtained results indicate that the greater the number of encoder threads produces the greater the fps value. At the same time, using the default encoder threads number provides better performance than the explicitly set value of 4 encoder threads.

A video compression-cum-classification network for classification from compressed video streams

A video compression-cum-classification network for classification from compressed video streams

Local Compressed Video Stream Learning for Generic Event Boundary Detection

Local Compressed Video Stream Learning for Generic Event Boundary Detection

Efficient Online Lecture Platform: Design and Implementation of Optimized Temporal Masking Technique for Compressed Video Streaming

Efficient Online Lecture Platform: Design and Implementation of Optimized Temporal Masking Technique for Compressed Video Streaming

Object detection and classification from compressed video streams

AbstractVideo Analytics is widely used by the internet‐based platforms to govern the mass consumption of videos. Traditionally, it is carried out from the decoded format of the videos. This requires the analytics server to perform both decoding and analytics computation. This process can be made fast and efficient if performed over the compressed format of the videos as it reduces the decoding stress over the analytics server. The field of video analytics from the binarized formats using modern deep learning techniques is still emerging and needs further exploration. This proposed work is based on the same notion. In this work, two analytics tasks that is, classification and object detection are carried out from the binarized videos. The binarized formats are produced by using an already‐designed end‐to‐end video compression network. The experiments have been carried out over standard datasets. The proposed MobileNetv2‐based classification network shows an accuracy of 66% over the YouTube UGC dataset and the YOLOX‐S‐based detection network shows mAP of 45% over IMAGENet datasets. The proposed work shows competitiveness and improvement in the detection outcomes on compressed data and also provides further motivation for the adoption of deep learning‐based video compression in practical analytics domains.

Tasmanian Devil Sail Fish Optimization with Deep Learning Enabled Macro Block Selection for Video Steganography in Compressed Video Stream

Tasmanian Devil Sail Fish Optimization with Deep Learning Enabled Macro Block Selection for Video Steganography in Compressed Video Stream

Compression loss-based spatial-temporal attention module for compressed video quality enhancement

Recently, deep learning technology has achieved remarkable progress in compressed video quality enhancement. However, the existing methods fail to consider the fact that the regions with different compression losses contain varied effective information. To address this issue, this paper proposes a Compression Loss-based Spatial-Temporal Attention (CLSTA) module that predicts the compression loss-based attention of each pixel when restoring target frames. Based on this, the pixels with lower compression losses contribute more to restoring target frames. Meanwhile, the compression-domain information in the compressed video stream is exploited to overcome the difficulty in inferring compression loss-based attention directly from the pixel-domain. With a slight modification, the CLSTA module can be easily integrated into the existing methods for end-to-end training. A series of experiments have been conducted in this study to validate the effectiveness and generality of the CLSTA module for compressed video quality enhancement. By adding the CLSTA module to the state-of-the-art method STDF, the proposed method achieves an average PSNR improvement of 0.96 dB and a BD-rate reduction of 26.85% under the Low-Delay P configuration, which is respectively 0.06 dB and 1.83% better than the baseline STDF. In the best case, the proposed method obtains 0.19 dB higher PSNR than STDF for a single frame.

A Study of the Possibility of Usage Motion Vectors of Compressed Videos to Create Video Identification

The use of motion vectors for identifying video sequences has been well studied (in the framework of research on the topic CBCD – Content-Based Copy Detection ‒ detecting copies of videos based on content analysis). This makes it possible to check the similarity of two video fragments or search for a fragment in a larger video sequence. Existing and well-known methods for forming identification datasets typically use complete video stream decoding. The authors suggested using the motion vectors of a compressed video stream, which reduces the computational costs for identifying video sequences and uses simplified algorithms to generate identification data. Unlike the previously proposed methods, which implement either modified video codecs or obsolete ones, the authors propose using data formed by compression codecs that are used in the most common video hosting platforms (Youtube, Vimeo, etc.) The possibility of forming an automated system of comparing video sequences, along with its possibilities and limitations, will be studied in the following works.

Project starline

We present a real-time bidirectional communication system that lets two people, separated by distance, experience a face-to-face conversation as if they were copresent. It is the first telepresence system that is demonstrably better than 2D videoconferencing, as measured using participant ratings (e.g., presence, attentiveness, reaction-gauging, engagement), meeting recall, and observed nonverbal behaviors (e.g., head nods, eyebrow movements). This milestone is reached by maximizing audiovisual fidelity and the sense of copresence in all design elements, including physical layout, lighting, face tracking, multi-view capture, microphone array, multi-stream compression, loudspeaker output, and lenticular display. Our system achieves key 3D audiovisual cues (stereopsis, motion parallax, and spatialized audio) and enables the full range of communication cues (eye contact, hand gestures, and body language), yet does not require special glasses or body-worn microphones/headphones. The system consists of a head-tracked autostereoscopic display, high-resolution 3D capture and rendering subsystems, and network transmission using compressed color and depth video streams. Other contributions include a novel image-based geometry fusion algorithm, free-space dereverberation, and talker localization.

Digital Retina: A Way to Make the City Brain More Efficient by Visual Coding

The ubiquitous camera networks in the city brain system grow at a rapid pace, creating massive amounts of images and videos at a range of spatial-temporal scales and thereby forming the “biggest” big data. However, the sensing system often lags behind the construction of the fast-growing city brain system, in the sense that such exponentially growing data far exceed today’s sensing capabilities. Therefore, critical issues arise regarding how to better leverage the existing city brain system and significantly improve the city-scale performance in intelligent applications. To tackle the unprecedented challenges, we articulate a vision towards a novel visual computing framework, termed as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">digital retina</i> , which aligns high-efficiency sensing models with the emerging Visual Coding for Machine (VCM) paradigm. In particular, digital retina may consist of video coding, feature coding, model coding, as well as their joint optimization. The digital retina is biologically-inspired, rooted on the widely accepted view that the retina encodes the visual information for human perception, and extracts features by the brain downstream areas to disentangle the visual objects. Within the digital retina framework, three streams, i.e., video stream, feature stream, and model stream, work collaboratively over the end-edge-cloud platform. In particular, the compressed video stream serves for human vision, the compact feature stream targets for machine vision, and the model stream incrementally updates deep learning models to improve the performance of human/machine vision tasks. We have developed a prototype to demonstrate the technical advantages of digital retina, and extensive experiments have been conducted to validate that it is able to effectively support the video big data analysis and retrieval in the intelligent city system. In particular, up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$7000\times $ </tex-math></inline-formula> compression ratio could be realized for visual data compression while maintaining competitive performance with pristine signal in a series of visual analysis tasks.

Deceptive Techniques to Hide a Compressed Video Stream for Information Security.

With the recent development of video compression methods, video transmission on traditional devices and video distribution using networks has increased in various devices such as drones, IP cameras, and small IoT devices. As a result, the demand for encryption techniques such as MPEG-DASH for transmitting streams over networks is increasing. These video stream security methods guarantee stream confidentiality. However, they do not hide the fact that the encrypted stream is being transmitted over the network. Considering that sniffing attacks can analyze the entropy of the stream and scan huge amounts of traffic on the network, to solve this problem, the deception method is required, which appears unencrypted but a confidential stream. In this paper, we propose the new deception method that utilizes standard NAL unit rules of video codec, where the unpromised device shows the cover video and the promised device shows the secret video for deceptive security. This method allows a low encryption cost and the stream to dodge entropy-based sniffing scan attacks. The proposed stream shows that successful decoding using five standard decoders and processing performance was 61% faster than the conventional encryption method in the test signal conformance set. In addition, a network encrypted stream scan method the HEDGE showed classification results that our stream is similar to a compressed video.

On the Performance of Video Resolution, Motion and Dynamism in Transmission Using Near-Capacity Transceiver for Wireless Communication.

This article investigates the performance of various sophisticated channel coding and transmission schemes for achieving reliable transmission of a highly compressed video stream. Novel error protection schemes including Non-Convergent Coding (NCC) scheme, Non-Convergent Coding assisted with Differential Space Time Spreading (DSTS) and Sphere Packing (SP) modulation (NCDSTS-SP) scheme and Convergent Coding assisted with DSTS and SP modulation (CDSTS-SP) are analyzed using Bit Error Ratio (BER) and Peak Signal to Noise Ratio (PSNR) performance metrics. Furthermore, error reduction is achieved using sophisticated transceiver comprising SP modulation technique assisted by Differential Space Time Spreading. The performance of the iterative Soft Bit Source Decoding (SBSD) in combination with channel codes is analyzed using various error protection setups by allocating consistent overall bit-rate budget. Additionally, the iterative behavior of SBSD assisted RSC decoder is analyzed with the aid of Extrinsic Information Transfer (EXIT) Chart in order to analyze the achievable turbo cliff of the iterative decoding process. The subjective and objective video quality performance of the proposed error protection schemes is analyzed while employing H.264 advanced video coding and H.265 high efficient video coding standards, while utilizing diverse video sequences having different resolution, motion and dynamism. It was observed that in the presence of noisy channel the low resolution videos outperforms its high resolution counterparts. Furthermore, it was observed that the performance of video sequence with low motion contents and dynamism outperforms relative to video sequence with high motion contents and dynamism. More specifically, it is observed that while utilizing H.265 video coding standard, the Non-Convergent Coding assisted with DSTS and SP modulation scheme with enhanced transmission mechanism results in gain of 20 dB with reference to the Non-Convergent Coding and transmission mechanism at the objective PSNR value of 42 dB. It is important to mention that both the schemes have employed identical code rate. Furthermore, the Convergent Coding assisted with DSTS and SP modulation mechanism achieved superior performance with reference to the equivalent rate Non-Convergent Coding assisted with DSTS and SP modulation counterpart mechanism, with a performance gain of 16 dB at the objective PSNR grade of 42 dB. Moreover, it is observed that the maximum achievable PSNR gain through H.265 video coding standard is 45 dB, with a PSNR gain of 3 dB with reference to the identical code rate H.264 coding scheme.

Deep learning-based video stream reconstruction in mass-production diffractive optical systems

Many recent studies have focused on developing image reconstruction algorithms in optical systems based on flat optics. These studies demonstrate the feasibility of applying a combination of flat optics and the reconstruction algorithms in real vision systems. However, additional causes of quality loss have been encountered in the development of such systems. This study investigates the influence on the reconstructed image quality of such factors as limitations of mass production technology for diffractive optics, lossy video stream compression artifacts, and specificities of a neural network approach to image reconstruction. The paper offers an end-to-end deep learning-based image reconstruction framework to compensate for the additional factors of quality losing. It provides the image reconstruction quality sufficient for applied vision systems.

Preserving Low-Quality Video through Deep Learning

Lossy video stream compression is performed to reduce the bandwidth and storage requirements. Moreover also image compression is a need that arises in many circumstances.It is often the case that older archive are stored at low resolution and with a compression rate suitable for the technology available at the time the video was created. Unfortunately, lossy compression algorithms cause artifact. Such artifacts, usually damage higher frequency details also adding noise or novel image patterns. There are several issues with this phenomenon. Low-quality images can be less pleasant to persons. Object detectors algorithms may have their performance reduced. As a result, given a perturbed version of it, we aim at removing such artifacts to recover the original image. To obtain that, one should reverse the compression process through a complicated non-linear image transformation. We propose a deep neural network able to improve image quality. We show that this model can be optimized either traditionally, directly optimizing an image similarity loss (SSIM), or using a generative adversarial approach (GAN). Our restored images have more photorealistic details with respect to traditional image enhancement networks. Our training procedure based on sub-patches is novel. Moreover, we propose novel testing protocol to evaluate restored images quantitatively. Differently from previously proposed approaches we are able to remove artifacts generated at any quality by inferring the image quality directly from data. Human evaluation and quantitative experiments in object detection show that our GAN generates images with finer consistent details and these details make a difference both for machines and humans.

Creating Bandwidth-Efficient Workflows with JPEG XS and SMPTE ST 2110

<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Internet protocol (IP) workflows have arrived and are here to stay. Next to reduced complexity and increased agility, one of the greatest advantages of moving to IP is cost reduction since it permits simple usage of commercially available off-the-shelf IP switches and cables. However, with new imaging technologies such as ultrahigh-definition (UHD)-1/4K, high frame rates, and high dynamic range, we have more pixels to manage, store, and transport. SMPTE ST 2110 supports both uncompressed and compressed videos. With uncompressed video essences, this requires 10 GbE workflows for HD, 25 GbE for UHD-1/4K, and even more for future UHD-2/8K workflows. Currently, many are struggling just to upgrade infrastructures from 1080i or 720p to 1080p60 at 3 Gb/s; handling UHD-1/4K video at 60 frames/sec, 4:2:2, and 10 bits is much more challenging. And with UHD-2/8K on the verge, a future-proof solution must be found. This paper explains how and why the new International Organization for Standardization (ISO) Joint Photographic Experts Group (JPEG) XS in combination with part 22 of SMPTE ST 2110 for compressed video streams addresses and solves the problems of more pixels over limited bandwidth, latency, and image quality</i> .

Cloud Gaming with Foveated Video Encoding

Cloud gaming enables playing high-end games, originally designed for PC or game console setups, on low-end devices such as netbooks and smartphones, by offloading graphics rendering to GPU-powered cloud servers. However, transmitting the high-resolution video requires a large amount of network bandwidth, even though it is a compressed video stream. Foveated video encoding (FVE) reduces the bandwidth requirement by taking advantage of the non-uniform acuity of human visual system and by knowing where the user is looking. Based on a consumer-grade real-time eye tracker and an open source cloud gaming platform, we provide a cloud gaming FVE prototype that is game-agnostic and requires no modifications to the underlying game engine. In this article, we describe the prototype and its evaluation through measurements with representative games from different genres to understand the effect of parametrization of the FVE scheme on bandwidth requirements and to understand its feasibility from the latency perspective. We also present results from a user study on first-person shooter games. The results suggest that it is possible to find a “sweet spot” for the encoding parameters so the users hardly notice the presence of foveated encoding but at the same time the scheme yields most of the achievable bandwidth savings.

Usage of characteristic points to reduce the time of performance of SAD metric calculation during video flow compression by motion compensation method

Today, it is difficult to imagine a video system that does not use compression of the input video stream for transmission or storage, because the uncompressed size of video files can reach several hundred gigabytes. There are many different video stream compression techniques used in various codecs, however, one of the most commonly used is the motion compensation technique, which allows you to transmit video stream frames in the form of compensated inter frame difference, by finding the motion vectors of individual blocks image. Despite the fact that the above-mentioned technique has been used for the last twenty years, it still has two problem-priority places - assessments of the similarity of image blocks and the algorithm for searching blocks. This article focuses on the problem of estimating the similarity of the image, in order to reduce the time of estimating the similarity of the blocks, which in modern algorithms takes from 40 to 80% of the time of the entire process of encoding the video stream. The article considers the usage of metrics for estimating the similarity of images and their importance for the process of motion compensation during video stream compression, such as SSD (standard deviation) and SAD (sum of absolute differences). The temporal contribution of metrics to the video stream encoding process, in particular to the motion compensation process, is estimated. An algorithm combining classical SAD and parameter estimation based on characteristic points is proposed, which will reduce the metric calculation time for estimating block similarity. The reduction of SAD calculation time due to reduction of comparison points is estimated for three proposed comparison templates - HSAD (half SAD), TSAD (third SAD), DSAD (diagonal SAD). For the selected templates, the results of processing the test video sequence were compared with the results obtained during the processing of the video sequence by the classic SAD. The main attention was paid to the assessment of the following parameters: the relative difference SAD, the increase in the number of candidate blocks, the overlap of candidate blocks.

Intra Coding Strategy for Video Error Resiliency: Behavioral Analysis

One challenge in video transmission is to deal with packet loss. Since the compressed video streams are sensitive to data loss, the error resiliency of the encoded video becomes important. When video data is lost and retransmission is not possible, the missed data should be concealed. But loss concealment causes distortion in the lossy frame which also propagates into the next frames even if their data are received correctly. One promising solution to mitigate this error propagation is intra coding. There are three approaches for intra coding: intra coding of a number of blocks selected randomly or regularly, intra coding of some specific blocks selected by an appropriate cost function, or intra coding of a whole frame. But Intra coding reduces the compression ratio; therefore, there exists a trade-off between bitrate and error resiliency achieved by intra coding. In this paper, we study and show the best strategy for getting the best rate-distortion performance. Considering the error propagation, an objective function is formulated, and with some approximations, this objective function is simplified and solved. The solution demonstrates that periodical I-frame coding is preferred over coding only a number of blocks as intra mode in P-frames. Through examination of various test sequences, it is shown that the best intra frame period depends on the coding bitrate as well as the packet loss rate. We then propose a scheme to estimate this period from curve fitting of the experimental results, and show that our proposed scheme outperforms other methods of intra coding especially for higher loss rates and coding bitrates.