Video Encoder Research Articles

A Tutorial on Immersive Video Delivery: From Omnidirectional Video to Holography

Video services are evolving from traditional two-dimensional video to virtual reality and holograms, which offer six degrees of freedom to users, enabling them to freely move around in a scene and change focus as desired. However, this increase in freedom translates into stringent requirements in terms of ultra-high bandwidth (in the order of Gigabits per second) and minimal latency (in the order of milliseconds). To realize such immersive services, the network transport, as well as the video representation and encoding, have to be fundamentally enhanced. The purpose of this tutorial article is to provide an elaborate introduction to the creation, streaming, and evaluation of immersive video. Moreover, it aims to provide lessons learned and to point at promising research paths to enable truly interactive immersive video applications toward holography.

A Novel Hybrid Multikey Cryptography Technique for Video Communication

Online video streaming is becoming more widespread in people’s everyday entertainment routines. Protecting copyright and piracy has become a key concern in real-time video streaming systems. This research provides a revolutionary multi-key and hybrid cryptography approach to offer security. This work describes the software implementation of video encryption and decryption employing continuous systems based on the Elliptic Curve Cryptography approach as pseudorandom encryption key generators. This approach creates several keys to encrypt and decode small chunks of video files that are produced dynamically based on the video data. The suggested approach was implemented on the Android platform, where applications for sender and recipients had been created to enable streaming. The security and performance of the proposed system have been examined by implementing it on devices and streaming videos. The outcomes demonstrate superiority in terms of performance and security.

A Robust Selective Encryption Scheme for H.265/HEVC Video

To protect the information of video stream, many selective video encryption schemes have been proposed based on the H.265/HEVC video. However, most of the existing algorithms are not robust, thus failing to decrypt under packet loss. To further improve the robustness capability of video protection, a robust selective encryption scheme is proposed in this paper. In H.265/HEVC standard, video is encoded into multiple slices, and the slices are decoded independently. Inspired by the feature, each slice is individually encrypted using RC4 stream cipher. The pseudorandom binary sequence (PRBS) for one slice is related to encoding parameters and the SHA-256 hash value of the corresponding slice header, thus ensuring the real-time update of the PRBS and increasing the resistance to chosen-plaintext attack. Two-rounds shifting algorithm is designed to scramble non-zero coefficients of the transform units (TUs) and then motion vector difference (MVD) parameters, quantized transform coefficients (QTCs) and intra prediction modes (IPMs) are selected for encryption. The combination of coefficient scrambling and syntax elements encryption further improves the encryption performance. The proposed scheme is slice-synchronized. Hence, it decrypts normally in the case of packet loss, and supports online real-time interaction. Furthermore, the simulation and analysis results show that the proposed algorithm has format compatibility, high security and low time complexity, which is a promising tool for video cryptographic applications.

A multi-level approach with visual information for encrypted H.265/HEVC videos

High-efficiency video coding (HEVC) encryption has been proposed to encrypt syntax elements for the purpose of video encryption. To achieve high video security, to the best of our knowledge, almost all of the existing HEVC encryption algorithms mainly encrypt the whole video, such that the user without permissions cannot obtain any viewable information. However, these encryption algorithms cannot meet the needs of customers who need part of the information but not the full information in the video. In many cases, such as professional paid videos or video meetings, users would like to observe some visible information in the encrypted video of the original video to satisfy their requirements in daily life. Aiming at this demand, this paper proposes a multi-level encryption scheme that is composed of lightweight encryption, medium encryption and heavyweight encryption, where each encryption level can obtain a different amount of visual information. First, we employ AES–CTR to generate a pseudo-random number sequence. Then, the main syntax elements in the H.265/HEVC encoding process are encrypted by a pseudorandom sequence. In the lightweight encryption level, the syntax element of the luma intraprediction model is chosen for encryption. In the medium encryption level, the syntax element of the discrete cosine transform (DCT) coefficient sign is employed for scrambling encryption. In the heavyweight encryption level, syntax elements of both the luma intraprediction model and the DCT coefficient sign are encrypted simultaneously by the pseudorandom sequence. It is found that both encrypting the luma intraprediction model (IPM) and scrambling the syntax element of the DCT coefficient sign can achieve the performance of a distorted video in which there is still residual visual information, while encrypting both of them can implement the intensity of encryption and one cannot gain any visual information. The experimental results meet our expectations appropriately, indicating that there is a different amount of visual information in each encryption level. Meanwhile, users can flexibly choose the encryption level according to their various requirements.

E 2 -VOR: An End-to-End En/Decoder Architecture for Efficient Video Object Recognition

High-resolution video object recognition (VOR) evolves so fast but is very compute-intensive. This is because VOR leverages compute-intensive deep neural network (DNN) for better accuracy. Although many works have been proposed for speedup, they mostly focus on DNN algorithm and hardware acceleration on the edge side. We observe that most video streams need to be losslessly compressed before going online and an encoder should have all the video information. Moreover, as the cloud should have abundant computing power to handle sophisticated VOR algorithms, we propose to take a one-shot effort for a modified VOR algorithm at the encoding stage in cloud and integrate the full VOR regeneration into a slightly extended decoder on the device. The scheme can enable lightweight VOR with server-class accuracy by simply leveraging the classic and economic video decoder universal to any mobile device. Meanwhile, the scheme can save massive computing power for not repetitively processing the same video on different user devices that makes it extremely sustainable for green computing across the whole network. We propose E 2 -VOR, an end-to-end encoder and decoder architecture for efficient VOR. We carefully design the scheme to have minimum impact on the video bitstream transmitted. In the cloud, the VOR extended video encoder tracks on a macro-block basis and packs intelligent information into the video stream for increased VOR accuracy and fast regenerating process. On the edge device, we extend the traditional video decoder with a small piece of dedicated hardware to enable the efficient VOR regeneration. Our experiment shows that E 2 -VOR can achieve 5.0× performance improvement with less than 0.4% VOR accuracy loss compared to the state-of-the-art FAVOS scheme. On average, E 2 -VOR can run over 54 frames-per-second (FPS) for 480P videos on an edge device.

E-Ensemble: A Novel Ensemble Classifier for Encrypted Video Identification

In recent years, video identification within encrypted network traffic has gained popularity for many reasons. For example, a government may want to track what content is being watched by its citizens, or businesses may want to block certain content for productivity. Many such reasons advocate for the need to track users on the internet. However, with the introduction of the secure socket layer (SSL) and transport layer security (TLS), it has become difficult to analyze traffic. In addition, dynamic adaptive streaming over HTTP (DASH), which creates abnormalities due to the variable-bitrate (VBR) encoding, makes it difficult for researchers to identify videos in internet traffic. The default quality settings in browsers automatically adjust the quality of streaming videos depending on the network load. These auto-quality settings also increase the challenge in video detection. This paper presents a novel ensemble classifier, E-Ensemble, which overcomes the abnormalities in video identification in encrypted network traffic. To achieve this, three different classifiers are combined by using two different combinations of classifiers: the hard-level and soft-level combinations. To verify the performance of the proposed classifier, the classifiers were trained on a video dataset collected over one month and tested on a separate video dataset captured over 20 days at a different date and time. The soft-level combination of classifiers showed more stable results in handling abnormalities in the dataset than those of the hard-level combination. Furthermore, the soft-level classifier combination technique outperformed the hard-level combination with a high accuracy of 81.81%, even in the auto-quality mode.

Aligning Source Visual and Target Language Domains for Unpaired Video Captioning.

Training supervised video captioning model requires coupled video-caption pairs. However, for many targeted languages, sufficient paired data are not available. To this end, we introduce the unpaired video captioning task aiming to train models without coupled video-caption pairs in target language. To solve the task, a natural choice is to employ a two-step pipeline system: first utilizing video-to-pivot captioning model to generate captions in pivot language and then utilizing pivot-to-target translation model to translate the pivot captions to the target language. However, in such a pipeline system, 1) visual information cannot reach the translation model, generating visual irrelevant target captions; 2) the errors in the generated pivot captions will be propagated to the translation model, resulting in disfluent target captions. To address these problems, we propose the Unpaired Video Captioning with Visual Injection system (UVC-VI). UVC-VI first introduces the Visual Injection Module (VIM), which aligns source visual and target language domains to inject the source visual information into the target language domain. Meanwhile, VIM directly connects the encoder of the video-to-pivot model and the decoder of the pivot-to-target model, allowing end-to-end inference by completely skipping the generation of pivot captions. To enhance the cross-modality injection of the VIM, UVC-VI further introduces a pluggable video encoder, i.e., Multimodal Collaborative Encoder (MCE). The experiments show that UVC-VI outperforms pipeline systems and exceeds several supervised systems. Furthermore, equipping existing supervised systems with our MCE can achieve 4% and 7% relative margins on the CIDEr scores to current state-of-the-art models on the benchmark MSVD and MSR-VTT datasets, respectively.

Audience and complexity aware live video encoders orchestration

Audience and complexity aware live video encoders orchestration

Detection of Double-Compressed Videos Using Descriptors of Video Encoders

In digital forensics, video becomes important evidence in an accident or a crime. However, video editing programs are easily available in the market, and even non-experts can delete or modify a section of an evidence video that contains adverse evidence. The tampered video is compressed again and stored. Therefore, detecting a double-compressed video is one of the important methods in the field of digital video tampering detection. In this paper, we present a new approach to detecting a double-compressed video using the proposed descriptors of video encoders. The implementation of real-time video encoders is so complex that manufacturers should develop hardware video encoders considering a trade-off between complexity and performance. According to our observation, hardware video encoders practically do not use all possible encoding modes defined in the video coding standard but only a subset of the encoding modes. The proposed method defines this subset of encoding modes as the descriptor of the video encoder. If a video is double-compressed, the descriptor of the double-compressed video is changed to the descriptor of the video encoder used for double-compression. Therefore, the proposed method detects the double-compressed video by checking whether the descriptor of the test video is changed or not. In our experiments, we show descriptors of various H.264 and High-Efficiency Video Coding (HEVC) video encoders and demonstrate that our proposed method successfully detects double-compressed videos in most cases.

Fast colored video encryption using block scrambling and multi-key generation

Multimedia information usage is increasing with new technologies such as the Internet of things (IoT), cloud computing, and big data processing. Video is one of the most widely used types of multimedia. Videos are played and transmitted over different networks in many IoT applications. Consequently, securing videos during transmission over various networks is necessary to prevent unauthorized access to the video's content. The existing securing schemes have limitations in terms of high resource consumption and high processing time, which are not liable to IoT devices with limited resources in terms of processor size, memory, time, and power consumption. This paper proposed a new encryption scheme for securing the colored videos. The video frames are extracted, and then, the frame components (red, green, and blue) are separated and padded by zero. Then, every frame component (channel) is split into blocks of different sizes. Then, the scrambled blocks of a component are obtained by applying a zigzag scan, rotating the blocks, and randomly changing the blocks' arrangements. Finally, a secret key produced from a chaotic logistic map is used to encrypt the scrambled frame component. Security analysis and time complexity are used to evaluate the efficiency of the proposed scheme in encrypting the colored videos. The results reveal that the proposed scheme has high-level security and encryption efficiency. Finally, a comparison between the proposed scheme and existing schemes is performed. The results confirmed that the proposed scheme has additional encryption efficiency.

An Efficient Video Encryption Algorithm Based on the Pseudorandom Number Generator of Zipf Distribution

In this study, a pioneer selective video encryption (PSVE) algorithm is proposed based on the pseudorandom number generator (PRNG) of the Zipf distribution (Z-PRNG). It is a general algorithm with high efficiency and security. The encryption process is completely separable from the video coding process. In the PSVE algorithm, Z-PRNG is designed based on the 3D SCL-HMC hyperchaotic map. Firstly, encapsulated byte sequence payloads (EBSPs) are extracted from the video bitstream. Secondly, random numbers of the Zipf distribution are generated by Z-PRNG, and they are used to randomly select encrypted data from each EBSP. Lastly, the extracted data are encrypted by AES-CTR to obtain the encrypted video. Compared with existing algorithms, the encryption position is more flexible, and the key space is further enhanced. The high efficiency video coding (HEVC) video and the advanced video coding (AVC) video are taken as examples to test the PSVE algorithm. The analysis results show that the proposed scheme can effectively resist common attacks, and its time complexity is much less than most existing algorithms.

Video streaming in ultra high definition (4K and 8K) on a portable device employing a Versatile Video Coding standard

The most commonly used H.264 codec standard video compression includes a wide range of devices and platforms. But this codec faces difficulties in capturing 4K and 8K resolution movies. Preserving, uploading, streaming, as well as distributing huge video files, is becoming more difficult. On the other hand, platforms such as Netflix, Vimeo, Hulu, YouTube, and others grow in popularity; they are seeking for techniques to provide viewers with a continuous and higher-quality video environment. To overcome these challenges and to provide effective user comfort, the advanced video compression technique namely HEVC is proposed in this paper. A convergent multi-screen encoder and its functional block of HEVC explain the process of multi-screen encoding of the input video. The parallel mapping process of HEVC enables autonomous functioning at multiple layers efficiently. The video compression of the HEVC standard is created to allow higher-resolution 8 K UHD videos to be displayed at a lower bandwidth. When it comes to compression technique, HEVC is the superior form to H.264 codec. It's cutting-edge, quick, and loaded with features. FVC is utilized for optimization with respect to encoding time and therefore, its supports video compression at a high rate. Thus, the proposed work is experimentally evaluated in terms of encoding time, PSNR, and Bitrate.

Video Encryption Technique Based on Hybrid Chaotic Maps and Multi- Operation keys

Video Encryption Technique Based on Hybrid Chaotic Maps and Multi- Operation keys

Design Space Exploration of Practical VVC Encoding for Emerging Media Applications

Versatile Video Coding (VVC/H.266) is the latest video coding standard designed for a broad range of next-generation media applications. This paper explores the design space of practical VVC encoding by profiling the Fraunhofer Versatile Video Encoder (VVenC). All experiments were conducted over five 2160p video sequences and their downsampled versions under the random access (RA) condition. The exploration was performed by analyzing the rate-distortion-complexity (RDC) of the VVC block structure and coding tools. First, VVenC was profiled to provide a breakdown of coding block distribution and coding tool utilization in it. Then, the usefulness of each VVC coding tool was analyzed for its individual impact on overall RDC performance. Finally, our findings were elevated to practical implementation guidelines: the highest coding gains come with the multi type tree (MTT) structure, adaptive loop filter (ALF), cross component linear model (CCLM), and bi-directional optical flow (BDOF) coding tools, whereas multi transform selection (MTS) and affine motion estimation are the primary candidates for complexity reduction. To the best of our knowledge, this is the first work to provide a comprehensive RDC analysis for practical VVC encoding. It can serve as a basis for practical VVC encoder implementation or optimization on various computing platforms.

A Novel Separable Scheme for Encryption and Reversible Data Hiding

With the increasing emphasis on security and privacy, video in the cloud sometimes needs to be stored and processed in an encrypted format. To facilitate the indexing and tampering detection of encrypted videos, data hiding is performed in encrypted videos. This paper proposes a novel separable scheme for encryption and reversible data hiding. In terms of encryption method, intra-prediction mode and motion vector difference are encrypted by XOR encryption, and quantized discrete cosine transform block is permutated based on logistic chaotic mapping. In terms of the reversible data hiding algorithm, difference expansion is applied in encrypted video for the first time in this paper. The encryption method and the data hiding algorithm are separable, and the embedded information can be accurately extracted in both encrypted video bitstream and decrypted video bitstream. The experimental results show that the proposed encryption method can resist sketch attack and has higher security than other schemes, keeping the bit rate unchanged. The embedding algorithm used in the proposed scheme can provide higher capacity in the video with lower quantization parameter and good visual quality of the labeled decrypted video, maintaining low bit rate variation. The video encryption and the reversible data hiding are separable and the scheme can be applied in more scenarios.

Semantic similarity information discrimination for video captioning

Video captioning is a task that aims to automatically describe objects and their actions in videos using natural language sentences. The correct understanding of vision and language information is critical for video captioning tasks. Many existing methods usually fuse different features to generate sentences. However, the sentences have many improper nouns and verbs. Inspired by the successes of fine-grained visual recognition, we treat the problem of improper words to discriminate semantic similarity information. In this paper, we designed a semantic bilinear block (SBB) to widen the gap between the probability of existing and nonexistent words, which can capture more fine-grained features to discriminate semantic information. Moreover, our designed linear attention block (LAB) implements the channelwise attention for the 1-D feature by simplifying the squeeze-and-excitation structure. Furthermore, we designed a semantic discrimination network (SDN) that integrates the LAB and SBB into video encoder and decoder to leverage successful channelwise attention and discriminate semantic similarity information for better video captioning. Experiments on two widely used datasets, MSVD and MSR-VTT, demonstrate that our proposed SDN can achieve better performance than state-of-the-art methods.

Novel lightweight video encryption method based on ChaCha20 stream cipher and hybrid chaotic map

<span lang="EN-US">In the recent years, an increasing demand for securing visual resource-constrained devices become a challenging problem due to the characteristics of these devices. Visual resource-constrained devices are suffered from limited storage space and lower power for computation such as wireless sensors, internet protocol (IP) camera and smart cards. Consequently, to support and preserve the video privacy in video surveillance system, lightweight security methods are required instead of the existing traditional encryption methods. In this paper, a new light weight stream cipher method is presented and investigated for video encryption based on hybrid chaotic map and ChaCha20 algorithm. Two chaotic maps are employed for keys generation process in order to achieve permutation and encryption tasks, respectively. The frames sequences are encrypted-decrypted based on symmetric scheme with assist of ChaCha20 algorithm. The proposed lightweight stream cipher method has been tested on several video samples to confirm suitability and validation in term of encryption–decryption procedures. The performance evaluation metrics include visual test, histogram analysis, information entropy, correlation analysis and differential analysis. From the experimental results, the proposed lightweight encryption method exhibited a higher security with lower computation time compared with state-of-the-art encryption methods.</span>

Joint video compression and encryption using parallel compressive sensing and improved chaotic maps

In this paper, we propose a method for joint compression and encryption of video using parallel compressive sensing (PCS) and improved chaotic maps. The method is shown to be computationally efficient without compromising the quality of the recovered video. The processing of the video frames are carried out at group of pictures (GOP) level which consists of both reference and non-reference frames. We apply DCT for sparsification of the reference frames and permute the DCT coefficients to improve the quality of the reconstructed video. We introduce a Logistic Tent Infinite Collapse Map (LT-ICM) which has a higher chaotic range and complex behavior, unpredictability, and sensitivity towards initial values and controlling parameters. The measurement matrix for PCS is generated by using one-dimensional LT-ICM. A novel substitution method is introduced using circular shift, xor, and modular addition operations. We finally shuffle the substituted frames using frame dependent two-dimensional LT-ICM chaotic sequence. Experimental analysis including comparisons with state-of-the-art approaches establishes the effectiveness of the proposed solution.

Design and exploration of low-power SAD architectures using approximate compressors for Integer Motion Estimation

Integer motion estimation performs a pivotal role in achieving video compression by exploiting the temporal redundancy of video sequences. However, the motion estimation’s computational complexity is huge and requires much computational power in real-time. On the other hand, portable mobile devices which operate on battery power have minimal computational capabilities. Since the video encoding can tolerate error up to a specific limit, the hardware complexity can be reduced by approximating the motion estimator’s fundamental modules. The major contributor to the power consumption in the motion estimator is the sum of the absolute difference (SAD) module. This paper presents SAD architectures implemented using new approximate 8:2 compressors to reduce the computational complexity. Thus, leading to a reduction in the hardware complexity and power consumption while maintaining the coding efficiency. Our proposed approximate SAD architecture achieves up to 56.7% and 49.8% savings in power compared to exact compressor-based SAD architectures and existing approximate SAD architectures respectively.

Testing Self-Adaptive Software With Probabilistic Guarantees on Performance Metrics: Extended and Comparative Results

This paper discusses methods to test the performance of the adaptation layer in a self-adaptive system. The problem is notoriously hard, due to the high degree of uncertainty and variability inherent in an adaptive software application. In particular, providing any type of formal guarantee for this problem is extremely difficult. In this paper we propose the use of a rigorous probabilistic approach to overcome the mentioned difficulties and provide probabilistic guarantees on the software performance. We describe the set up needed for the application of a probabilistic approach. We then discuss the traditional tools from statistics that could be applied to analyse the results, highlighting their limitations and motivating why they are unsuitable for the given problem. We propose the use of a novel tool – <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">the Scenario Theory</i> – to overcome said limitations. We conclude the paper with a thorough empirical evaluation of the proposed approach, using three adaptive software applications: the Tele-Assistance Service, the Self-Adaptive Video Encoder, and the Traffic Reconfiguration via Adaptive Participatory Planning. With the first, we empirically expose the trade-off between data collection and confidence in the testing campaign. With the second, we demonstrate how to compare different adaptation strategies. With the third, we discuss the role of the randomisation in the selection of test inputs. In the evaluation, we apply the scenario theory and also classical statistical tools: Monte Carlo and Extreme Value Theory. We provide a complete evaluation and a thorough comparison of the confidence and guarantees that can be given with all the approaches.