Ultra High Definition Research Articles

Analysis of coding unit partitioning and complexity reduction at intra-prediction mode of HEVC

ABSTRACT The emerging trend of ultra-high definition (4 K, UHD) video services necessitates coding efficiency, which is much higher than H.264. High-Efficiency Video Coding (HEVC) has been designed with the intention of drastically reducing the bit rate without compromising the video quality. The key feature is the introduction of a larger block structure with flexible block sizes from 64 × 64 to 8 × 8 samples. The enhanced feature has imposed higher complexity, specifically at the intra-prediction mode level of HEVC. The proposed research work has presented a novel hybrid approach to reducing the number of candidate Coding Units (CU) passing through irrelevant partition patterns and time-consuming iterative Rate-Distortion (RD) cost estimation process. The hybrid approach is implemented in two steps. In the first step gradient evaluation is performed and the CU is categorized as homogeneous or non-homogeneous. In the second step Early Termination is adopted by using machine learning for partitioning non-homogeneous CU. As the number of candidate Coding Units undergoing the time-consuming RD process is reduced by using the two-step hybrid approach, the proposed approach proves to be the best of its kind. The reduction in the number of candidate units undergoing the RD process reduces the complexity of the intra-prediction process. Encoding time is the measure of complexity and the proposed hybrid approach reduces the encoding time by an average of 36% compared with the HEVC reference model and also by an average of 26% as compared with the previous work. The Peak Signal to Noise Ratio (PSNR) reduced by an average of 0.1% indicates a slight reduction in the encoded image quality. The proposed approach significantly reduced the encoding time without affecting the RD performance of the encoder. The image quality is not compromised substantially when considering the improvement in the encoding time and the reduction in the complexity at an intra-prediction level.

Optimizing UHD Production with Super Resolution Techniques

Super resolution (SR) is a technique that is getting increased attention. It was first deployed in high-end TVs to upconvert content, and now content providers are looking at how it can be used on the production side to guarantee the same high quality on all devices. Different techniques can be used to apply SR to standard definition (SD) and high definition (HD) content for transmitting ultra high definition (UHD) resolution. During a recent trial with France TV at the French Open, SD content with a 16:9 aspect ratio was processed offline to create UHD content, using a hybrid pipeline of deep restoration and SR methods. A pretrained deep neural network was used to produce an intermediate upscaled and deblurred HD version, which was then passed on to a faster, less memory-intensive, and machine-learning (ML)-based SR filter to produce the final UHD 4K resolution output. The result is a significantly better subjective viewing experience in terms of details and sharpness compared with classical filters used in live systems. Adequately addressing analog tape recordings, which have been digitized, is a significantly bigger technical challenge, which will need to be resolved in the future. While a one-model-fits-all approach for video enhancement sounds attractive due to its simplicity, the best visual quality is produced by applying purpose-built models trained for addressing the general types of degradation found in the source video.

Video streaming in ultra high definition (4K and 8K) on a portable device employing a Versatile Video Coding standard

The most commonly used H.264 codec standard video compression includes a wide range of devices and platforms. But this codec faces difficulties in capturing 4K and 8K resolution movies. Preserving, uploading, streaming, as well as distributing huge video files, is becoming more difficult. On the other hand, platforms such as Netflix, Vimeo, Hulu, YouTube, and others grow in popularity; they are seeking for techniques to provide viewers with a continuous and higher-quality video environment. To overcome these challenges and to provide effective user comfort, the advanced video compression technique namely HEVC is proposed in this paper. A convergent multi-screen encoder and its functional block of HEVC explain the process of multi-screen encoding of the input video. The parallel mapping process of HEVC enables autonomous functioning at multiple layers efficiently. The video compression of the HEVC standard is created to allow higher-resolution 8 K UHD videos to be displayed at a lower bandwidth. When it comes to compression technique, HEVC is the superior form to H.264 codec. It's cutting-edge, quick, and loaded with features. FVC is utilized for optimization with respect to encoding time and therefore, its supports video compression at a high rate. Thus, the proposed work is experimentally evaluated in terms of encoding time, PSNR, and Bitrate.

Memory-Efficient Deformable Convolution Based Joint Denoising and Demosaicing for UHD Images

This paper introduces deformable convolution in deep learning based joint denoising and demosaicing (JDD), which yields more adaptable representation and larger receptive fields in features extraction for a superior restoration performance. However, the deformable convolution generally leads to considerable computational load and irregular memory access bottleneck, limiting its extensive deployment on edge devices. To address this issue, we develop grouping strategy and assign independent offsets to each kernel group to reduce the computation latency while keeping the accuracy. Motivated by the exploration for aggregate distribution characteristics of deformable offsets, we present the offset sharing methodology to simplify the memory access complexity of deformable convolution. As for hardware acceleration, we specially design a novel deformable matrix multiplication workflow incorporated with a deformable memory mapping unit to boost the computational throughput. The verification experiments on FPGA demonstrate that the proposed deformable convolution based JDD can restore 4K Ultra High Definition (UHD) images at 70FPS and yields significant promotion in visual effect and objective quality assessment.

Feeling of Presence Maximization: mmWave-Enabled Virtual Reality Meets Deep Reinforcement Learning

This paper investigates the problem of providing ultra-reliable and power-efficient virtual reality (VR) experiences for wireless mobile users. To ensure reliable ultra-high-definition (UHD) video frame delivery to mobile users and enhance their immersive visual experiences, a coordinated multipoint (CoMP) transmission technique and millimeter wave (mmWave) communications are exploited. Owing to user movement and time-varying wireless channels, the wireless VR experience enhancement problem is formulated as a sequence-dependent and mixed-integer problem with a goal of maximizing users’ feeling of presence (FoP) in the virtual world, subject to power consumption constraints on access points (APs) and users’ head-mounted displays (HMDs). The problem, however, is hard to be directly solved due to the lack of users’ accurate tracking information and the sequence-dependent and mixed-integer characteristics. To overcome this challenge, we develop a parallel echo state network (ESN) learning method to predict users’ tracking information by training fresh and historical tracking samples separately collected by APs. With the learnt results, we propose a deep reinforcement learning (DRL) based optimization algorithm to solve the formulated problem. In this algorithm, we implement deep neural networks (DNNs) as a scalable solution to produce integer decision variables and solve a continuous power control problem to criticize the integer decision variables. Finally, the performance of the proposed algorithm is compared with various benchmark algorithms, and the impact of different design parameters is also discussed. Simulation results demonstrate that the proposed algorithm is more 4.14% power-efficient than the benchmark algorithms.

Deep Learning Based Frame Interpolation for Stop-motion Animation

This study focuses on deep-learning-based frame interpolation facilitating stop-motion animation production. Through deep learning of the movements of stop-motion animation, frame interpolation works systematically toward its own production. Based on an understanding of the exaggeration or deletion of motions, the proposed frame interpolation technique differs from other linear frame interpolation techniques. This project used the stop motion animation Teenie Scouts BigFive produced by Taktoon Enterprise, a subsidiary of Chung-Ang University Technology Holdings, which was created using 4K UHD high-resolution images at a frame rate of 15 fps.

Photostable aggregation-induced emission of iridium(III) complex realizing robust and high-resolution imaging of latent fingerprints

Preparation of AIEgens with excellent photostability is of vital importance to acquire the optimal performance of aggregation-induced emission (AIE) and remains challenging. In this paper, a handy structural design strategy based on iridium(III) complexes for the creation of photostable AIEgens is reported. The synergistic effect of phenyl and carbazole improves the rotational activity of substituents, then activates the AIE property of iridium(III) complexes with considerably improved photostability. In addition, prompt and high-resolution imaging of latent fingerprints on tinfoil, glass, plastic, and steel is presented by immersion in an aqueous solution of IrCzPh for 60 s. With the aid of photostable AIE, legible and intact fingerprint images demonstrate ultrahigh definition, which can display down to level 3 of fingerprint details and accurately authenticate individual characteristics. Meanwhile, outstanding anti-aging performance of fingerprint images is emerged with no contrast attenuation under 365 nm irradiation for 60 min or aged in natural light for 7 days. Noteworthily, ultrahigh-contrast dynamic imaging of local fingerprint details is achieved.

Ultra-High Definition AV1 FME Interpolation Architectures Exploring Approximate Computing

Modern video encoders like the AOMedia Video 1 (AV1) implement several complex tools to allow the required high level of compression efficiency. The Fractional Motion Estimation (FME) is one of these complex tools, and AV1 FME defines 42 different interpolation filters. To handle such complexity, hardware acceleration using approximate computing has become an interesting alternative to be explored. This paper presents three optimized approximate architectures for the AV1 FME interpolation filters. The architectures reach real time interpolation for UHD 4K videos at 30 frames per second in a low cost, low power, and memory-efficient design. The architectures were synthesized for a 40nm TSMC standard-cells technology reaching power gains up to 83%, when compared to a precise architecture, and up to 20% when compared to a previously published approximated solution. The area gains were also expressive: up to 83% and 40%, respectively. The architectures also allow a memory bandwidth reduction of up to 59.5%, in comparison with the state-of-the-art solutions. The approximations implied small coding efficiency degradation of 0.54% and 1.25% in BD-BR. The presented architectures have the best results found in the literature when considering the trade-off among hardware cost, power dissipation, processing rate, memory bandwidth, and coding efficiency.

Perceptual ultra-high definition video coding based on adaptive just noticeable distortion model

Ultra-high definition (UHD) video is gradually applied into many fields. The huge amount of data in UHD video limits the realization of these applications. Perception video coding can exploit the visual redundancy, and improve the compression performance. So far, the perpetual video coding based on just noticeable distortion (JND) model cannot fully exploit the perceptual redundancy of ultra-high definition (UHD) video/image in 10-bit depth. Thus, an adaptive JND model based on perceptual noise Bayesian network (PNBN) is firstly established for UHD video/image. Then, combined perceptual noise channel encoding/decoding with human visual characteristics, hidden side information of image is dynamically estimated by PNBN. Thirdly, the adaptive JND model is considered both luminance masking effect and texture complexity, which can adaptively exploit redundancy in different texture regions of image. Finally, an effective perceptual video coding (PVC) scheme is proposed for versatile video coding. A novel distortion compensation factor is appropriately designed for rate-distortion optimization. Experimental results show that the proposed PVC scheme achieves significant bitrate reduction with better subjective video quality than the state-of-the-art PVC schemes.

Recent Activities of the Ultra HD Forum

The Ultra HD Forum was established in 2015 to accelerate Ultra HD deployment by bringing together market leaders from every part of the industry. Broadcasters, service providers, consumer electronics, and technology vendors collaborate on solving real-world obstacles in deploying advanced media formats and publishing these solutions for broad adoption. The Forum’s Guidelines Working Group has published Guidelines that describe a uniform set of characteristics for “Ultra HD” content and consistent methods for creating and delivering Ultra HD content.

Development of Transmitter and Receiver With Set Partitioning 64APSK Coded Modulation Designed on Basis of Channel Capacity

In this paper, we present a prototype transmitter and receiver with set partitioning (SP)-64 amplitude and phase shift keying (APSK) coded modulation we developed for expanding satellite transmission capacity. SP-multi-level coded modulation is a transmission scheme that can improve noise immunity by properly combining a modulation scheme, bit allocation, and error-correction codes. We designed SP-64APSK coded modulation on the basis of channel capacity for expanding satellite transmission capacity. We confirmed that it has higher robustness to additive white Gaussian noise (AWGN) than DVB-S2X’s 64APSK modulation with Gray mapping through computer simulation. We also developed a prototype transmitter and receiver to implement our SP-64APSK coded modulation and evaluated its transmission performances in an AWGN channel and in a non-linear channel simulating 12-GHz-band satellite transponder characteristics. We describe the method of designing our SP-64APSK coded modulation through computer simulation, the modulation/demodulation process, and transmission performances of the prototype transmitter and receiver. We also indicate that 8K ultra-high definition television (UHDTV) with an encoded bit rate of 150 Mbps can be transmitted using the prototype within 34.5 MHz, which is the bandwidth of a single transponder for satellite broadcasting in Japan.

Evolution of AVS video coding standards: twenty years of innovation and development

Ever since the founding of the Audio and Video Coding Standard (AVS) Workgroup of China in 2002, it has been dedicated to advancing and innovating the digital audio-video industry with highly efficient and economical encoding/decoding technologies. Three representative generations of video coding standards have been finalized and published, consistently improving the coding performance in the past two decades. The series of AVS standards establish solid foundations for ubiquitous video applications in the areas including acquisition, coding, production, delivery, integrated system, public service, general screen content, and mixed reality media. Along with the standardization process, an extensive amount of studies have been carried out on efficiency-aware designation, algorithm optimization, and hardware implementation of these innovative video coding techniques. This paper explains how those developed techniques provide a lasting impact on the video coding community, extensively, technologically, and systematically. In particular, we provide a comprehensive survey of the three generations of the standards, and timely and in-depth summarize the efforts of the AVS video coding standards in the twenty years. The rate-distortion performance comparisons, in particular in terms of the 8K ultra-high-definition (UHD) contents, reflect the elegant design of the state-of-the-art AVS3 standards. We have also elaborated on a variety of well-established and promising applications, including commercial level real-time 8K encoder, high-frame-rate decoder chip for cell phone, and live streaming solution for sports. In addition, the China Central Television (CCTV) of China Media Group (CMG), the state television of China, has officially launched the first 8K broadcasting channel (CCTV-8K) since 2021 using AVS3. Given the significant success realized by the AVS standards, it is envisioned that a new era of 8K UHD video is arriving.

A Real-Time Ultra-High Definition Video Decoder of AVS3 on Heterogeneous Systems

Recent years have witnessed the exponential increase in the demand for ultra high definition (UHD) video compression. The third generation of Audio Video Coding Standard (AVS3), which is also known as IEEE Standard 1857.10, is the latest audio and video coding standard developed by the China AVS working group. In AVS3, targeting for UHD videos, a series of efficient coding tools have been introduced, leading to the dramatical increase of computational burden. In this scenario, real-time decoding of UHD videos becomes extremely challenging. This paper presents an improved hybrid CPU + GPU accelerated framework for AVS3 decoding. In particular, the motion vector (MV) derivation process is extracted from entropy decoding threads on the CPU. Therefore, the dependency between threads is removed and the entropy decoding can be performed by multiple threads efficiently. Regarding the GPU, we design compact data structures for transform, prediction, and in-loop filtering to reduce the burden of data transmission. A flexible information buffer supporting multi-thread random writing is further created to coordinate the computation between CPU and GPU. Through asynchronous operations on the buffer, the computation of different computing units and the data transmission between them could be performed in parallel. With NVIDIA GeForce RTX 2080Ti GPU and Intel Core i7 8700K CPU, the proposed decoder achieves 151 frames per second (fps) for 4K videos and 55 fps for 8K videos in all intra configuration. In random access configuration, 218 fps and 74 fps are obtained for 4K and 8K videos, respectively.

OpenKinoAI: A Framework for Intelligent Cinematography and Editing of Live Performances

Abstract OpenKinoAI is an open source framework for postproduction of ultra-high definition video, which makes it possible to emulate professional multiclip editing techniques for the case of single-camera recordings. OpenKinoAI includes tools for uploading raw video footage of live performances on a remote web server; detecting, tracking, and recognizing the performers in the original material; reframing the raw video into a large choice of cinematographic rushes; editing the rushes into movies; and annotating rushes and movies for documentation purposes. OpenKinoAI is made available to promote research in multiclip video editing of ultra-high definition video and to allow performing artists and companies to use this research for archiving, documenting, and sharing their work online in an innovative fashion.

An Efficient Real-Time FPGA-Based ORB Feature Extraction for an UHD Video Stream for Embedded Visual SLAM

The detection and description of feature points are important components of many computer vision systems. For example, in the field of autonomous unmanned aerial vehicles (UAV), these methods form the basis of so-called Visual Odometry (VO) and Simultaneous Localisation and Mapping (SLAM) algorithms. In this paper, we present a hardware feature points detection system able to process a 4K video stream in real-time. We use the ORB algorithm—Oriented FAST (Features from Accelerated Segment Test) and Rotated BRIEF (Binary Robust Independent Elementary Features)—to detect and describe feature points in the images. We make numerous modifications to the original ORB algorithm (among others, we use the RS-BRIEF instead of classic R-BRIEF) to adapt it to the high video resolution, make it computationally efficient, reduce the resource utilisation and achieve lower power consumption. Our hardware implementation supports a 4 ppc (pixels per clock) format (with simple adaptation to 2 ppc, 8 ppc, and more) and real-time processing of a 4K video stream (UHD—Ultra High Definition, 3840×2160 pixels) @ 60 frames per second (150 MHz clock). We verify our system using simulations in the Vivado IDE and implement it in hardware on the ZCU 104 evaluation board with the AMD Xilinx Zynq UltraScale+ MPSoC device. The proposed design consumes only 5 watts.

UHD Low-light image enhancement via interpretable bilateral learning

Convolutional neural networks (CNNs) have achieved unparalleled success in the single Low-light Image Enhancement (LIE) task. Existing CNN-based LIE models over-focus on pixel-level reconstruction effects, hence ignoring the theoretical guidance for sustainable optimization, which hinders their application to Ultra-High Definition (UHD) images. To address the above problems, we propose a new interpretable network, which capable of performing LIE on UHD images in real time on a single GPU. The proposed network consists of two CNNs: the first part is to use the first-order unfolding Taylor’s formula to build an interpretable network, and combine two UNets in the form of first-order Taylor’s polynomials. Then we use this constructed network to extract the feature maps of the low-resolution input image, and finally process the feature maps to form a multi-dimensional tensor termed a bilateral grid that acts on the original image to yield an enhanced result. The second part is the image enhancement using the bilateral grid. In addition, we propose a polynomial channel enhancement method to enhance UHD images. Experimental results show that the proposed method significantly outperforms state-of-the-art methods for UHD LIE on a single GPU with 24G RAM (100 fps).

Real-Time Efficient FPGA Implementation of the Multi-Scale Lucas-Kanade and Horn-Schunck Optical Flow Algorithms for a 4K Video Stream.

The information about optical flow, i.e., the movement of pixels between two consecutive images from a video sequence, is used in many vision systems, both classical and those based on deep neural networks. In some robotic applications, e.g., in autonomous vehicles, it is necessary to calculate the flow in real time. This represents a challenging task, especially for high-resolution video streams. In this work, two gradient-based algorithms—Lucas–Kanade and Horn–Schunck—were implemented on a ZCU 104 platform with Xilinx Zynq UltraScale+ MPSoC FPGA. A vector data format was used to enable flow calculation for a 4K (Ultra HD, 3840 × 2160 pixels) video stream at 60 fps. In order to detect larger pixel displacements, a multi-scale approach was used in both algorithms. Depending on the scale, the calculations were performed for different data formats, allowing for more efficient processing by reducing resource utilisation. The presented solution allows real-time optical flow determination in multiple scales for a 4K resolution with estimated energy consumption below 6 W. The algorithms realised in this work can be a component of a larger vision system in advanced surveillance systems or autonomous vehicles.

A Very Fast Copy-Move Forgery Detection Method for 4K Ultra HD Images

Copy-move forgery detection is a challenging task in digital image forensics. Keypoint-based detection methods have proven to be very efficient to detect copied-moved forged areas in images. Although these methods are effective, the keypoint matching phase has a high complexity, which takes a long time to detect forgeries, especially for very large images such as 4K Ultra HD images. In this paper, we propose a new keypoint-based method with a new fast feature matching algorithm, based on the generalized two nearest-neighbor (g2NN) algorithm allowing us to greatly reduce the complexity and thus the computation time. First, we extract keypoints from the input image. After ordering them, we perform a match search restricted to a window around the current keypoint. To detect the keypoints, we propose not to use a threshold, which allows low intensity keypoint matching and a very efficient detection of copy-move forgery, even in very uniform or weakly textured areas. Then, we apply a new matching algorithm, and finally we compute the cluster thanks to the DBSCAN algorithm. Our experimental results show that the method we propose can detect copied-moved areas in forged images very accurately and with a very short computation time which allows for the fast detection of forgeries on 4K images.

Red-emitting micro PeLEDs for UHD displays by using capillary force lithography

Conventional techniques for fabricating micro-light-emitting diode (LED) pixel arrays for ultra-high definition (UHD) displays have drawbacks of high cost and complex processing. Furthermore, it is technically difficult to improve the display resolution using traditional patterning and deposition processes. As the size of the chips decreases to the microscale, there is an increase in the cost due to the additional transfer process, and inaccuracy of mechanical positioning during the repeated transfer process. To solve these problems, a micro-patterning method using capillary force lithography was introduced. Using this method, a 10 µm-scale CsPbBrxI3-x micro-pattern array was fabricated. Moreover, multi-ligand (NMABr0.7I0.3–DMPDAB–PDAB) and poly(ethylene oxide) (PEO) as a hydrophilic polymer were added to the perovskite precursors to control the heterogeneous nucleation and crystal growth during the patterning process. To demonstrate the feasibility of the 10 µm-scale micro-pattern array for the micro-perovskite LEDs (micro-PeLEDs), CsPbBrxI3-x–multi-ligand–PEO precursors were adopted. These micro-PeLEDs exhibited characteristic emission at 650.9 nm with a luminance of 688.4 cd m−2 and external quantum efficiency of 5.9% at 4 V.

P‐24: Distinguished Student Poster: Gate Driver with LTPO‐TFT Circuits for Low Power Consumption and Narrow Bezel AMOLED Displays

We propose a novel gate driver circuit using low‐temperature polycrystalline silicon and oxide (LTPO) thin‐film transistors (TFTs). The proposed circuit operates well when oxide TFTs have VTH of −7.5 V, which is very depletion mode. In addition, by utilizing extremely low off‐state current of oxide TFTs, ultra‐low power consumption can be achieved. The proposed circuit consists of only six TFTs, four p‐type LTPS and two n‐type a‐IGZO TFTs, without capacitor. The fabricated circuit operates with a pulse width of 1 μs corresponding to operating speed of 500 kHz with very small pitch. The proposed circuit can be applied to ultra‐high definition (UHD) and narrow bezel AMOLED display.