Ultra High Definition Research Articles

Video streaming in ultra high definition (4K and 8K) on a portable device employing a Versatile Video Coding standard

The most commonly used H.264 codec standard video compression includes a wide range of devices and platforms. But this codec faces difficulties in capturing 4K and 8K resolution movies. Preserving, uploading, streaming, as well as distributing huge video files, is becoming more difficult. On the other hand, platforms such as Netflix, Vimeo, Hulu, YouTube, and others grow in popularity; they are seeking for techniques to provide viewers with a continuous and higher-quality video environment. To overcome these challenges and to provide effective user comfort, the advanced video compression technique namely HEVC is proposed in this paper. A convergent multi-screen encoder and its functional block of HEVC explain the process of multi-screen encoding of the input video. The parallel mapping process of HEVC enables autonomous functioning at multiple layers efficiently. The video compression of the HEVC standard is created to allow higher-resolution 8 K UHD videos to be displayed at a lower bandwidth. When it comes to compression technique, HEVC is the superior form to H.264 codec. It's cutting-edge, quick, and loaded with features. FVC is utilized for optimization with respect to encoding time and therefore, its supports video compression at a high rate. Thus, the proposed work is experimentally evaluated in terms of encoding time, PSNR, and Bitrate.

Memory-Efficient Deformable Convolution Based Joint Denoising and Demosaicing for UHD Images

This paper introduces deformable convolution in deep learning based joint denoising and demosaicing (JDD), which yields more adaptable representation and larger receptive fields in features extraction for a superior restoration performance. However, the deformable convolution generally leads to considerable computational load and irregular memory access bottleneck, limiting its extensive deployment on edge devices. To address this issue, we develop grouping strategy and assign independent offsets to each kernel group to reduce the computation latency while keeping the accuracy. Motivated by the exploration for aggregate distribution characteristics of deformable offsets, we present the offset sharing methodology to simplify the memory access complexity of deformable convolution. As for hardware acceleration, we specially design a novel deformable matrix multiplication workflow incorporated with a deformable memory mapping unit to boost the computational throughput. The verification experiments on FPGA demonstrate that the proposed deformable convolution based JDD can restore 4K Ultra High Definition (UHD) images at 70FPS and yields significant promotion in visual effect and objective quality assessment.

Feeling of Presence Maximization: mmWave-Enabled Virtual Reality Meets Deep Reinforcement Learning

This paper investigates the problem of providing ultra-reliable and power-efficient virtual reality (VR) experiences for wireless mobile users. To ensure reliable ultra-high-definition (UHD) video frame delivery to mobile users and enhance their immersive visual experiences, a coordinated multipoint (CoMP) transmission technique and millimeter wave (mmWave) communications are exploited. Owing to user movement and time-varying wireless channels, the wireless VR experience enhancement problem is formulated as a sequence-dependent and mixed-integer problem with a goal of maximizing users’ feeling of presence (FoP) in the virtual world, subject to power consumption constraints on access points (APs) and users’ head-mounted displays (HMDs). The problem, however, is hard to be directly solved due to the lack of users’ accurate tracking information and the sequence-dependent and mixed-integer characteristics. To overcome this challenge, we develop a parallel echo state network (ESN) learning method to predict users’ tracking information by training fresh and historical tracking samples separately collected by APs. With the learnt results, we propose a deep reinforcement learning (DRL) based optimization algorithm to solve the formulated problem. In this algorithm, we implement deep neural networks (DNNs) as a scalable solution to produce integer decision variables and solve a continuous power control problem to criticize the integer decision variables. Finally, the performance of the proposed algorithm is compared with various benchmark algorithms, and the impact of different design parameters is also discussed. Simulation results demonstrate that the proposed algorithm is more 4.14% power-efficient than the benchmark algorithms.

Deep Learning Based Frame Interpolation for Stop-motion Animation

This study focuses on deep-learning-based frame interpolation facilitating stop-motion animation production. Through deep learning of the movements of stop-motion animation, frame interpolation works systematically toward its own production. Based on an understanding of the exaggeration or deletion of motions, the proposed frame interpolation technique differs from other linear frame interpolation techniques. This project used the stop motion animation Teenie Scouts BigFive produced by Taktoon Enterprise, a subsidiary of Chung-Ang University Technology Holdings, which was created using 4K UHD high-resolution images at a frame rate of 15 fps.

Photostable aggregation-induced emission of iridium(III) complex realizing robust and high-resolution imaging of latent fingerprints

Preparation of AIEgens with excellent photostability is of vital importance to acquire the optimal performance of aggregation-induced emission (AIE) and remains challenging. In this paper, a handy structural design strategy based on iridium(III) complexes for the creation of photostable AIEgens is reported. The synergistic effect of phenyl and carbazole improves the rotational activity of substituents, then activates the AIE property of iridium(III) complexes with considerably improved photostability. In addition, prompt and high-resolution imaging of latent fingerprints on tinfoil, glass, plastic, and steel is presented by immersion in an aqueous solution of IrCzPh for 60 s. With the aid of photostable AIE, legible and intact fingerprint images demonstrate ultrahigh definition, which can display down to level 3 of fingerprint details and accurately authenticate individual characteristics. Meanwhile, outstanding anti-aging performance of fingerprint images is emerged with no contrast attenuation under 365 nm irradiation for 60 min or aged in natural light for 7 days. Noteworthily, ultrahigh-contrast dynamic imaging of local fingerprint details is achieved.

Ultra-High Definition AV1 FME Interpolation Architectures Exploring Approximate Computing

Modern video encoders like the AOMedia Video 1 (AV1) implement several complex tools to allow the required high level of compression efficiency. The Fractional Motion Estimation (FME) is one of these complex tools, and AV1 FME defines 42 different interpolation filters. To handle such complexity, hardware acceleration using approximate computing has become an interesting alternative to be explored. This paper presents three optimized approximate architectures for the AV1 FME interpolation filters. The architectures reach real time interpolation for UHD 4K videos at 30 frames per second in a low cost, low power, and memory-efficient design. The architectures were synthesized for a 40nm TSMC standard-cells technology reaching power gains up to 83%, when compared to a precise architecture, and up to 20% when compared to a previously published approximated solution. The area gains were also expressive: up to 83% and 40%, respectively. The architectures also allow a memory bandwidth reduction of up to 59.5%, in comparison with the state-of-the-art solutions. The approximations implied small coding efficiency degradation of 0.54% and 1.25% in BD-BR. The presented architectures have the best results found in the literature when considering the trade-off among hardware cost, power dissipation, processing rate, memory bandwidth, and coding efficiency.

Perceptual ultra-high definition video coding based on adaptive just noticeable distortion model

Ultra-high definition (UHD) video is gradually applied into many fields. The huge amount of data in UHD video limits the realization of these applications. Perception video coding can exploit the visual redundancy, and improve the compression performance. So far, the perpetual video coding based on just noticeable distortion (JND) model cannot fully exploit the perceptual redundancy of ultra-high definition (UHD) video/image in 10-bit depth. Thus, an adaptive JND model based on perceptual noise Bayesian network (PNBN) is firstly established for UHD video/image. Then, combined perceptual noise channel encoding/decoding with human visual characteristics, hidden side information of image is dynamically estimated by PNBN. Thirdly, the adaptive JND model is considered both luminance masking effect and texture complexity, which can adaptively exploit redundancy in different texture regions of image. Finally, an effective perceptual video coding (PVC) scheme is proposed for versatile video coding. A novel distortion compensation factor is appropriately designed for rate-distortion optimization. Experimental results show that the proposed PVC scheme achieves significant bitrate reduction with better subjective video quality than the state-of-the-art PVC schemes.

Recent Activities of the Ultra HD Forum

The Ultra HD Forum was established in 2015 to accelerate Ultra HD deployment by bringing together market leaders from every part of the industry. Broadcasters, service providers, consumer electronics, and technology vendors collaborate on solving real-world obstacles in deploying advanced media formats and publishing these solutions for broad adoption. The Forum’s Guidelines Working Group has published Guidelines that describe a uniform set of characteristics for “Ultra HD” content and consistent methods for creating and delivering Ultra HD content.

Development of Transmitter and Receiver With Set Partitioning 64APSK Coded Modulation Designed on Basis of Channel Capacity

In this paper, we present a prototype transmitter and receiver with set partitioning (SP)-64 amplitude and phase shift keying (APSK) coded modulation we developed for expanding satellite transmission capacity. SP-multi-level coded modulation is a transmission scheme that can improve noise immunity by properly combining a modulation scheme, bit allocation, and error-correction codes. We designed SP-64APSK coded modulation on the basis of channel capacity for expanding satellite transmission capacity. We confirmed that it has higher robustness to additive white Gaussian noise (AWGN) than DVB-S2X’s 64APSK modulation with Gray mapping through computer simulation. We also developed a prototype transmitter and receiver to implement our SP-64APSK coded modulation and evaluated its transmission performances in an AWGN channel and in a non-linear channel simulating 12-GHz-band satellite transponder characteristics. We describe the method of designing our SP-64APSK coded modulation through computer simulation, the modulation/demodulation process, and transmission performances of the prototype transmitter and receiver. We also indicate that 8K ultra-high definition television (UHDTV) with an encoded bit rate of 150 Mbps can be transmitted using the prototype within 34.5 MHz, which is the bandwidth of a single transponder for satellite broadcasting in Japan.

Evolution of AVS video coding standards: twenty years of innovation and development

Ever since the founding of the Audio and Video Coding Standard (AVS) Workgroup of China in 2002, it has been dedicated to advancing and innovating the digital audio-video industry with highly efficient and economical encoding/decoding technologies. Three representative generations of video coding standards have been finalized and published, consistently improving the coding performance in the past two decades. The series of AVS standards establish solid foundations for ubiquitous video applications in the areas including acquisition, coding, production, delivery, integrated system, public service, general screen content, and mixed reality media. Along with the standardization process, an extensive amount of studies have been carried out on efficiency-aware designation, algorithm optimization, and hardware implementation of these innovative video coding techniques. This paper explains how those developed techniques provide a lasting impact on the video coding community, extensively, technologically, and systematically. In particular, we provide a comprehensive survey of the three generations of the standards, and timely and in-depth summarize the efforts of the AVS video coding standards in the twenty years. The rate-distortion performance comparisons, in particular in terms of the 8K ultra-high-definition (UHD) contents, reflect the elegant design of the state-of-the-art AVS3 standards. We have also elaborated on a variety of well-established and promising applications, including commercial level real-time 8K encoder, high-frame-rate decoder chip for cell phone, and live streaming solution for sports. In addition, the China Central Television (CCTV) of China Media Group (CMG), the state television of China, has officially launched the first 8K broadcasting channel (CCTV-8K) since 2021 using AVS3. Given the significant success realized by the AVS standards, it is envisioned that a new era of 8K UHD video is arriving.

A Real-Time Ultra-High Definition Video Decoder of AVS3 on Heterogeneous Systems

Recent years have witnessed the exponential increase in the demand for ultra high definition (UHD) video compression. The third generation of Audio Video Coding Standard (AVS3), which is also known as IEEE Standard 1857.10, is the latest audio and video coding standard developed by the China AVS working group. In AVS3, targeting for UHD videos, a series of efficient coding tools have been introduced, leading to the dramatical increase of computational burden. In this scenario, real-time decoding of UHD videos becomes extremely challenging. This paper presents an improved hybrid CPU + GPU accelerated framework for AVS3 decoding. In particular, the motion vector (MV) derivation process is extracted from entropy decoding threads on the CPU. Therefore, the dependency between threads is removed and the entropy decoding can be performed by multiple threads efficiently. Regarding the GPU, we design compact data structures for transform, prediction, and in-loop filtering to reduce the burden of data transmission. A flexible information buffer supporting multi-thread random writing is further created to coordinate the computation between CPU and GPU. Through asynchronous operations on the buffer, the computation of different computing units and the data transmission between them could be performed in parallel. With NVIDIA GeForce RTX 2080Ti GPU and Intel Core i7 8700K CPU, the proposed decoder achieves 151 frames per second (fps) for 4K videos and 55 fps for 8K videos in all intra configuration. In random access configuration, 218 fps and 74 fps are obtained for 4K and 8K videos, respectively.

OpenKinoAI: A Framework for Intelligent Cinematography and Editing of Live Performances

Abstract OpenKinoAI is an open source framework for postproduction of ultra-high definition video, which makes it possible to emulate professional multiclip editing techniques for the case of single-camera recordings. OpenKinoAI includes tools for uploading raw video footage of live performances on a remote web server; detecting, tracking, and recognizing the performers in the original material; reframing the raw video into a large choice of cinematographic rushes; editing the rushes into movies; and annotating rushes and movies for documentation purposes. OpenKinoAI is made available to promote research in multiclip video editing of ultra-high definition video and to allow performing artists and companies to use this research for archiving, documenting, and sharing their work online in an innovative fashion.

An Efficient Real-Time FPGA-Based ORB Feature Extraction for an UHD Video Stream for Embedded Visual SLAM

The detection and description of feature points are important components of many computer vision systems. For example, in the field of autonomous unmanned aerial vehicles (UAV), these methods form the basis of so-called Visual Odometry (VO) and Simultaneous Localisation and Mapping (SLAM) algorithms. In this paper, we present a hardware feature points detection system able to process a 4K video stream in real-time. We use the ORB algorithm—Oriented FAST (Features from Accelerated Segment Test) and Rotated BRIEF (Binary Robust Independent Elementary Features)—to detect and describe feature points in the images. We make numerous modifications to the original ORB algorithm (among others, we use the RS-BRIEF instead of classic R-BRIEF) to adapt it to the high video resolution, make it computationally efficient, reduce the resource utilisation and achieve lower power consumption. Our hardware implementation supports a 4 ppc (pixels per clock) format (with simple adaptation to 2 ppc, 8 ppc, and more) and real-time processing of a 4K video stream (UHD—Ultra High Definition, 3840×2160 pixels) @ 60 frames per second (150 MHz clock). We verify our system using simulations in the Vivado IDE and implement it in hardware on the ZCU 104 evaluation board with the AMD Xilinx Zynq UltraScale+ MPSoC device. The proposed design consumes only 5 watts.

UHD Low-light image enhancement via interpretable bilateral learning

Convolutional neural networks (CNNs) have achieved unparalleled success in the single Low-light Image Enhancement (LIE) task. Existing CNN-based LIE models over-focus on pixel-level reconstruction effects, hence ignoring the theoretical guidance for sustainable optimization, which hinders their application to Ultra-High Definition (UHD) images. To address the above problems, we propose a new interpretable network, which capable of performing LIE on UHD images in real time on a single GPU. The proposed network consists of two CNNs: the first part is to use the first-order unfolding Taylor’s formula to build an interpretable network, and combine two UNets in the form of first-order Taylor’s polynomials. Then we use this constructed network to extract the feature maps of the low-resolution input image, and finally process the feature maps to form a multi-dimensional tensor termed a bilateral grid that acts on the original image to yield an enhanced result. The second part is the image enhancement using the bilateral grid. In addition, we propose a polynomial channel enhancement method to enhance UHD images. Experimental results show that the proposed method significantly outperforms state-of-the-art methods for UHD LIE on a single GPU with 24G RAM (100 fps).

Real-Time Efficient FPGA Implementation of the Multi-Scale Lucas-Kanade and Horn-Schunck Optical Flow Algorithms for a 4K Video Stream.

The information about optical flow, i.e., the movement of pixels between two consecutive images from a video sequence, is used in many vision systems, both classical and those based on deep neural networks. In some robotic applications, e.g., in autonomous vehicles, it is necessary to calculate the flow in real time. This represents a challenging task, especially for high-resolution video streams. In this work, two gradient-based algorithms—Lucas–Kanade and Horn–Schunck—were implemented on a ZCU 104 platform with Xilinx Zynq UltraScale+ MPSoC FPGA. A vector data format was used to enable flow calculation for a 4K (Ultra HD, 3840 × 2160 pixels) video stream at 60 fps. In order to detect larger pixel displacements, a multi-scale approach was used in both algorithms. Depending on the scale, the calculations were performed for different data formats, allowing for more efficient processing by reducing resource utilisation. The presented solution allows real-time optical flow determination in multiple scales for a 4K resolution with estimated energy consumption below 6 W. The algorithms realised in this work can be a component of a larger vision system in advanced surveillance systems or autonomous vehicles.

A Very Fast Copy-Move Forgery Detection Method for 4K Ultra HD Images

Copy-move forgery detection is a challenging task in digital image forensics. Keypoint-based detection methods have proven to be very efficient to detect copied-moved forged areas in images. Although these methods are effective, the keypoint matching phase has a high complexity, which takes a long time to detect forgeries, especially for very large images such as 4K Ultra HD images. In this paper, we propose a new keypoint-based method with a new fast feature matching algorithm, based on the generalized two nearest-neighbor (g2NN) algorithm allowing us to greatly reduce the complexity and thus the computation time. First, we extract keypoints from the input image. After ordering them, we perform a match search restricted to a window around the current keypoint. To detect the keypoints, we propose not to use a threshold, which allows low intensity keypoint matching and a very efficient detection of copy-move forgery, even in very uniform or weakly textured areas. Then, we apply a new matching algorithm, and finally we compute the cluster thanks to the DBSCAN algorithm. Our experimental results show that the method we propose can detect copied-moved areas in forged images very accurately and with a very short computation time which allows for the fast detection of forgeries on 4K images.

Red-emitting micro PeLEDs for UHD displays by using capillary force lithography

Conventional techniques for fabricating micro-light-emitting diode (LED) pixel arrays for ultra-high definition (UHD) displays have drawbacks of high cost and complex processing. Furthermore, it is technically difficult to improve the display resolution using traditional patterning and deposition processes. As the size of the chips decreases to the microscale, there is an increase in the cost due to the additional transfer process, and inaccuracy of mechanical positioning during the repeated transfer process. To solve these problems, a micro-patterning method using capillary force lithography was introduced. Using this method, a 10 µm-scale CsPbBrxI3-x micro-pattern array was fabricated. Moreover, multi-ligand (NMABr0.7I0.3–DMPDAB–PDAB) and poly(ethylene oxide) (PEO) as a hydrophilic polymer were added to the perovskite precursors to control the heterogeneous nucleation and crystal growth during the patterning process. To demonstrate the feasibility of the 10 µm-scale micro-pattern array for the micro-perovskite LEDs (micro-PeLEDs), CsPbBrxI3-x–multi-ligand–PEO precursors were adopted. These micro-PeLEDs exhibited characteristic emission at 650.9 nm with a luminance of 688.4 cd m−2 and external quantum efficiency of 5.9% at 4 V.

P‐24: Distinguished Student Poster: Gate Driver with LTPO‐TFT Circuits for Low Power Consumption and Narrow Bezel AMOLED Displays

We propose a novel gate driver circuit using low‐temperature polycrystalline silicon and oxide (LTPO) thin‐film transistors (TFTs). The proposed circuit operates well when oxide TFTs have VTH of −7.5 V, which is very depletion mode. In addition, by utilizing extremely low off‐state current of oxide TFTs, ultra‐low power consumption can be achieved. The proposed circuit consists of only six TFTs, four p‐type LTPS and two n‐type a‐IGZO TFTs, without capacitor. The fabricated circuit operates with a pulse width of 1 μs corresponding to operating speed of 500 kHz with very small pitch. The proposed circuit can be applied to ultra‐high definition (UHD) and narrow bezel AMOLED display.

MmWave Indoor Channel Measurement Campaign for 5G New Radio Indoor Broadcasting

Due to the great demand of throughput and reliability for multimedia applications in Fifth Generation (5G) networks, many broadcasting systems adopt the Millimeter Wave (mmWave) technology to address the lack of the spectrum resources. As one of 5G-PPP projects, Internet of Radio Light (IoRL) project adopts 40GHz mmWave band to support a high-speed and stable Ultra-High-Definition (UHD) television broadcasting service in the indoor environment. Because of the high frequency property, mmWave bands usually suffers from the high path loss and the penetration loss. Thus, in order to overcome these issues, directional antennas are employed to provide additional power gain while increasing transmission distance. However, the mmWave with directional antennas brings additional problems, such as limited transmission angle and more multipath effects. Therefore, in this paper, for better understanding of impact factors on the signal quality and transmission coverage of the directional 40GHz mmWave band in the indoor environment, a measurement campaign is introduced in detail and the channel characteristics are measured and analysed in varying cases. The mainly concerned characteristics are path loss, shadow fading, average Power Delay Profile (PDP), Root-Mean-Square (RMS) delay spread, arrival rate and coherence bandwidth. All Measured characteristic values are summarised in three tables at the end of this paper. Besides of these, as a reference of channel analysis and a metric of signal quality and effective coverage, Error Vector Magnitude (EVM) of received signal in each case is measured and discussed. Moreover, a simulation is performed based on a statistical channel model to validate the measured results.

Topographic Surveying using Low-Cost Amateur Drones & 4K Ultra-High-Definition Videos

These days, the science of photogrammetry is frequently used in a wide range of applications, including engineering geology, medication, security, navigation, and topographic surveying works. Drones have gradually become an effectual and frequent technique for obtaining a number of photogrammetric products, such as ortho-mosaics, Digital Elevation Model (DEM), and land detailed topographic maps. This paper investigates the possibility of employing low-cost amateur drones with 4K Ultra-High-Definition (UHD) video for topographic surveying works and creating DEM's. DJI Mavic 2 PRO drone with Hasselblad L1D-20C 20 MP camera has been used for covering an expected gas & oil pipeline route in the South of Libya with length of nearly 15 km and width of 200 m. 60 well-distributed high-quality 3D ground points, divided as 30 Ground Control Points (GCP's) and 30 check points, have been used for more stable and robust photogrammetric processing and for reliable evaluation, respectively. The camera has been calibrated twice, before and after executing the flight mission for trustworthy Interior Orientation Elements (IOE's). The recorded video has been split into smaller videos based on the capturing time of the required frames, and the small videos have been extracted to individual UHD photos using Matlab image tools. Leica Photogrammetric Suite (LPS) software has been used for all processing steps, except the automatic filtration of the generated tie points, which has been carried out using self-developed Matlab algorithm for epipolar geometry and 2D transformation based filters. Results show that DEM's with quality of nearly 1.2 to 2 decimeter in plane and elevation, respectively can be obtained using DJI Mavic 2 PRO drone, 4K UHD video taken by Hasselblad L1D-20C 20 MP camera, pixel ground footprint of 8 cm, and flight height of 350 m. This level of accuracy is appropriate for many engineering applications, such as initial-planning projects stretched on huge areas, urban development plans, GIS data collection, inventory of earth works materials, and 3D modeling. The obtained quality of the generated DEM depends on the flight height and the camera quality, IOE's, and resolution. Tests show that using 4K UHD video for photogrammetric applications can provide UHD extracted frames, similar to that captured singly, especially with fit flight speed and camera settings, namely: ISO sensitivity, shatter speed, and aperture size. Also, using video facilitates the aerial photography process, overcoming the difficulties of determining the suitable capturing time and location of individual photos in site. The other advantage is the ability of taking alternative frames if the selected images are not suitable for photogrammetric works in terms of tilting and blurring. The opportunity of changing the overlapping rates across the route is another important advantage of using video, especially for curvy routes. Moreover, using different groups of overlapped images for the same route helps for creating different DEM's for the same area, resulting more precise and dense topographic surveying works.