4K Resolution Research Articles

Monitoring Body Length of Hokkaido Native Horse’s Foals Using a Drone

Monitoring Body Length of Hokkaido Native Horse’s Foals Using a Drone

CES 2022: Less is More

AbstractScaled‐down show still delivers top‐notch exhibits and innovation aplenty.

Perceptual Quality Assessment of HEVC and VVC Standards for 8K Video

With the growing data consumption of emerging video applications and users’ requirement for higher resolutions, up to 8K, a huge effort has been made in video compression technologies. Recently, versatile video coding (VVC) has been standardized by the moving picture expert group (MPEG), providing a significant improvement in compression performance over its predecessor high efficiency video coding (HEVC). In this paper, we provide a comparative subjective quality evaluation between VVC and HEVC standards for 8K resolution videos. In addition, we evaluate the perceived quality improvement offered by 8K over UHD 4K resolution. The compression performance of both VVC and HEVC standards has been conducted in random access (RA) coding configuration, using their respective reference software, VVC test model (VTM-11) and HEVC test model (HM-16.20). Objective measurements, using PSNR, MS-SSIM and VMAF metrics have shown that the bitrate gains offered by VVC over HEVC for 8K video content are around 31%, 26% and 35%, respectively. Subjectively, VVC offers an average of around 41% of bitrate reduction over HEVC for the same visual quality. A compression gain of 50% has been reached for some tested video sequences regarding a Student’s t-test analysis. In addition, for most tested scenes, a significant visual difference between uncompressed 4K and 8K has been noticed.

A Vision System With 1-inch 17-Mpixel 1000-fps Block-Controlled Coded-Exposure Stacked-CMOS Image Sensor for Computational Imaging and Adaptive Dynamic Range Control

This study introduces a vision system that can acquire images at high speeds and high resolutions. Image sensors are used not only in digital still cameras but also in various applications that require capturing wide luminance differences beyond human perception. For example, fast, high-resolution object recognition, and motion tracking in automatic driving systems are essential, particularly in dark tunnels or the mid-summer sunshine. However, the resolution, frame rate, pixel size, and dynamic range should be traded off to achieve a high performance in capturing moving objects with a high contrast. We have developed a high-speed vision system with a readout operation of 1000 fps, resolution of 4K <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${\times }\,4\text{K}$ </tex-math></inline-formula> , dynamic range of 110 dB, and fine pixels of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.7~{\mu }\text{m}$ </tex-math></inline-formula> . These characteristics were achieved using several technologies such as 1) coded exposure (CE), which divides the image plane into smaller blocks and controls the exposure time of each block individually, 2) arrangement of analog-to-digital converters in parallel for each block, and 3) three-dimensional wafer stacking, which enables high-density integration of circuits and pixels. The proposed system can be applied in CE-based computational imaging in addition to high-dynamic-range applications for handling both the dark and bright areas in a scene.

4K Real Time Image to Image Translation Network With Transformers

CNNs have traditionally been applied in computer vision. Recently, applying Transformer networks, originally a technique in natural language processing, to computer vision has received much attention and produced superior results. However, Transformers and their derivation have drawbacks that the computational cost and memory usage increase rapidly with the image resolution. In this paper, we propose the Laplacian Pyramid Translation Transformer (LPTT) for image to image translation. The Laplacian Pyramid Translation Network, a previous study of this work, creates Laplacian pyramid of the input images and processes each component with CNNs. However, LPTT transforms the high-frequency components with CNNs and the low-frequency components with Axial Transformer blocks. LPTT can have Transformer&#x2019;s expressive power while reducing the computational cost and memory usage. LPTT significantly improves the quality of generated images and inference speed for high-resolution images over conventional methods. LPTT is the first network with a Transformer that can perform practical inference in real time on 4K resolution images. LPTT can also process 8K images in real time depending on the model conditions and the performance of the GPU. The ablation study in this paper suggests that even when processing high-resolution images, the performance is improved while maintaining the inference speed by computing the low-resolution component with a Transformer. LPTT improves PSNR value by 0.41 dB in MIT-Adobe FiveK dataset. The greater the number of layers in the Laplacian pyramid, the greater the improvement of LPTT over the Laplacian Pyramid Translation Network.

Adaptive Computation Offloading for Mobile Augmented Reality

Augmented reality (AR) underpins many emerging mobile applications, but it increasingly requires more computation power for better machine understanding and user experience. While computation offloading promises a solution for high-quality and interactive mobile AR, existing methods work best for high-definition videos but cannot meet the real-time requirement for emerging 4K videos due to the long uploading latency. We introduce ACTOR, a novel computation-offloading framework for 4K mobile AR. To reduce the uploading latency, ACTOR dynamically and judiciously downscales the mobile video feed to be sent to the remote server. On the server-side, it leverages image super-resolution technology to scale back the received video so that high-quality object detection, tracking and rendering can be performed on the full 4K resolution. ACTOR employs machine learning to predict which of the downscaling resolutions and super-resolution configurations should be used, by taking into account the video content, server processing delay, and user expected latency. We evaluate ACTOR by applying it to over 2,000 4K video clips across two typical WiFi network settings. Extensive experimental results show that ACTOR consistently and significantly outperforms competitive methods for simultaneously meeting the latency and user-perceived video quality requirements.

Comparison of 2D 4K vs. 3D HD laparoscopic imaging systems using a pelvitrainer model: a randomized controlled study

Laparoscopic surgery provides well-known benefits, but it has technological limitations. Depth perception is particularly crucial, with three-dimensional (3D) imaging being superior to two-dimensional (2D) HD imaging. However, with the introduction of 4K resolution monitors, 2D rendering is capable of providing higher-quality visuals. Therefore, this study aimed to compare 3D HD and 2D 4K imaging using a pelvitrainer model. Eight experts and 32 medical students were performing the same four standardized tasks using 2D 4K and 3D HD imaging systems. Task completion time and the number of errors made were recorded. The Wilcoxon test and mixed-effects models were used to analyze the results. Students were significantly faster in all four tasks when using the 3D HD perspective. The median difference ranged from 18 s in task 3 (P < 0.003) up to 177.5 s in task 4 (P < 0.001). With the exception of task 4, students demonstrated significantly fewer errors in all tasks involving 3D HD imaging. The experts’ results confirmed these findings, as they were also faster in all four tasks using 3D HD, which was significant for task 1 (P < 0.001) and task 4 (P < 0.006). The expert group also achieved better movement accuracy using the 3D HD system, with fewer mistakes made in all four tasks, which was significant in task 4 (P < 0.001). Participants in both groups achieved better results with the 3D HD imaging system than with the 2D 4K system. The 3D HD image system should be used when available. Trial registration: this trial is registered at research registry under the identifier researchregistry6852.

SR-Inpaint: A General Deep Learning Framework for High Resolution Image Inpainting

Recently, deep learning has enabled a huge leap forward in image inpainting. However, due to the memory and computational limitation, most existing methods are able to handle only low-resolution inputs, typically less than 1 K. With the improvement of Internet transmission capacity and mobile device cameras, the resolution of image and video sources available to users via the cloud or locally is increasing. For high-resolution images, the common inpainting methods simply upsample the inpainted result of the shrinked image to yield a blurry result. In recent years, there is an urgent need to reconstruct the missing high-frequency information in high-resolution images and generate sharp texture details. Hence, we propose a general deep learning framework for high-resolution image inpainting, which first hallucinates a semantically continuous blurred result using low-resolution inpainting and suppresses computational overhead. Then the sharp high-frequency details with original resolution are reconstructed using super-resolution refinement. Experimentally, our method achieves inspiring inpainting quality on 2K and 4K resolution images, ahead of the state-of-the-art high-resolution inpainting technique. This framework is expected to be popularized for high-resolution image editing tasks on personal computers and mobile devices in the future.

21.1: Invited Paper: Diversification of Processing Requirements for Large Format IoT Displays: 8K, Programmatic and LED Walls

Large format IoT displays support a range of exciting visual experiences which are encountered in our day‐to‐day lives. Four processor roles, media player, display controller, touch controller and CV/AI module, leverage four processor types, scalar, vector, spatial and matrix, to power these experiences. Algorithm characteristics such as arithmetic, memory access and data‐dependence complexity influence the suitability of an architecture to a workload. Including, GPUs to rendering, CPUs to interactive application, and FPGAs to custom video pipelines. Programmatic advertising, touch‐less kiosks and CODEC enhancements to minimize bitrate of 8K video streams can require inference (AI) workloads. Matrix (VPU) type processors are designed to address this workload type. The narrow pitch LED wall display market is growing at 10x the overall video wall display market rate and leverages spatial FPGA processing in LED controllers, also requiring 8K resolution at smaller surface area sizes as pixel pitch reduces. Requirements on each heterogenous processor type is concluded to be growing with the combination of market trends.

Saliency Computation for Virtual Cinematography in 360° Videos.

Recent advances in virtual reality cameras have contributed to a phenomenal growth of 360$^{\circ }$∘ videos. Estimating regions likely to attract user attention is critical for efficiently streaming and rendering 360$^{\circ }$∘ videos. In this article, we present a simple, novel, GPU-driven pipeline for saliency computation and virtual cinematography in 360$^{\circ }$∘ videos using spherical harmonics (SH). We efficiently compute the 360$^{\circ }$∘ video saliency through the spectral residual of the SH coefficients between multiple bands at over 60FPS for 4K resolution videos. Further, our interactive computation of spherical saliency can be used for saliency-guided virtual cinematography in 360$^{\circ }$∘ videos.

Artist guided generation of video game production quality face textures

We develop a high resolution face texture generation system which uses artist provided appearance controls as the conditions for a generative network. Artists are able to control various elements in the generated textures, such as the skin, eye, lip, and hair color. This is made possible by reparameterizing our dataset to the same UV mapping, allowing us to utilize image-to-image translation networks. Although our dataset is limited in size, only 126 samples in total, our system is still able to generate realistic face textures which strongly adhere to the input appearance attribute conditions because of our training augmentation methods. Once our system has generated the face texture, it is ready to be used in a modern game production environment. Thanks to our novel SuperResolution and material property recovery methods, our generated face textures are 4K resolution and have the associated material property maps required for raytraced rendering.

High-speed computer-generated holography using an autoencoder-based deep neural network.

Learning-based computer-generated holography (CGH) provides a rapid hologram generation approach for holographic displays. Supervised training requires a large-scale dataset with target images and corresponding holograms. We propose an autoencoder-based neural network (holoencoder) for phase-only hologram generation. Physical diffraction propagation was incorporated into the autoencoder's decoding part. The holoencoder can automatically learn the latent encodings of phase-only holograms in an unsupervised manner. The proposed holoencoder was able to generate high-fidelity 4K resolution holograms in 0.15 s. The reconstruction results validate the good generalizability of the holoencoder, and the experiments show fewer speckles in the reconstructed image compared with the existing CGH algorithms.

57‐3: Invited Paper: Diversification of Processing Requirements for Large Format IoT Displays: 8K, Programmatic and LED Walls

Large format IoT displays support a range of exciting visual experiences which are encountered in our day‐to‐day lives. Four processor roles, media player, display controller, touch controller and CV/AI module, leverage four processor types, scalar, vector, spatial and matrix, to power these experiences. Algorithm characteristics such as arithmetic, memory access and data‐dependence complexity influence the suitability of an architecture to a workload. Including, GPUs to rendering, CPUs to interactive application, and FPGAs to custom video pipelines. Programmatic advertising, touch‐less kiosks and CODEC enhancements to minimize bitrate of 8K video streams can require inference (AI) workloads. Matrix (I PU) type processors are designed to address this workload type. The narrow pitch LED wall display market is growing at 10x the overall video wall display market rate and leverages spatial FPGA processing in LED controllers, also requiring 8K resolution at smaller surface area sizes as pixel pitch reduces. Requirements on each heterogenous processor type is concluded to be growing with the combination of market trends.

Integral 3D/2D partially convertible display using geometric phase lens array

Conversion between three-dimensional (3D) and high-definition two-dimensional (2D) images is required to reproduce various video content. Conventional 3D/2D conversion methods based on integral 3D displays mainly aim to convert between 3D and 2D images on the entire display screen. Here, we propose a display method for partial conversion between 3D and 2D images using a geometric phase lens (GPL) array for the backlight control of a liquid crystal (LC) panel. In our method, the backlight is switched between a point light source (PLS) array and diffused light using a GPL array, diffusion plate, and polarization switching element. Integral 3D and high-definition 2D images are simultaneously reproduced on the same screen by displaying elemental images and 2D images on the LC panel according to the state of the backlight. Through numerical simulation experiments, we analyzed the resolution characteristics of 3D images and obtained the design policy of the PLS size and pixel size of the LC panel. Based on the analysis results, we prototyped a display system using a monochromatic LC panel with 8K resolution and evaluated the quality of backlight switching and resolution characteristics of 3D images. Further, we confirmed that 3D images with approximately 16,000 pixels and 2D images with 8K resolution are partially convertible.

47.1: Invited Paper: Development of 55inch 8k AMOLED TV by Inkjet Printing Process

In this paper, we presented 55 inch 8K4K AMOLED TV employing coplanar Oxide TFT backplane, top emissive inkjet‐printing OLED device, Gate Driver on Array(GOA) and compensation technologies. It is so far the largest prototype AMOLED TV fabricated by using inkjet printing process with 8K resolution. It shows the stunning display quality thanks to the high resolution and fast refresh frequency. It proves that the inkjet printing process is not only cost competitive but also can deliver premium display.

Sourcing and Qualifying Passive Polarised 3D TVs

Millions of Stereoscopic 3D capable TVs were sold into the consumer market from 2007 through to 2016. A wide range of display technologies were supported including rear-projection DLP, Plasma, LCD and OLED. Some displays supported the Active 3D method using liquid-crystal shutter glasses, and some displays supported the Passive 3D method using circularly polarised 3D glasses. Displays supporting Full-HD and Ultra-HD (4K) resolution were available in sizes ranging from 32" to 86" diagonal. Unfortunately display manufacturers eventually changed their focus to promoting other display technologies and 2016 was the last year that new 3D TVs were made for the consumer market. Fortunately, there are still millions of 3D displays available through the secondhand- market, however it can be difficult to know which displays have 3D display support. This paper will provide a listing of specifically Passive 3D TVs manufactured by LG, however it has been our experience that the 3D quality varied considerably from one display to another hence it is necessary to qualify the quality of the 3D available on these displays using a testing technique that will be described in the paper.

Enhancement of Pixel-based Video Quality Models using Meta-data

Current state-of-the-art pixel-based video quality models for 4K resolution do not have access to explicit meta information such as resolution and framerate and may not include implicit or explicit features that model the related effects on perceived video quality. In this paper, we propose a meta concept to extend state-of-the-art pixel-based models and develop hybrid models incorporating meta-data such as framerate and resolution. Our general approach uses machine learning to incooperate the meta-data to the overall video quality prediction. To this aim, in our study, we evaluate various machine learning approaches such as SVR, random forest, and extreme gradient boosting trees in terms of their suitability for hybrid model development. We use VMAF to demonstrate the validity of the meta-information concept. Our approach was tested on the publicly available AVT-VQDB-UHD-1 dataset. We are able to show an increase in the prediction accuracy for the hybrid models in comparison with the prediction accuracy of the underlying pixel-based model. While the proof-of-concept is applied to VMAF, it can also be used with other pixel-based models.

Building a video classifier to improve the accuracy of depth-aware frame interpolation

With the current growth in video technology, 4K resolution and sixty frames per second are becoming the industry standards for live, pre-recorded, and animated footage. Although many old movies have a lower frame rate because of insufficient technology, the larger issue lies in the processing of videos. As videos are streamed online, the frame rate can be significantly impaired. So, if we can computationally increase the frame rate, we can dramatically reduce the amount of data being sent to the user. With the recent rising popularity, efficiency, and effectiveness of artificial intelligence, deep learning is undoubtedly the most plausible solution to this problem. Multiple projects have used neural networks to interpolate videos and improve their frame rate. However, since different categories of videos, such as two-dimensional and three-dimensional, can have drastically different color schemes and motion paths, training a single model to handle all of them leads to overfitting, which can be seen in current interpolation algorithms that are specialized to interpolate certain categories of videos. To combat this issue, we researched whether it would be possible to find a single method to perform frame interpolation invariant to the type of video inputted. In this process, we found several pre-existing models that performed well with either 2D or 3D footage, but not both; therefore, we hypothesized that building a video classifier to categorize the input video’s dimensionality would thus improve their accuracy. After integrating the classifier with two depth-aware frame interpolation models, we improved the average accuracy to 97.2%.

8K/16K Video and 3D Audio Coding and Playback for Large-Screen Immersive Spaces

In this paper, we present a set of tools designed to simplify the distribution, playback, and adaptation of high-quality immersive content for large-screen immersive environments. The tools are based on a high-performance high-efficiency video coding (HEVC) codec that allows one to encode, decode, and play ultrahigh-resolution video in 8K and 16K resolutions using standard computing systems. A flexible video-rendering engine allows the content to be adapted to different output display environments. In addition, an immersive audio workflow, based on high-order ambisonics, provides support for 3D and interactive audio. The new tools have been validated in two advanced immersive installations using new immersive content created using various techniques that has been produced in the context of the Immersify European Project. The results show that the new tools enable ultrahigh-resolution immersive video playback combined with 3D audio, and first evaluations indicate that a better immersive experience for final users can be achieved, while at the same time using a simpler workflow.

Improvement for image‐sticking phenomenon occurred in 8K‐liquid crystal displays with adaptive pre‐charge driving technique

AbstractWe proposed adaptive pre‐charge driving (APD) technique that is appropriate for large‐size and high‐resolution liquid crystal display (LCD) panels for TVs to compensate the shortage of charging voltage to a pixel. APD technique enables to realize LCD panels of 8K resolution even though thin‐film‐transistor (TFT) is amorphous‐Si. Recently, this APD renamed line‐overdrive (line‐OD), and it has been reported to be applied to 8K‐LCD panels, and its technical value is increasing. This paper describes an image‐sticking phenomenon with APD. We have made clear that this phenomenon is triggered by the residual DC that was created by the charging performance to the pixel of positive polarity being inferior to the charging performance to that of negative polarity. To solve this issue, it further explains how best to optimize the pre‐charge table for positive and negative polarities, respectively, by effectively using the high‐precision pixel simulation.