Abstract
In this paper, we propose a foveation-based super resolution (SR) algorithm to create high resolution images from low resolution inputs for virtual reality head mounted displays. Because the proposed SR algorithm is integrated in the previous foveation-based driving technology to cover the small area around the foveation point that requires high rendering quality, the overall computational complexity is substantially reduced, compared to the whole area SR. The target display has 4 times as high resolution as the input image, therefore, the proposed SR algorithm generates $4\times $ as well as $2\times $ SR images at the same time. To support two SR output images, small area SR, and small number of weights, we employ cropping as well as progressive and recursive framework used in the previous MS-LapSRN. We reduce the number of neurons by placing the deconvolutional layer after convolutional layers, compared to the MS-LapSRN. PSNR and SSIM performances of the proposed SR for the $4\times $ scale are estimated as 31.152 dB and 0.935 for Set5, 26.656 dB and 0.858 for Set14, 27.138 dB and 0.830 for BSD100, and 25.078 dB and 0.836 for Urban100. For the target 8K display of $7,680\times 4.320$ , the proposed FovSR-integrated driving technology achieves the substantial reductions by 76.7 % and 99.02 % on the number of lines from 7,680 to 1,788 and the number of neurons from 24,518,246,400 to 239,541,184, respectively.
Highlights
V IRTUAL reality (VR) has attracted a lot of attention due to its increased potential in various fields such as healthcare, movie, virtual travel, professional sports, education, and gaming [1]–[3]
This paper proposes a low complexity super resolution (SR) framework integrated with the previous foveation-based driving scheme for VR head mounted displays (HMDs) [11], [12]
The proposed foveated SR (FovSR) model sets the number of channels in all layers to 64 excluding the convolutional layers for the RGB image reconstruction, Conv_o2 and Conv_o4, that consist of 3 channels
Summary
V IRTUAL reality (VR) has attracted a lot of attention due to its increased potential in various fields such as healthcare, movie, virtual travel, professional sports, education, and gaming [1]–[3]. The deep learning algorithm has been proposed to expedite the motion tracking function by predicting in advance [10], which enables the early preparation of the image They all need high resolution and high frame rate panels of several thousands of lines, where pixels must be driven within the very short period of time of less than 1 μs. The different network structures are studied including progressive up-sampling frameworks [34]–[36], recursive learning schemes [36]–[38], as well as generative adversarial networks [39], [40] for more stable and higher performances Their performance improvements are obtained by huge computational complexity such as the large numbers of layers, channels, and dimensionalities. The computational complexity can be substantially reduced, compared to the existing SR networks applied to the whole image area
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
