Paper] Fast Large-Scale Multi-View Rendering for Coarse Integral Imaging

seeing through a mask

We present a new parallel algorithm and a system, GigaWalk, for interactive walkthrough of complex, gigabyte-sized environments. Our approach combines occlusion culling and levels-of-detail and uses two graphics pipelines with one or more processors. GigaWalk uses a unified scene graph representation for multiple acceleration techniques, and performs spatial clustering of geometry, conservative occlusion culling, and load-balancing between graphics pipelines and processors. GigaWalk has been used to render CAD environments composed of tens of millions of polygons at interactive rates on systems consisting of two graphics pipelines. Overall, our system's combination of levels-of-detail and occlusion culling techniques results in significant improvements in frame-rate over view-frustum culling or either single technique alone.

Article HLODs for faster display of large static and dynamic environments Share on Authors: Carl Erikson Univ. of North Carolina, Chapel Hill Univ. of North Carolina, Chapel HillView Profile , Dinesh Manocha Univ. of North Carolina, Chapel Hill Univ. of North Carolina, Chapel HillView Profile , William V. Baxter Univ. of North Carolina, Chapel Hill Univ. of North Carolina, Chapel HillView Profile Authors Info & Claims I3D '01: Proceedings of the 2001 symposium on Interactive 3D graphicsMarch 2001 Pages 111–120https://doi.org/10.1145/364338.364376Online:01 March 2001Publication History 64citation781DownloadsMetricsTotal Citations64Total Downloads781Last 12 Months17Last 6 weeks3 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Due to the limitations of communication bandwidth, the transmission of high-resolution images or videos between spacecrafts such as satellites and planetary landers and ground stations has always been a challenging task. To address the issue of high-resolution image transmission, we propose a novel image transmission paradigm based on reference-based super resolution (Ref-SR). This approach involves the simultaneous transmission of both high-resolution and low-resolution images between the spacecraft and ground stations. The low-resolution images are transmitted at a high frame rate, while the high-resolution images are transmitted at a lower frame rate. The high-resolution images serve as references to enhance the quality of the low-resolution images, thereby optimizing bandwidth usage while maintaining image clarity. Compared to directly applying super resolution to low-resolution images, which introducing image distortion due to the domain bias between the space image and its training images, our proposed method keeps the consistency of the super-resolved images and the ground truth images. By using high-resolution images as references, the super resolution process is guided to produce more accurate and reliable high-resolution images. Previous ground-based experiments have demonstrated the feasibility of Ref-SR in image and video restoration, and we believe this method has great potential for lunar communication scenarios.

In this study, we propose a method of estimating high-frame-rate, high-resolution (HR) images with two cameras having different resolutions and frame rates in real time. Estimation of high-frame-rate HR images is important in many applications of high-speed vision for image recognition at a speed of 1 kHz or greater. In general, high-speed vision is useful for real-time control. However, in conventional systems, the image resolution is often insufficient. Our approach involves estimating unknown HR images from low-resolution (LR), high-frame-rate images which are filtered based on an a priori relation between the HR and LR images. In an experiment, we verified the proposed method based on objective and subjective evaluations.

Echocardiography is a key assessment tool for the evaluation of cardiac structure and function. The ability to image cardiac structures using 3-dimensional (3D) echocardiography is evolving. In this article, we present some of the key features of the emerging 3D technology and review its applications with an emphasis on real-time 3D transesophageal echocardiography.

A general-purpose graphics processing unit (GPGPU) has been used for improving computing power in medical ultrasound imaging systems. Recently, a mobile GPU becomes powerful to deal with 3D games and videos at high frame rates on Full HD or HD resolution displays. This paper proposes the method to implement ultrasound signal processing on a mobile GPU available in the high-end smartphone (Galaxy S4, Samsung Electronics, Seoul, Korea) with programmable shaders on the OpenGL ES 2.0 platform. To maximize the performance of the mobile GPU, the optimization of shader design and load sharing between vertex and fragment shader was performed. The beamformed data were captured from a tissue mimicking phantom (Model 539 Multipurpose Phantom, ATS Laboratories, Inc., Bridgeport, CT, USA) by using a commercial ultrasound imaging system equipped with a research package (Ultrasonix Touch, Ultrasonix, Richmond, BC, Canada). The real-time performance is evaluated by frame rates while varying the range of signal processing blocks. The implementation method of ultrasound signal processing on OpenGL ES 2.0 was verified by analyzing PSNR with MATLAB gold standard that has the same signal path. CNR was also analyzed to verify the method. From the evaluations, the proposed mobile GPU-based processing method has no significant difference with the processing using MATLAB (i.e., PSNR<52.51 dB). The comparable results of CNR were obtained from both processing methods (i.e., 11.31). From the mobile GPU implementation, the frame rates of 57.6 Hz were achieved. The total execution time was 17.4 ms that was faster than the acquisition time (i.e., 34.4 ms). These results indicate that the mobile GPU-based processing method can support real-time ultrasound B-mode processing on the smartphone.

Dynamic Frame Rate (DFR) is the change in frame rate of a movie sequence in real time as the sequence is playing. Throughout the majority of the past century and after the introduction of sound in films, frame rates used in films have been kept at a standardization of 24 frame per second despite technological advancement [Salmon et. Al 2011]. In the past decade, spatial resolution has been increasing in display systems while the temporal resolution, the frame rate, has not been changed. Because of this, researchers and filmmakers stress that motion judders and blurriness are much more apparent and they propose that high frame rates will solve the issue [Emoto et. Al 2014] [Turnock 2013]. Some industry experts and critics, however, oppose the use of high frame rates [Wilcox 2015]. Despite all the research and attempts in using high frame rate, the idea of using dynamic frame rate in digital cinema has not been explored in depth. As such, there is very limited information on how people perceive DFR and how it actually works. By understanding DFR and how viewers perceive the changes in frame rate, it will help us adapt new techniques in the creation of cinema. We can utilize high frame rate in sequences that could benefit from high frame rate while keeping the rest of the sequences at standard frame rate. This thesis aims to understand the basics of DFR, how different implementations of DFR changes viewer perception and how people perceive a change of frame rate in an animated movie sequence displayed.

Although motion is a defining feature of moving images, it is also one of their most problematic aspects due to blurred or partially stuttering images (strobe effect) at the standard frame rate of 24 frames per second (fps). This research was conducted to test the aesthetic and perceptual consequences of higher frame rates in narrative films. In a first step, typical camera movements were recorded at different frame rates and shutter angles to compare assumptions about production aesthetics. Film sequences were then tested using questionnaires and eye-tracking measurements in a cinema experiment involving 69 participants. The results showed that while participants valued the enhanced image quality of higher frame rates, they rated the standard frame rate as more realistic. Movements recorded with higher frame rates seem slower. In general, high frame rates produced more perceptual exploration (e.g. higher number of fixations). Second, a complete short film was shot at 96 fps and finished in different versions (96 fps, 48 fps, 24 fps, and a variable frame rate). All frame rate conversions were carried out in postproduction. The version with a variable frame rate was produced to assess its aesthetic potential. Several presentations and subsequent discussions with film professionals, experts, and students revealed an initial preference for the “cinematic look” of the standard frame rate of 24 fps. Some film professionals believe that both staging and editing need to be adapted to higher frame rates. This paper discusses the aesthetic, perceptual, and artistic consequences of higher frame rates.

Scene change detection, one of the fundamental and most important problem of computer vision, plays a very important role in the realization of a complete industrial vision system as well as automated video surveillance system - for automatic scene analysis, monitoring, and generation of alerts based on relevant changes in a video stream. Therefore, in addition to being accurate and robust, a successful scene change detection system must also be of very high frame rate in order to detect scene changes which goes off within a glimpse of the eye and often goes unnoticeable by the conventional frame rate cameras. Keeping the high frame rate processing as main focus, a very high frame rate real-time scene change detection system is developed by leveraging VLSI design to achieve high performance. This is accomplished by proposing, designing, and implementing an area-efficient scene change detection VLSI architecture on FPGA-based IDP Express platform. The developed prototype of complete real-time scene change detection system is capable of processing 2000 frames per second for 512 × 512 video resolution and is tested for live incoming video streams from high speed camera. The proposed and implemented system architecture is adaptable and scalable for different video resolutions and frame rates.

The analysis of knee kinematics and kinematics during high-risk movements associated with anterior cruciate ligament (ACL) injury is essential to studying the mechanism of ACL injury. Motion capture at low frame rates may not always detect the actual peak values during high-speed movements. However, the knee kinetic differences between various frame rate measurements during high-risk movements have not been reported. The purpose of this study was to investigate whether 3D knee kinetics would be measured differently between frame rates such as a high frame rate (1200 Hz), 400 Hz, 240 Hz and the popular low frame rate (120 Hz). Knee kinetics during a single-leg drop landing and side-step cutting under different frame rates were repeatedly measured and statistically compared. Peak knee valgus and tibial internal rotation moments measured at a popular low frame rate were significantly lower than those measured at the other frame rates of 400 Hz and 1200 Hz. The peak anterior and superior forces measured at the highest (1200 Hz) frame rate were significantly higher than those measured at any other frame rates. In addition, the variations in the peak kinetic values were significantly larger at the lowest frame rate capture and trended toward being smaller at the higher frame rates. In conclusion, significant differences in the knee kinetics between the frame rates suggest that the high frequency capturing increases the accuracy of the knee kinetics measurement for high risk maneuvers for the study of ACL injury.

Single-photon avalanche diodes (SPADs) are advanced sensors capable of detecting individual photons and recording their arrival times with picosecond resolution using time-correlated single-photon counting (TCSPC) detection techniques. They are used in various applications, such as LiDAR and low-light imaging. These single-photon cameras can capture high-speed sequences of binary single-photon images, offering great potential for reconstructing 3D environments with high motion dynamics. To complement single-photon data, these cameras are often paired with conventional passive cameras, which capture high-resolution intensity images at a lower frame rate. However, 3D reconstruction from SPAD data faces challenges. Aggregating multiple binary measurements improves precision and reduces noise but can cause motion blur in dynamic scenes. Additionally, SPAD arrays often have lower resolution than passive cameras. To address these issues, we propose a novel computational imaging algorithm to improve the 3D reconstruction of moving scenes from SPAD data by addressing the motion blur and increasing the native spatial resolution. The goal is to turn the high-speed SPAD events, recorded at a high frame rate, into non-blurred high-resolution depth images at the frame rate of the passive sensor. We adopt a plug-and-play approach within an optimization scheme alternating between guided video super-resolution of the 3D scene, and precise image realignment using optical flow. Experiments on synthetic data show that our method significantly improves image resolution across various signal-to-noise ratios and photon levels. We validate our method using real-world SPAD measurements in three practical situations with dynamic objects. First on fast-moving scenes (i.e. fan) in laboratory conditions at a short range (3 meters); second very low-resolution imaging of people with a consumer-grade SPAD sensor from STMicroelectronics; and finally, high-resolution imaging of people walking outdoors in daylight at a range of 325 meters under eye-safe illumination conditions using a short-wave infrared SPAD camera. These results demonstrate the robustness and versatility of our approach.

Plane-wave imaging is desirable for its ability to achieve high frame rates, allowing the capture of fast dynamic events and continuous Doppler data. In most implementations of plane-wave imaging, multiple low-resolution images from different plane wave tilt angles are compounded to form a single high-resolution image, thereby reducing the frame rate. Compounding improves the lateral beam profile in the high-resolution image, but it also acts as a low-pass filter in slow time that causes attenuation and aliasing of signals with high Doppler shifts. This paper introduces a spread-spectrum color Doppler imaging method that produces high-resolution images without the use of compounding, thereby eliminating the tradeoff between beam quality, maximum unaliased Doppler frequency, and frame rate. The method uses a long, random sequence of transmit angles rather than a linear sweep of plane wave directions. The random angle sequence randomizes the phase of off-focus (clutter) signals, thereby spreading the clutter power in the Doppler spectrum, while keeping the spectrum of the in-focus signal intact. The ensemble of randomly tilted low-resolution frames also acts as the Doppler ensemble, so it can be much longer than a conventional linear sweep, thereby improving beam formation while also making the slow-time Doppler sampling frequency equal to the pulse repetition frequency. Experiments performed using a carotid artery phantom with constant flow demonstrate that the spread-spectrum method more accurately measures the parabolic flow profile of the vessel and outperforms conventional plane-wave Doppler in both contrast resolution and estimation of high flow velocities. The spread-spectrum method is expected to be valuable for Doppler applications that require measurement of high velocities at high frame rates.

The frame rate (FR) of a video plays an important role in affecting the perceptual video quality. Most studies about the effect of FR on the video quality mainly focused on low frame rate, e.g. less than 30 frames per second (fps), at low resolutions like CIF or QCIF. As the video frame rate and resolution advance, we reconsider this issue and investigate the relationship between frame rate and the perceptual video quality under high frame rate and high resolution. In this paper, we discuss the impact of frame rate on the perceptual quality of High Definition (HD) video with high frame rates (up to 120 fps) considered. Firstly, we design and conduct subjective experiment to construct the video dataset, which includes video sequences at different frame rates and the corresponding mean opinion scores (MOS) which represent the perceptual video quality. Based on the MOS results, we analyze how perceptual video quality changes as frame rate varies among different video sequences and propose some meaningful findings. The video dataset will be made publicly available. We deem that this study will enrich video quality assessment and benefit the development of high frame rate and high definition video business.KeywordsVideo quality assessmentSubjective experienceHigh frame rateHigh definition

Frame rate has a direct impact on the energy consumption of smartphones: the higher the frame rate, the higher the power consumption. Hence, reducing display refreshes will reduce the power consumption. However, it is risky to manipulate frame rate drastically as it can deteriorate user satisfaction with the device. In this work, we introduce a screen management system that controls the frame rate on smartphone displays based on a model that detects user dissatisfaction due to display refreshes. This approach is based on understanding when higher frame rates are necessary, and providing lower frame rates —thus, saving power— if the lower rate is predicted not to cause user dissatisfaction. According to the results of our first user survey with 20 participants, individuals show highly varying requirements: while some users require high frame rates for the highest satisfaction, others are equally satisfied with lower frame rates. Based on this observation, we develop a system that predicts user dissatisfaction on the runtime and either increases or decreases the maximum frame rate setting. For user dissatisfaction predictions, we have compared two different approaches: (1) static model, which uses dissatisfaction characteristics of a fixed group of people, and (2) user-specific model, which is learning only from the specific user. Our second set of experiments with 20 participants shows that users report 32% less dissatisfaction and 4% more dissatisfaction than the default Android system with user-specific and static systems, respectively. These experiments also show that, compared to the default scheme, our mechanisms reduce the power consumption of the phone by 7.2% and 1.8% on average with the user-specific and static models, respectively.