Frame Rate Requirements Research Articles

Perceptions of Hybrid Lighting for Virtual Reality

Virtual reality (VR) applications are computationally demanding due to high frame rate requirements, which precludes large numbers of realtime lights from being used. In this paper, we explore a hybrid lighting approach that combines the benefits of realtime and baked lights based on the importance of each source. First, we demonstrate that the hybrid approach affords better frames per second than realtime and mixed lighting. We then present the results of an online paired-comparison study, in which participants (n = 60) compared videos of the four lighting conditions (baked, mixed, realtime, and hybrid) in terms of preference. Our results indicate that the hybrid lighting approach is better than realtime lighting in terms of graphical performance and also yields better perceptions of quality for scenes with a small to moderately large number of lights.

Separable ConvNet Spatiotemporal Mixer for Action Recognition

Video action recognition is vital in the research area of computer vision. In this paper, we develop a novel model, named Separable ConvNet Spatiotemporal Mixer (SCSM). Our goal is to develop an efficient and lightweight action recognition backbone that can be applied to multi-task models to increase the accuracy and processing speed. The SCSM model uses a new hierarchical spatial compression, employing the spatiotemporal fusion method, consisting of a spatial domain and a temporal domain. The SCSM model maintains the independence of each frame in the spatial domain for feature extraction and fuses the spatiotemporal features in the temporal domain. The architecture can be adapted to different frame rate requirements due to its high scalability. It is suitable to serve as a backbone for multi-task video feature extraction or industrial applications with its low prediction and training costs. According to the experimental results, SCSM has a low number of parameters and low computational complexity, making it highly scalable with strong transfer learning capabilities. The model achieves video action recognition accuracy comparable to state-of-the-art models with a smaller parameter size and fewer computational requirements.

Four-dimensional laser absorption cinematography of species and temperature dynamics at 2 kHz in reacting flows.

A four-dimensional (4D) mid-infrared laser absorption imaging technique has been developed and demonstrated for quantitative, time-resolved, volumetric measurements of temperature and species concentration in dynamic combustion flows. This technique employs a dual high-speed infrared camera setup to capture turnable radiation from a quantum cascade laser near 4.85 µm to resolve rovibrational absorption transitions of carbon monoxide at two orthogonal projection angles. The laser is modulated with a customized waveform to adaptively resolve two target transitions with an increased density of data samples in proximity to the transition peaks, therefore ensuring accurate and quantitative spectral interpretation while minimizing the required frame rate. A 3D masked Tikhonov regularized inversion was performed to reconstruct spectrally resolved absorbance at every grid point of each frame, which enables subsequent interpretation of local gas properties in time. These methods are applied to achieve quantitative 4D cinematography of temperature and carbon monoxide in a propagating C2H4/O2 flame with a spatial pixel resolution of ∼70 µm and a temporal resolution of 2 kHz.

Optical Fiber Alignment Aid with Image Processing on FPGA-based System

This paper introduces an innovative and efficient solution for achieving precise optical fiber alignment in fiber optic splicing applications. By harnessing the capabilities of FPGA-based real-time image processing, the system surpasses the required frame rate of 20 frames per second, ensuring swift and accurate alignment. With a verified resolution of approximately 1 pixel per µm, the system offers high precision suitable for various applications. Through advanced image processing techniques, the system calculates precise correction values for each frame, resulting in fast and accurate alignment of fiber cores and cladding. This unique feature enhances performance and adaptability, leading to substantial reduction in splice wastage. By integrating motorized alignment stages and guided machine vision, the system provides a robust and automated approach to fiber optic alignment. The FPGA-driven stepper motors enable precise and controlled fiber movements with micrometer-level step resolutions. The calculated correction factors effectively guide the alignment process, with deviations from actual results impressively low – a maximum of 12 µm in the Y correction factor and 7 µm in the X correction factor. This reliability makes the system a crucial component in any optical fiber splicing machine. Furthermore, the paper emphasizes the ability to handle specialty fiber alignment contributes significantly to cost-effective manufacturing of fiber optic components and modules. By ensuring precise and optimal fiber alignment, the system mitigates signal loss, leading to improved performance and reduced production costs.

Deep-Learning-Based Multitask Ultrasound Beamforming

In this paper, we present a new method for multitask learning applied to ultrasound beamforming. Beamforming is a critical component in the ultrasound image formation pipeline. Ultrasound images are constructed using sensor readings from multiple transducer elements, with each element typically capturing multiple acquisitions per frame. Hence, the beamformer is crucial for framerate performance and overall image quality. Furthermore, post-processing, such as image denoising, is usually applied to the beamformed image to achieve high clarity for diagnosis. This work shows a fully convolutional neural network that can learn different tasks by applying a new weight normalization scheme. We adapt our model to both high frame rate requirements by fitting weight normalization parameters for the sub-sampling task and image denoising by optimizing the normalization parameters for the speckle reduction task. Our model outperforms single-angle delay and sum on pixel-level measures for speckle noise reduction, subsampling, and single-angle reconstruction.

Intelligent Repair Method for Archival Videos Based on the Super-SloMo Technology

Abstract Due to equipment limitations in the early shooting of video files, and the damage and aging in the later file storage process, some historic video materials in video archives can easily suffer from defects such as poor action coherence. This paper proposes an intelligent enhanced repair method for archival videos. The method first obtains the frame rate of the input video, then calculates the dynamic coefficient of the video according to the frame rate and duration number, uses the artificial intelligence algorithm Super-SloMo technology to fix the problematic digital video to the required frame rate, improves the video's dynamic coefficient, and finally achieves the purpose of getting the enhanced repaired digital copies of the original video materials from video archives. This paper also proposes an intelligent enhanced repair prototype system based on the method. The experiment shows that the enhanced repair effect is effective and improves the clarity and coherence of the video. Users get more coherent copies of historic video materials. This solves the problem of the poor perception caused by the loss of frames, preserves the original information, and makes the information obtained from archival videos intuitive.

Multi-Resolution 3D Rendering for High-Performance Web AR.

In the context of web augmented reality (AR), 3D rendering that maintains visual quality and frame rate requirements remains a challenge. The lack of a dedicated and efficient 3D format often results in the degraded visual quality of the original data and compromises the user experience. This paper examines the integration of web-streamable view-dependent representations of large-sized and high-resolution 3D models in web AR applications. The developed cross-platform prototype exploits the batched multi-resolution structures of the Nexus.js library as a dedicated lightweight web AR format and tests it against common formats and compression techniques. Built with AR.js and Three.js open-source libraries, it allows the overlay of the multi-resolution models by interactively adjusting the position, rotation and scale parameters. The proposed method includes real-time view-dependent rendering, geometric instancing and 3D pose regression for two types of AR: natural feature tracking (NFT) and location-based positioning for large and textured 3D overlays. The prototype achieves up to a 46% speedup in rendering time compared to optimized glTF models, while a 34 M vertices 3D model is visible in less than 4 s without degraded visual quality in slow 3D networks. The evaluation under various scenes and devices offers insights into how a multi-resolution scheme can be adopted in web AR for high-quality visualization and real-time performance.

Frame rate effects and their compensation on super-resolution microvessel imaging using ultrasound localization microscopy

Ultrasound localization microscopy (ULM) breaks the diffraction limit and allows imaging microvasculature at micrometric resolution while preserving the penetration depth. Frame rate plays an important role for high-quality ULM imaging, but there is still a lack of review and investigation of the frame rate effects on ULM. This work aims to clarify how frame rate influences the performance of ULM, including the effects of microbubble detection, localization and tracking. The performance of ULM was evaluated using an in vivo rat brain dataset (15.6 MHz, 3 tilted plane waves (−5°, 0°, +5°), at a compounded frame rate of 1000 Hz) with different frame rates. Quantification methods, including Fourier ring correlation and saturation parameter, were applied to analyze the spatial resolution and reconstruction efficiency, respectively. In addition, effects on each crucial step in ULM processing were further analyzed. Results showed that when frame rates dropped from 1000 Hz to 250 Hz, the spatial resolution deteriorated from 9.9 μm to 15.0 μm. Applying a velocity constraint was able to improve the ULM performance, but inappropriate constraint may artificially result in high apparent resolution. For the dataset, compared with the results of 1000 Hz frame rate, the velocity was underestimated at 100 Hz with 47.18% difference and the saturation was reduced from 55.00% at 1000 Hz to 43.34% at 100 Hz. Analysis showed that inadequate frame rate generated unreliable microbubble detection, localization and tracking as well as incomplete track reconstruction, resulting in the deterioration in spatial resolution, the underestimation in velocity measurement and the decrease in saturation. Finally, a guidance of determining the frame rate requirement was discussed by considering the required spatial sampling points based on vessel morphology, clutter filtering method, tracking algorithm and acquisition time, which provides indications for future clinical application of ULM method.

Quantifying cilia beat frequency using high-speed video microscopy: Assessing frame rate requirements when imaging different ciliated tissues.

Motile cilia are found in numerous locations throughout our body and play a critical role in various physiological processes. The most commonly used method to assess cilia motility is to quantify cilia beat frequency (CBF) via video microscopy. However, a large heterogeneity exists within published literature regarding the framerate used to image cilia motility for calculating CBF. The aim of this study was to determine the optimal frame rate required to image cilia motility for CBF assessment, and if the Nyquist theorem may be used to set this rate. One‐second movies of cilia were collected at >600 fps from mouse airways and ependyma at room‐temperature or 37°C. Movies were then down‐sampled to 30–300 fps. CBF was quantified for identical cilia at different framerates by either manual counting or automated MATLAB script. Airway CBF was significantly impaired in 30 fps movies, while ependymal CBF was significantly impaired in both 60 and 30 fps movies. Pairwise comparison showed that video framerate should be at least 150 fps to accurately measure CBF, with minimal improvement in CBF accuracy in movies >150 fps. The automated script was also found to be less accurate for measuring CBF in lower fps movies than manual counting, however, this difference disappeared in higher framerate movies (>150 fps). In conclusion, our data suggest the Nyquist theorem is unreliable for setting sampling rate for CBF measurement. Instead, sampling rate should be 3–4 times faster than CBF for accurate CBF assessment. Especially if CBF calculation is to be automated.

Clustered voxel real-time global illumination

Real-time global illumination is an extremely challenging problem because of its intrinsic complexity due to the interplay between complex geometry, multiple light bounces, and stringent real-time frame-rate requirements. In this paper, we present a new technique that enables the real-time computation of global illumination in generic scenes for diffuse surfaces and static geometry. Our technique combines a voxel-based representation of the scene, a voxel-clustering algorithm, and an iterative light-propagation algorithm based on the resulting clusters. Although our implementation is currently designed to handle the first (and main) bounce, the technique allows for multiple light bounces. Summing up all these components results in a flexible algorithm capable of providing global illumination effects and on-the-fly illumination computations.

Real-time implementation of fast discriminative scale space tracking algorithm

Real-time object tracking is an important step of many modern image processing applications. The efficient hardware design of real-time object tracker must achieve the desired accuracy while satisfying the frame rate requirements for a variety of image sizes. The existing methods of visual tracking employ sophisticated algorithms and challenge the capabilities of most embedded architectures. Discriminative scale space tracking is one algorithm that is capable of demonstrating good performance with affordable complexity. It has a high degree of parallelism which can be exploited for efficient implementation of reconfigurable hardware architectures. This paper proposes a real-time implementation of the discriminative scale-space tracker on FPGA for the major blocks. A careful design exploration of core mathematical operations of the tracking algorithm is performed to improve their hardware utilization and timing performance. Among the core functional units optimized in this work, the discrete Fourier transform achieves a computational time improvement of 92% relative to existing works, QR factorization achieves a 2.3times reduction in resource utilization, and singular value decomposition yields a 3.8times improvement in processing time. The proposed data path architecture is designed using Vivado HLS tool set and implemented for Zync Zed Board (xc7z020clg484-1). For an input image size of 320 times 240, the proposed architecture achieves a mean 25.38 fps.

An Efficient Algorithm Based on Spectrum Migration for High Frame Rate Ultrasound Imaging

The high frame rate (HFR) imaging technique requires only one emission event for imaging. Therefore, it can achieve ultrafast imaging with frame rates up to the kHz regime, which satisfies the frame rate requirements for imaging moving tissues in scientific research and clinics. Lu’s Fourier migration method is based on a non-diffraction beam to obtain HFR images and can improve computational speed and efficiency. However, in order to obtain high-quality images, Fourier migration needs to make full use of the spectrum of echo signals for imaging, which requires a large number of Fast Fourier Transform (FFT) points and increases the complexity of the hardware when the echo frequency is high. Here, an efficient algorithm using the spectrum migration technique based on the spectrum’s distribution characteristics is proposed to improve the imaging efficiency in HFR imaging. Since the actual echo signal spectrum is of limited bandwidth, low-frequency and high-frequency parts with low-energy have little contribution to the imaging spectrum. We transform the effective part that provides the main energy in the signal spectrum to the imaging spectrum while the ineffective spectrum components are not utilized for imaging. This can significantly reduce the number of Fourier transform points, improve Fourier imaging efficiency, and ensure the imaging quality. The proposed method is evaluated on simulated and experimental datasets. Results demonstrated that the proposed method could achieve equivalent image quality with a reduced point number for FFT compared to the complete spectrum migration. In this paper, it only requires a quarter of the FFT points used in the complete spectrum migration, which can improve the computational efficiency; thus, it is more suitable for real-time data processing. The proposed spectrum migration method has a specific significance for the study and clinical application of HFR imaging.

Human Pose Simulation and Detection in Real Time Using Video Streaming Data

This paper proposes a simulation approach to detect different human poses in real time with streaming data. Pose detection in real time is a critical area for many of the applications in different domains where the available literature deals with training-based models on huge amount of data and the methods use 3D cameras for accurate predictions. It also requires huge computational efforts and GPU machines to obtain different human poses in real time. The available methods in the literature are with high frame rate requirement and use previous frames as well for predicting the poses of human. If the frame rate is less, then the methods fail to predict the poses accurately and efficiently. The proposed simulation mechanism describes the simulation of different poses of human and generates feature descriptors for each of the pose and trains the model using simple classifier. The trained model predicts the real-time human pose detection on video streaming data. The different poses are predicted with less frame rate using simple 2D cameras and with accurate predictions by reducing the processing time and with less computational efforts. The proposed solution will be used to predict the candidate poses or gestures in the virtual interview application.

Research on Potentialities of Audio Information Recovery from Video Without Audio Track

The article analyzed the possibility of the appearance of an acoustic information leakage channel that is formed by analyzing the video stream on the video record. The authors investigated the possibilities of speech recovery from a low quality recording, determined by the signal-to-noise ratio (SNR), sampling frequency, number of quantization levels, and clipping level, taking into account the features of the leakage channel under study. As a result, the required frame rate of the video image, the minimum SNR, the number of quantization levels, and a sufficient dynamic displacement range of the oscillating object are determined. The authors also investigated the requirements for the leakage channel parameters and possible ways for an attacker to improve its quality. The requirements for the displacement of an object oscillating under the action of acoustic waves in a video were calculated. The article justified the potential of reducing the requirements for the displacement of an object by applying averaging of a large number of different points on the object. The authors performed an assessment of the existing noise reduction software for sound recordings, which is used to increase the intelligibility of the message that is intercepted by the attacker in the considered information leakage channel. Obtained results revealed that there are potential causes for the leakage of acoustic information by analyzing the video stream on the video. The conditions for the emergence of such a channel are not excessive. Therefore, the possibility of its appearance is a security risk and it is necessary to provide the means to protect the object of information activity from it.

Distributing Synthetic Focusing Over Multiple Push-Detect Events Enhances Shear Wave Elasticity Imaging Performance.

Plane wave (PW) imaging is a commonly used method for tracking waves during shear wave elasticity imaging (SWEI), but its unfocused transmission beam reduces tracking accuracy and precision. Coherent compounding minimizes this problem, but SWEI's stringent frame rate requirement and the coarse pitch of most clinical transducers limit its effectiveness. Synthetic aperture imaging (SAI) is an alternate ultrasound imaging approach with a much tighter focus than PW imaging, but its lower transmission power has deterred researchers from using SAI in SWEI. Hadamard-encoded multielement SAI can overcome this limitation. However, only a limited number of subapertures (3-5) can be transmitted in a single push-detect event. We have developed methods to distribute more subapertures or more compounding angles over multiple push-detect events. In this paper, we report the results of experiments conducted on phantoms to assess SWEI's performance when using Hadamard-encoded distributed-multielement synthetic aperture (HDMSA) imaging or distributed plane wave compounding (DPWC) to track shear waves. Tracking shear waves with HDMSA improved the elastographic signal-to-noise ratio (SNRe) by 61.6%-89.5% depending on the phantom employed. Similarly, DPWC tracking improved SNRe by 56.2%-93.3% for the same group of phantoms. Compared to focused ultrasound tracking (at the focus), SNRe improved by 28.6% and 32.5% when tracking shear waves with HDMSA and DPWC, respectively. Long acquisitions could introduce decoding errors that decrease the performance when performing HDMSA tracking within the clinical setting. Nevertheless, the results of studies performed on the bicep muscle of three healthy volunteers demonstrate that for stationary organs, tracking shear waves with HDMSA yielded repeatable elastograms that offer better elastographic performance than those produced with current tracking methods.

Immersive Virtual Colonoscopy.

Virtual colonoscopy (VC) is a non-invasive screening tool for colorectal polyps which employs volume visualization of a colon model reconstructed from a CT scan of the patient's abdomen. We present an immersive analytics system for VC which enhances and improves the traditional desktop VC through the use of VR technologies. Our system, using a head-mounted display (HMD), includes all of the standard VC features, such as the volume rendered endoluminal fly-through, measurement tool, bookmark modes, electronic biopsy, and slice views. The use of VR immersion, stereo, and wider field of view and field of regard has a positive effect on polyp search and analysis tasks in our immersive VC system, a volumetric-based immersive analytics application. Navigation includes enhanced automatic speed and direction controls, based on the user's head orientation, in conjunction with physical navigation for exploration of local proximity. In order to accommodate the resolution and frame rate requirements for HMDs, new rendering techniques have been developed, including mesh-assisted volume raycasting and a novel lighting paradigm. Feedback and further suggestions from expert radiologists show the promise of our system for immersive analysis for VC and encourage new avenues for exploring the use of VR in visualization systems for medical diagnosis.

OPTIMIZED ORGANIZATION AND ADAPTIVE VISUALIZATION OF COMPLICATED MOUNTAIN DISASTER 3D SCENES FOR DIVERSE TERMINALS

Abstract. Mountain disaster scenes usually contains various geographical entities, which are dynamic and complicated. Therefore, the construction of mountain disaster 3D scenes has great significance for disaster simulation, analysis and prediction. 3D visualization and interactive analysis of complicated mountain disaster scenes should accommodate users who have access to various terminals in disaster emergency response. However, most of the existing 3D visualization methods can only deal with 3D scene data organization and scheduling for single terminal or limited kinds of geographical entities. Due to performance constraints, it’s difficult for Mobile devices to support efficient visualization of complicated mountain disaster 3D scenes, either. To address these issues, we research the key technologies for efficient visualization of mountain disasters on diverse terminals. We utilize the B/S architecture and research its impact on rendering frame rate and the relationship of terminal characteristics and parameters through analysis of terminal characteristics and parameters, e.g. hardware performance, screen size and resolution, network environment, and rendering frame rate requirements. Then we analyze the diverse organization of mountain disaster scene data and explore the methods of constructing efficient spatial index by taking into account the characteristics of diverse terminals. An adaptive scene analysis method is subsequently designed to select the optimal model. Finally, based on the diverse organization of various data in the scene, a corresponding dynamic scheduling method is proposed to realize the adaptive visualization of complicated mountain disaster 3D scenes for diverse terminals.

Lightweight Visual Odometry for Autonomous Mobile Robots.

Vision-based motion estimation is an effective means for mobile robot localization and is often used in conjunction with other sensors for navigation and path planning. This paper presents a low-overhead real-time ego-motion estimation (visual odometry) system based on either a stereo or RGB-D sensor. The algorithm’s accuracy outperforms typical frame-to-frame approaches by maintaining a limited local map, while requiring significantly less memory and computational power in contrast to using global maps common in full visual SLAM methods. The algorithm is evaluated on common publicly available datasets that span different use-cases and performance is compared to other comparable open-source systems in terms of accuracy, frame rate and memory requirements. This paper accompanies the release of the source code as a modular software package for the robotics community compatible with the Robot Operating System (ROS).

An FPGA-based implementation of corner detection and matching with outlier rejection

ABSTRACTTo meet the high frame rate requirements of correct point correspondences with a sub-pixel precision, this paper first proposes a Field Programmable Gate Array (FPGA) architecture that consists of corner detection, corner matching, outlier rejection, and sub-pixel precision localisation. In the architecture, a combined Features from Accelerated Segment Test (FAST)+ Binary Robust Independent Elementary Features (BRIEF) algorithm is adopted for detection and matching with pixel precision, a combined algorithm of Slope-based Rejection (SR) and Correlation-Coefficient-based Rejection (CCR) is used to reject the outliers, and a gradient centroid-based algorithm is used for sub-pixel precision localisation. The whole FPGA architecture is implemented on a single FPGA platform (Xilinx XC72K325T). Five image datasets with different spatial resolutions, textures, lights, rotate angles, and viewpoints are used to evaluate the performance of the FPGA-based implementation. The experimental results show that (1) the SR and CCR algorithms are effective for outlier rejection; (2) a higher correct matching rate is achieved for the image pairs that cover artificial textures than for those that cover natural textures; (3) the proposed architecture is also suitable for image pairs with small change of lights, rotate angles, and viewpoints; (4) the speed of the FPGA-based implementation can reach 280 Frames Per Second (FPS), which is 35 times faster than the Personal Computer (PC)-based implementation; and (5) the usage of FPGA resources is acceptable for the selected FPGA platform. The speed and usage of the FPGA resources can be improved when the whole FPGA-based implementation is further optimised.

Construction and Optimization of Three-Dimensional Disaster Scenes within Mobile Virtual Reality

Because mobile virtual reality (VR) is both mobile and immersive, three-dimensional (3D) visualizations of disaster scenes based in mobile VR enable users to perceive and recognize disaster environments faster and better than is possible with other methods. To achieve immersion and prevent users from feeling dizzy, such visualizations require a high scene-rendering frame rate. However, the existing related visualization work cannot provide a sufficient solution for this purpose. This study focuses on the construction and optimization of a 3D disaster scene in order to satisfy the high frame-rate requirements for the rendering of 3D disaster scenes in mobile VR. First, the design of a plugin-free browser/server (B/S) architecture for 3D disaster scene construction and visualization based in mobile VR is presented. Second, certain key technologies for scene optimization are discussed, including diverse modes of scene data representation, representation optimization of mobile scenes, and adaptive scheduling of mobile scenes. By means of these technologies, smartphones with various performance levels can achieve higher scene-rendering frame rates and improved visual quality. Finally, using a flood disaster as an example, a plugin-free prototype system was developed, and experiments were conducted. The experimental results demonstrate that a 3D disaster scene constructed via the methods addressed in this study has a sufficiently high scene-rendering frame rate to satisfy the requirements for rendering a 3D disaster scene in mobile VR.