Variable Frame Rate Research Articles

Portable Head-Mounted System for Mobile Forearm Tracking.

Computer vision (CV)-based systems using cameras and recognition algorithms offer touchless, cost-effective, precise, and versatile hand tracking. These systems allow unrestricted, fluid, and natural movements without the constraints of wearable devices, gaining popularity in human-system interaction, virtual reality, and medical procedures. However, traditional CV-based systems, relying on stationary cameras, are not compatible with mobile applications and demand substantial computing power. To address these limitations, we propose a portable hand-tracking system utilizing the Leap Motion Controller 2 (LMC) mounted on the head and controlled by a single-board computer (SBC) powered by a compact power bank. The proposed system enhances portability, enabling users to interact freely with their surroundings. We present the system's design and conduct experimental tests to evaluate its robustness under variable lighting conditions, power consumption, CPU usage, temperature, and frame rate. This portable hand-tracking solution, which has minimal weight and runs independently of external power, proves suitable for mobile applications in daily life.

FAVER: Blind quality prediction of variable frame rate videos

Video quality assessment (VQA) remains an important and challenging problem that affects many applications at the widest scales. Recent advances in mobile devices and cloud computing techniques have made it possible to capture, process, and share high resolution, high frame rate (HFR) videos across the Internet nearly instantaneously. Being able to monitor and control the quality of these streamed videos can enable the delivery of more enjoyable content and perceptually optimized rate control. Accordingly, there is a pressing need to develop VQA models that can be deployed at enormous scales. While some recent effects have been applied to full-reference (FR) analysis of variable frame rate and HFR video quality, the development of no-reference (NR) VQA algorithms targeting frame rate variations has been little studied. Here, we propose a first-of-a-kind blind VQA model for evaluating HFR videos, which we dub the Framerate-Aware Video Evaluator w/o Reference (FAVER). FAVER uses extended models of spatial natural scene statistics that encompass space–time wavelet-decomposed video signals, and leverages the advantages of the deep neural network to provide motion perception, to conduct efficient frame rate sensitive quality prediction. Our extensive experiments on several HFR video quality datasets show that FAVER outperforms other blind VQA algorithms at a reasonable computational cost. To facilitate reproducible research and public evaluation, an implementation of FAVER is being made freely available online: https://github.com/uniqzheng/HFR-BVQA.

Motion-resolved four-dimensional abdominal diffusion-weighted imaging using PROPELLER EPI (4D-DW-PROPELLER-EPI).

To develop a distortion-free motion-resolved four-dimensional diffusion-weighted PROPELLER EPI (4D-DW-PROPELLER-EPI) technique for benefiting clinical abdominal radiotherapy (RT). An improved abdominal 4D-DWI technique based on 2D diffusion-weighted PROPELLER-EPI (2D-DW-PROPELLER-EPI), termed 4D-DW-PROPELLER-EPI, was proposed to improve the frame rate of repeated data acquisition and produce distortion-free 4D-DWI images. Since the radial or PROPELLER sampling with golden-angle rotation can achieve an efficient k-space coverage with a flexible time-resolved acquisition, the golden-angle multi-blade acquisition was used in the proposed 4D-DW-PROPELLER-EPI to improve the performance of data sorting. A new k-space and blade (K-B) amplitude binning method was developed for the proposed 4D-DW-PROPELLER-EPI to optimize the number of blades and the k-space uniformity before performing conventional PROPELLER-EPI reconstruction, by using two metrics to evaluate the adequacy of the acquired data. The proposed 4D-DW-PROPELLER-EPI was preliminarily evaluated in both simulation experiments and in vivo experiments with varying frame rates and different numbers of repeated acquisition. The feasibility of achieving distortion-free 4D-DWI images by using the proposed 4D-DW-PROPELLER-EPI technique was demonstrated in both digital phantom and healthy subjects. Evaluation of the 4D completeness metrics shows that the K-B amplitude binning method could simultaneously improve the acquisition efficiency and data reconstruction performance for 4D-DW-PROPELLER-EPI. 4D-DW-PROPELLER-EPI with K-B amplitude binning is an advanced technique that can provide distortion-free 4D-DWI images for resolving respiratory motion, and may benefit the application of image-guided abdominal RT.

A Highly Uniform Luminance and Low-Flicker Pixel Circuit and Its Driving Methods for Variable Frame Rate AMOLED Displays

A Highly Uniform Luminance and Low-Flicker Pixel Circuit and Its Driving Methods for Variable Frame Rate AMOLED Displays

AMOLED Pixel Circuit Using LTPO Technology Supporting Variable Frame Rate from 1 to 120 Hz for Portable Displays.

This paper proposes a new 6T1C pixel circuit based on low-temperature polycrystalline oxide (LTPO) technology for portable active-matrix organic light-emitting diode (AMOLED) displays with variable refresh rates ranging from 1 to 120 Hz. The proposed circuit has a simple structure and is based on the design of sharing lines of switch-controlling signals. It also provides low-voltage driving and immunity to OLED degeneration issues. The calculation and analysis of programming time are discussed, and the optimal storage capacitor for the proposed circuit’s high-speed driving is selected. The results of the simulation reveal that threshold voltage variations in driving thin-film transistors of ±0.33 V can be well sensed and compensated with a 1.8% average shift of OLED currents in high-frame-rate operation (120 Hz), while the maximum variation in OLED currents within all gray levels is only 3.56 nA in low-frame-rate operation (1 Hz). As a result, the proposed 6T1C pixel circuit is a good candidate for use in portable AMOLED displays.

Refresh Rate and Graphical Benchmarks for Mobile VR Application Development

Virtual reality (VR) technology is quickly becoming more accessible to the general public due to the availability and capabilities of modern smartphone devices. However, such mobile devices are not as powerful as high-end desktop systems where VR is mostly established. Running demanding VR apps leads to performance issues such as lag, excessive heat, and fast battery drainage. To avoid these problems, software factors must be optimised. The user evaluation (N=51)(N=51) involved presenting multiple VR scenes (with varying frame rates), requiring participants to judge which scenes felt smooth; the results indicate that anything below 50 FPS was tolerable at best and nauseating at worst. To also measure the performance impact of various software settings, benchmarks were conducted on different smartphones. The results highlight the effects when varying the number of displayed on-screen objects, as well as outlining which settings should be avoid when specifically targeting mobile VR platforms. Heat and battery life were found to be non-issues at recommended performance levels. The proposed work established valuable guidelines which can be helpful for real time applications in development time reduction and complexity simplification from graphical and refresh rate optimization perspectives.

Target Detection Based on Variable Frame Rate Sampling of Active Light Source

In the process of target detection with active light sources as calibration objects, air scattering and air absorption cause a significant loss of light energy, resulting in distortion and fragmentation of the spot shape. Inspired by band-pass filtering, this study proposes a target detection method based on variable frame rate sampling of an active light source. It primarily adopts i) image modulation for collecting the active light source signal with a specified frequency and subtracting the background, and ii) variable frame rate sampling for further weighted average to attenuate the dynamic noise. The experimental results show that the proposed method can efficiently eliminate static background, suppress dynamic noise, and detect the target location without illumination and background requirements.

Making Video Quality Assessment Models Sensitive to Frame Rate Distortions

We consider the problem of capturing distortions arising from changes in frame rate as part of Video Quality Assessment (VQA). Variable frame rate (VFR) videos have become much more common, and streamed videos commonly range from 30 frames per second (fps) up to 120 fps. VFR-VQA offers unique challenges in terms of distortion types as well as in making non-uniform comparisons of reference and distorted videos having different frame rates. The majority of current VQA models require compared videos to be of the same frame rate, but are unable to adequately account for frame rate artifacts. The recently proposed Generalized Entropic Difference (GREED) VQA model succeeds at this task, using natural video statistics models of entropic differences of temporal band-pass coefficients, delivering superior performance on predicting video quality changes arising from frame rate distortions. Here we propose a simple fusion framework, whereby temporal features from GREED are combined with existing VQA models, towards improving model sensitivity towards frame rate distortions. We find through extensive experiments that this feature fusion significantly boosts model performance on both HFR/VFR datasets as well as fixed frame rate (FFR) VQA databases. Our results suggest that employing efficient temporal representations can result much more robust and accurate VQA models when frame rate variations can occur.

Simultaneous measurements of 3D wall shear stress and pulse wave velocity in the murine aortic arch

PurposeWall shear stress (WSS) and pulse wave velocity (PWV) are important parameters to characterize blood flow in the vessel wall. Their quantification with flow-sensitive phase-contrast (PC) cardiovascular magnetic resonance (CMR), however, is time-consuming. Furthermore, the measurement of WSS requires high spatial resolution, whereas high temporal resolution is necessary for PWV measurements. For these reasons, PWV and WSS are challenging to measure in one CMR session, making it difficult to directly compare these parameters. By using a retrospective approach with a flexible reconstruction framework, we here aimed to simultaneously assess both PWV and WSS in the murine aortic arch from the same 4D flow measurement.MethodsFlow was measured in the aortic arch of 18-week-old wildtype (n = 5) and ApoE−/− mice (n = 5) with a self-navigated radial 4D-PC-CMR sequence. Retrospective data analysis was used to reconstruct the same dataset either at low spatial and high temporal resolution (PWV analysis) or high spatial and low temporal resolution (WSS analysis). To assess WSS, the aortic lumen was labeled by semi-automatically segmenting the reconstruction with high spatial resolution. WSS was determined from the spatial velocity gradients at the lumen surface. For calculation of the PWV, segmentation data was interpolated along the temporal dimension. Subsequently, PWV was quantified from the through-plane flow data using the multiple-points transit-time method. Reconstructions with varying frame rates and spatial resolutions were performed to investigate the influence of spatiotemporal resolution on the PWV and WSS quantification.Results4D flow measurements were conducted in an acquisition time of only 35 min. Increased peak flow and peak WSS values and lower errors in PWV estimation were observed in the reconstructions with high temporal resolution. Aortic PWV was significantly increased in ApoE−/− mice compared to the control group (1.7 ± 0.2 versus 2.6 ± 0.2 m/s, p < 0.001). Mean WSS magnitude values averaged over the aortic arch were (1.17 ± 0.07) N/m2 in wildtype mice and (1.27 ± 0.10) N/m2 in ApoE−/− mice.ConclusionThe post processing algorithm using the flexible reconstruction framework developed in this study permitted quantification of global PWV and 3D-WSS in a single acquisition. The possibility to assess both parameters in only 35 min will markedly improve the analyses and information content of in vivo measurements.

Duplicate Frame Video Forgery Detection Using Siamese-based RNN

Video and image data is the most important and widely used format of communication today. It is used as evidence and authenticated proof in different domains such as law enforcement, forensic studies, journalism, and others. With the increase of video applications and data, the problem of forgery in video and images has also originated. Although a lot of work has been done on image forgery, video forensic is still a challenging area. Videos are manipulated in many ways. Frame insertion, deletion, and frame duplication are a few of the major challenges. Moreover, in the perspective of duplicated frames, frame rate variation and loop detection are also key issues. Identification of forged duplication frames for large videos with variant frame rates in real-time is not applicable due to computational limitations, lack of generalization, and low-performance accuracy. This research has investigated the problem of frame duplication with varied frame rates using a deep learning approach. A novel deep learning framework consisting of Inflated 3D (I3D) and Siamese-based Recurrent Neural Network (RNN) is proposed to resolve the aforementioned issues. The first step in the proposed framework is to extract the features and convert videos into frames. I3D network receives an original and a forged video to detect frame-to-frame duplication. Then multiple frames are merged to create a sequence. This sequence is passed to Siamese-based RNN which is used for the sequence to sequence forgery detection in video. Media Forensic Challenge (MFC) is a relatively new dataset with various frame rates, and a huge volume of videos. MFC and Image Retrieval and Analysis Tool (VIRAT) datasets are used for training and validation of the proposed model. The accuracy of the proposed method with the VIRAT dataset is 86.6% and with the MFC dataset 93%. The comparative analysis with state-of-the-art approaches has shown the robustness of the proposed approach.

Quality-Driven Variable Frame-Rate for Green Video Coding in Broadcast Applications

The Digital Video Broadcasting (DVB) has proposed to introduce the Ultra-High Definition services in three phases: UHD-1 phase 1, UHD-1 phase 2 and UHD-2. The UHD-1 phase 2 specification includes several new features such as High Dynamic Range (HDR) and High Frame-Rate (HFR). It has been shown in several studies that HFR (+100 fps) enhances the perceptual quality and that this quality enhancement is content-dependent. On the other hand, HFR brings several challenges to the transmission chain including codec complexity increase and bit-rate overhead, which may delay or even prevent its deployment in the broadcast echo-system. In this paper, we propose a Variable Frame Rate (VFR) solution to determine the minimum (critical) frame-rate that preserves the perceived video quality of HFR video. The frame-rate determination is modeled as a 3-class classification problem which consists in dynamically and locally selecting one frame-rate among three: 30, 60 and 120 frames per second. Two random forests classifiers are trained with a ground truth carefully built by experts for this purpose. The subjective results conducted on ten HFR video contents, not included in the training set, clearly show the efficiency of the proposed solution enabling to locally determine the lowest possible frame-rate while preserving the quality of the HFR content. Moreover, our VFR solution enables significant bit-rate savings and complexity reductions at both encoder and decoder sides.

Variable Frame Rate Display for Cinematic Presentations

While motion pictures have traditionally been captured and presented at a frame rate of 24 frames/s, recent advancements have enabled the capture and presentation at higher frame rates of 48, 60, and even 120 frames/s. Capture and presentation at higher frame rates provide improvements in clarity and motion rendition but come at the cost of increased storage space, processing, and workload for the entire motion picture. Motion picture creators want the capability to reduce the motion judder and blur associated with capturing moving images while still maintaining the cinematic feel of their creations. By introducing a method where the projector can vary the frame rate during playback, the scenes that most benefit from higher frame rates can be captured and presented at that rate, while the scenes that do not benefit from higher frame rates can be captured and presented at lower frame rates. The engineers working with DLP Cinema at Texas Instruments have come up with a method to allow changes to the displayed frame rate to be communicated effectively through the entire playback chain during a presentation with no visible artifacts on screen before, during, or after the transition. This variable frame rate capability enabled by the latest DLP Cinema systems is further described in this paper.

Channel Length Modeling and Experimental Demonstration of Adaptive Threshold Under-Sampled CamCom for Low-SNR Transmitters

Under-sampled camera communication (CamCom) systems realized using fast shutter speeds limit the communication channel length and impose restrictions on the minimum signal-to-noise ratio (SNR) requirement of LED based transmitters. In this paper, we present an empirical model to estimate the effective channel length of a CamCom system. Based upon the model, a transceiver methodology is proposed and demonstrated to support low-SNR transmitters, thereby increasing the effective communication distance of an under-sampled CamCom link. Moreover, an adaptive threshold decoding technique is presented to make the system robust against the variable frame rate phenomenon. The proposed methodology can be utilized to increase the communication distance of under-sampled CamCom links for infrastructure-to-vehicle communication employing low-SNR transmitters.

大幅宽长波红外变帧频变焦成像

大幅宽长波红外变帧频变焦成像

A Low-power CMOS Image Sensor Based on Variable Frame Rate Operation

A Low-power CMOS Image Sensor Based on Variable Frame Rate Operation

Frame Selection for Robust Speaker Identification: A Hybrid Approach

Identification of a person using voice is a challenging task under environmental noises. Important and reliable frame selection for feature extraction from the time-domain speech signal under noise can play a significant role in improving speaker identification accuracy. Therefore, this paper presents a frame selection method using hybrid technique, which combines two techniques, namely, voice activity detection (VAD) and variable frame rate (VFR) analysis. It efficiently captures the active speech part, the changes in the temporal characteristics of the speech signal, taking into account the signal-to-noise ratio, and thereby speaker-specific information. Experimental results on noisy speech, generated by artificially adding various noise signals to the clean YOHO speech at different SNRs have shown improved results for the frame selection by the hybrid technique in comparison with any one of the techniques used for the hybrid. The proposed hybrid technique outperformed both the VFR and the widely used Gaussian statistical model based VAD method for all noise scenarios at different SNRs, except for the Babble noise corrupted speech at 5 dB SNR, for which, VFR performed better. Considering the average identification accuracies of different noise scenarios, a relative improvement of 9.79% over the VFR, and 18.05% over the Gaussian statistical model based VAD method has been achieved.

Smartphone Image Receiver Architecture for Optical Camera Communication

As an extension of visible light communications (VLC), optical camera communications (OCC) are poised to play an important role in the success of VLC technology. Comparing with VLC systems based on photodiodes, OCC systems have a number of advantages in terms of hardware, communications channels, and business trends. However, these systems also have several disadvantages, such as limitations on the data rate, the need for synchronization between the transmitter and receiver, and inter-channel interference. These issues are a result of limitations in the camera sampling rates, frame rate variations, and motion stabilization. To address these limitations, we propose a new image sensor architecture for smartphone-based OCC that incorporates three primary functions: motion stabilization, frame rate control, and auto-exposure control. First, the motion stabilization function combines image sensor data and gyroscope motion information in order to select accurate pixel data. Second, the frame rate control function increases the frame rate using an over-scan scheme and contributes to the data throughput and motion compensation in the time domain. Finally, we control both the auto-exposure and focus functions in order to avoid blurring effects and variations in the frame rate. Experiments show that the proposed receiver architecture with adaptive frame rates and motion stabilization holds great promise for OCC.

Design of current mirror integration ROIC for snapshot mode operation

Current mirror integration (CMI) read out integrated circuit (ROIC) topology provides a low input impedance to photo-detectors and provides large injection efficiency, large charge handling capacity and snapshot mode operation without in-pixel opamps. The ROIC described in this paper has been implemented with a modified current mirror circuit, with matched PMOS pairs for detector input stage and its biasing. The readout circuit has been designed for 30 × 30 μm2 pixel size, 4 × 4 array size, variable frame rate, 5 Mega pixel per second (Mpps). Experimental performance of the test chip has achieved 15 Me charge handling capacity, a high dynamic range of 83 dB, 99.8% linearity and 99.96% injection efficiency. The ROIC design has been fabricated in 3.3 V 1P6M UMC 180 nm CMOS process and tested up to 5 MHz pixel rate at room and at cryogenic temperature.

Emulating displays with continuously varying frame rates

The visual quality of a motion picture is significantly influenced by the choice of the presentation frame rate. Increasing the frame rate improves the clarity of the image and helps to alleviate many artifacts, such as blur, strobing, flicker, or judder. These benefits, however, come at the price of losing well-established film aesthetics, often referred to as the "cinematic look". Current technology leaves artists with a sparse set of choices, e.g., 24 Hz or 48 Hz, limiting the freedom in adjusting the frame rate to artistic needs, content, and display technology. In this paper, we solve this problem by proposing a novel filtering technique which enables emulating the whole spectrum of presentation frame rates on a single-frame-rate display. The key component of our technique is a set of simple yet powerful filters calibrated and evaluated in psychophysical experiments. By varying their parameters we can achieve an impression of continuously varying presentation frame rate in both the spatial and temporal dimensions. This allows artists to achieve the best balance between the aesthetics and the objective quality of the motion picture. Furthermore, we show how our technique, informed by cinematic guidelines, can adapt to the content and achieve this balance automatically.

Methods for reliable estimation of pulse transit time and blood pressure variations using smartphone sensors

Hypertension is known to affect around one third of adults globally and early diagnosis is essential to reduce the effects of this affliction. Today’s Blood Pressure (BP) monitoring cuffs are obtrusive and inconvenient for performing regular measurements, and continuous non-invasive blood pressure devices are too complex and expensive for ambulatory use. Hence, there is a strong need for affordable systems that can measure blood pressure (BP) variations throughout the day as this will allow to monitor, diagnose and follow-up not only patients at risk, but also healthy population in general for early diagnosis. A promising method for arterial BP estimation is to measure the Pulse Transit Time (PTT) and derive pressure values from it. However, current methods for measuring this surrogate marker of BP require complex sensing and analysis circuitry and the related medical devices are expensive and inconvenient for the user. In this paper, we present new methods to estimate PTT reliably and subsequently BP, from the baseline sensors of smartphones. This new approach involves determining PTT by simultaneously measuring the time the blood leaves the heart, by recording the heart sound using the standard microphone of the phone, and the time it reaches the finger, by measuring the pulse wave using the phone’s camera. We present algorithms that can be executed directly on current smartphones to obtain clean and robust heart sound signals and to extract the pulse wave characteristics. We also present methods to ensure a synchronous capture of the waveforms, which is essential to obtain reliable PTT values with inexpensive sensors. Additionally, we combine Autocorrelation and Fast Fourier Transform (FFT)-based methods for reliably estimating the user heart rate (HR) from his/her heart sounds, and describe how to use the calculate HR to compensate for the camera frame rate variations and to improve the robustness of PTT estimation. Our experiments show that the computational overhead of the proposed processing methods is minimum, which allows real-time feedback to the user, and that the PTT values are fully accurate (beat-to-beat), thereby enabling state-of-the-art smartphones to be used as affordable medical devices.