Frame Rate Requirements Research Articles

A Single-Tier Virtual Queuing Memory Controller Architecture for Heterogeneous MPSoCs

Heterogeneous MPSoCs typically integrate diverse cores, including application CPUs, GPUs, and HD coders. These cores commonly share an off-chip memory to save cost and energy, but their memory accesses often interfere with each other, leading to undesirable consequences like a slowdown of application performance or a failure to sustain real-time performance. The memory controller plays a central role in meeting the QoS needs of real-time cores while maximizing CPU performance. Previous QoS-aware memory controllers are based on a classic two-tier queuing architecture that buffers memory transactions at the first tier, followed by a second tier that buffers translated DRAM commands. In these designs, QoS-aware policies are used to schedule competing transactions at the first stage, but the translated DRAM commands are served in FIFO order at the second stage. Unfortunately, once the scheduled transactions have been forwarded to the command stage, newly arriving transactions that may be more critical cannot be served ahead of those translated commands that are already queued at the second stage. To address this, we propose a scalable memory controller architecture based on single-tier virtual queuing (STVQ) that maintains a single tier of request queues and employs an efficacious scheduler that considers both QoS requirements and DRAM bank states. In comparison with previous QoS-aware memory controllers, the proposed STVQ memory controller reduces CPU slowdown by up to 13.9% while satisfying all frame rate requirements. We propose further optimizations that can significantly increase row-buffer hits by up to 66.2% and reduce memory latency by up to 19.8%.

An Accelerator for High Efficient Vision Processing

In recent years, neural network accelerators have been shown to achieve both high energy efficiency and high performance for a broad application scope within the important category of recognition and mining applications. Still, both the energy efficiency and performance of such accelerators remain limited by memory accesses. In this paper, we focus on image applications, arguably the most important category among recognition and mining applications. The neural networks which are state-of-the-art for these applications are convolutional neural networks (CNNs), and they have an important property: weights are shared among many neurons, considerably reducing the neural network memory footprint. This property allows to entirely map a CNN within an SRAM, eliminating all DRAM accesses for weights. By further hoisting this accelerator next to the image sensor, it is possible to eliminate all remaining DRAM accesses, i.e., for inputs and outputs. In this paper, we propose such a CNN accelerator, placed next to a CMOS or CCD sensor. The absence of DRAM accesses combined with a careful exploitation of the specific data access patterns within CNNs allows us to design an accelerator which is highly energy-efficient. We present a single-core implementation down to the layout at 65 nm, with a modest footprint of 5.94mm <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^{\boldsymbol {2}}$ </tex-math></inline-formula> and consuming only 336mW, but still about <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\boldsymbol {30\times }$ </tex-math></inline-formula> faster than high-end GPUs. For visual processing with higher resolution and frame-rate requirements, we further present a multicore implementation with elevated performance.

Real-time simulation of ultrasound refraction phenomena using ray-trace based wavefront construction method

Ultrasound (US) imaging is one of the most popular techniques used in clinical diagnosis, mainly due to lack of adverse effects on patients and the simplicity of US equipment. However, the characteristics of the medium cause US imaging to imprecisely reconstruct examined tissues. The artifacts are the results of wave phenomena, i.e. diffraction or refraction, and should be recognized during examination to avoid misinterpretation of an US image. Currently, US training is based on teaching materials and simulators and ultrasound simulation has become an active research area in medical computer science. Many US simulators are limited by the complexity of the wave phenomena, leading to intensive sophisticated computation that makes it difficult for systems to operate in real time. To achieve the required frame rate, the vast majority of simulators reduce the problem of wave diffraction and refraction.The following paper proposes a solution for an ultrasound simulator based on methods known in geophysics. To improve simulation quality, a wavefront construction method was adapted which takes into account the refraction phenomena. This technique uses ray tracing and velocity averaging to construct wavefronts in the simulation. Instead of a geological medium, real CT scans are applied. This approach can produce more realistic projections of pathological findings and is also capable of providing real-time simulation.

On the Single-Photon-Counting (SPC) modes of imaging using an XFEL source

The requirements to achieve high detection efficiency (above 50%) and gigahertz (GHz) frame rate for the proposed 42-keV X-ray free-electron laser (XFEL) at Los Alamos are summarized. Direct detection scenarios using C (diamond), Si, Ge and GaAs semiconductor sensors are analyzed. Single-photon counting (SPC) mode and weak SPC mode using Si can potentially meet the efficiency and frame rate requirements and be useful to both photoelectric absorption and Compton physics as the photon energy increases. Multilayer three-dimensional (3D) detector architecture, as a possible means to realize SPC modes, is compared with the widely used two-dimensional (2D) hybrid planar electrode structure and 3D deeply entrenched electrode architecture. Demonstration of thin film cameras less than 100-μm thick with onboard thin ASICs could be an initial step to realize multilayer 3D detectors and SPC modes for XFELs.

Real-time high-resolution downsampling algorithm on many-core processor for spatially scalable video coding

The progression toward spatially scalable video coding (SVC) solutions for ubiquitous endpoint systems introduces challenges to sustain real-time frame rates in downsampling high-resolution videos into multiple layers. In addressing these challenges, we put forward a hardware accelerated downsampling algorithm on a parallel computing platform. First, we investigate the principal architecture of a serial downsampling algorithm in the Joint-Scalable-Video-Model reference software to identify the performance limitations for spatially SVC. Then, a parallel multicore-based downsampling algorithm is studied as a benchmark. Experimental results for this algorithm using an 8-core processor exhibit performance speedup of 5.25× against the serial algorithm in downsampling a quantum extended graphics array at 1536p video resolution into three lower resolution layers (i.e., Full-HD at 1080p, HD at 720p, and Quarter-HD at 540p). However, the achieved speedup here does not translate into the minimum required frame rate of 15 frames per second (fps) for real-time video processing. To improve the speedup, a many-core based downsampling algorithm using the compute unified device architecture parallel computing platform is proposed. The proposed algorithm increases the performance speedup to 26.14× against the serial algorithm. Crucially, the proposed algorithm exceeds the target frame rate of 15 fps, which in turn is advantageous to the overall performance of the video encoding process.

Experimental investigation on minimum frame rate requirements of high-speed videoendoscopy for clinical voice assessment

This study investigated the impact of high-speed videoendoscopy (HSV) frame rates on the assessment of nine clinically relevant vocal-fold vibratory features. Fourteen adult patients with voice disorder and 14 adult normal controls were recorded using monochromatic rigid HSV at a rate of 16,000 frames per second (fps) and spatial resolution of 639×639 pixels. The 16,000-fps data were downsampled to 16 other rate denominations. Using paired comparisons design, nine common clinical vibratory features were visually compared between the downsampled and the original images. Three raters reported the thresholds at which: (1) a detectable difference between the two videos was first noticed, and (2) differences between the two videos would result in a change of clinical rating. Results indicated that glottal edge, mucosal wave magnitude and extent, aperiodicity, contact and loss of contact of the vocal folds were the vibratory features most sensitive to frame rate. Of these vibratory features, the glottal edge was selected for further analysis, due to its higher rating reliability, universal prevalence and consistent definition. Rates of 8000fps were found to be free from visually perceivable feature degradation, and for rates of 5333fps, degradation was minimal. For rates of 4000fps and higher, clinical assessments of glottal edge were not affected. Rates of 2000fps changed the clinical ratings in over 16% of the samples, which could lead to inaccurate functional assessment.

A Design Approach to Creating Scalable Video Processing Systems beyond UHDTV1 on an FPGA

An unquenchable end-user thirst for enhanced video quality results in ever-scaling video frame size and frame rate requirements. As we move from UHDTV1 to UHDTV2 and 120 frames/sec to 300 frames/sec, inevitably the computational complexity of video processing systems required to consume, process, and deliver video content increases. The need for solutions to support combinations of frame sizes and rates, as well as future increments, emphasizes the need for system scalability. The computational complexity and scalability requirements pose exciting challenges for field-programmable gate array (FPGA) implementation of video processing pipelines. This paper presents implementation techniques and methodologies to overcome these challenges. We specifically concentrate on architectures whereby the input per-pixel video sample rate exceeds the system clock rate. Novel results include classifying pixel processing orders and presenting a component-based design approach for future-proofing video processing solutions against an ever-scaling computational complexity requirement. Resource and memory bandwidth requirements of such systems are also analyzed and trends are presented.

Analysis of 2-D Ultrasound Cardiac Strain Imaging Using Joint Probability Density Functions

Ultrasound frame rates play a key role for accurate cardiac deformation tracking. Insufficient frame rates lead to an increase in signal de-correlation artifacts resulting in erroneous displacement and strain estimation. Joint probability density distributions generated from estimated axial strain and its associated signal-to-noise ratio provide a useful approach to assess the minimum frame rate requirements. Previous reports have demonstrated that bi-modal distributions in the joint probability density indicate inaccurate strain estimation over a cardiac cycle. In this study, we utilize similar analysis to evaluate a 2-D multi-level displacement tracking and strain estimation algorithm for cardiac strain imaging. The effect of different frame rates, final kernel dimensions and a comparison of radio frequency and envelope based processing are evaluated using echo signals derived from a 3-D finite element cardiac model and five healthy volunteers. Cardiac simulation model analysis demonstrates that the minimum frame rates required to obtain accurate joint probability distributions for the signal-to-noise ratio and strain, for a final kernel dimension of 1 λ by 3 A-lines, was around 42 Hz for radio frequency signals. On the other hand, even a frame rate of 250 Hz with envelope signals did not replicate the ideal joint probability distribution. For the volunteer study, clinical data was acquired only at a 34 Hz frame rate, which appears to be sufficient for radio frequency analysis. We also show that an increase in the final kernel dimensions significantly affect the strain probability distribution and joint probability density function generated, with a smaller effect on the variation in the accumulated mean strain estimated over a cardiac cycle. Our results demonstrate that radio frequency frame rates currently achievable on clinical cardiac ultrasound systems are sufficient for accurate analysis of the strain probability distribution, when a multi-level 2-D algorithm and kernel dimensions on the order of 1 λ by 3 A-lines or smaller are utilized.

Area Efficient, High Speed EBCOT Architecture for Digital Cinema

Embedded block coding with optimised truncation (EBCOT) is a key algorithm in digital cinema (DC) distribution system. Though several high speed EBCOT architectures exist, all are not capable of meeting the DC specifications. To meet this challenge, the relationship between contents of a code block (CB) and context generation is studied. Our study reveals that it is difficult to predict number of contexts generated in a bit plane. Even the nature of number of contexts produced varies from CB to CB. In such a situation, it is difficult to ensure the frame rate requirement of DC. To avoid this uncertainty, a pass parallel, concurrent sample coding EBCOT architecture is proposed in this paper. It is capable of encoding one bit plane in 288 clock cycles under any circumstances. This design is prototyped on XC4VLX80-12 FPGA with multiple clock domains. After synthesizing, the bit plane coder (BPC) and MQ coder operate at 450 MHz and 123 MHz, respectively. In order to maintain synchronism among different clock domains, the BPC and MQ coder units are operated at 432 MHz and 108 MHz, respectively. This entails that the proposed design is capable of processing size 57 DC frames in a second.

Frame Rate Effects on Visual Discrimination of Landing Aircraft Deceleration: Implications for Virtual Tower Design and Speed Perception

In order to help determine the required visual frame rate for the design of remote/virtual airport towers, thirteen active air traffic controllers viewed high dynamic-fidelity simulations of landing aircraft and decided whether the aircraft would stop before the end of the runway, as if to be able to make a runway turnoff. The viewing conditions and simulation dynamics replicated visual rates and environments of transport aircraft landing at small commercial airports. Three frame rates were used: 6, 12, and 24 fps. The frame rate that would be needed to produce asymptotic performance was estimated from a model fit to perceptual discriminability (d’) of the condition in which the aircraft would stop. The required frame rate appears to range from 30-60 fps, but definitive recommendations require further testing at a higher rate in the range of 45-60 fps. Errors and reports of judgment certainty show performance was roughly steady state. Anecdotal reports of increased apparent speed due to low frame rates are objectively confirmed. Some implications for the perceptual design of a remote tower are briefly discussed.

A novel cross-layer strategy of video transmission to solve performance anomaly in multi-rate IEEE 802.11 networks

When wireless hosts use different rates to transmit data in IEEE 802.11 networks, it will take on the state of performance anomaly which will severely decrease the throughputs of all the higher rate hosts. Hence, it is bad for video service transmission. Considering that video is very sensitive to packet delivery delay but can tolerate some packet losses, we propose a novel cross-layer scheme which takes these two characteristics into consideration. Firstly, the maximum number of retransmissions for a video Medium Access Control (MAC) frame is computed in MAC layer according to video frame rate requirement of application layer and current access delay of MAC layer. Secondly, within the margin of the tolerant Packet Loss Rate (PLR) of application layer, several video MAC frames are allowed to drop so that we can adaptively select the transmission rate as high as possible for the rest of video MAC frames in terms of current channel quality and the maximum number of retransmissions. Experiment results show that the proposed method can reduce the delay and jitter of video service and improve the throughputs of fast hosts. Therefore, it increases the quality of reconstructed video to a certain extent and relieves the performance anomaly of network effectively.

Theoretical Analysis of Shear Wave Interference Patterns by Means of Dynamic Acoustic Radiation Forces

Acoustic radiation forces associated with high intensity focused ultrasound stimulate shear wave propagation allowing shear wave speed and shear viscosity estimation of tissue structures. As wave speeds are meters per second, real time displacement tracking over an extend field-of-view using ultrasound is problematic due to very high frame rate requirements. However, two spatially separated dynamic external sources can stimulate shear wave motion leading to shear wave interference patterns. Advantages are shear waves can be imaged at lower frame rates and local interference pattern spatial properties reflect tissue's viscoelastic properties. Here a theoretical analysis of shear wave interference patterns by means of dynamic acoustic radiation forces is detailed. Using a viscoelastic Green's function analysis, tissue motion due to a pair of focused ultrasound beams and associated radiation forces are presented. Overall, this paper theoretically demonstrates shear wave interference patterns can be stimulated using dynamic acoustic radiation forces and tracked using conventional ultrasound imaging.

Six Degrees-of-Freedom Haptic Interaction with Fluids

We often interact with fluids in our daily life, either through tools such as when holding a glass of water or directly with our body when we swim or we wash our hands. Multimodal interactions with virtual fluids would greatly improve the simulations realism, particularly through haptic interaction. However, achieving realistic, stable, and real-time force feedback from fluids is particularly challenging. In this work, we propose a novel approach that allows real-time six Degrees of Freedom (DoF) haptic interaction with fluids of variable viscosity. Our haptic rendering technique, based on a Smoothed-Particle Hydrodynamics physical model, provides a realistic haptic feedback through physically based forces. 6DoF haptic interaction with fluids is made possible thanks to a new coupling scheme and a unified particle model, allowing the use of arbitrary-shaped rigid bodies. Particularly, fluid containers can be created to hold fluid and hence transmit to the user force feedback coming from fluid stirring, pouring, shaking, and scooping, to name a few. Moreover, we adapted an existing visual rendering algorithm to meet the frame rate requirements of the haptic algorithms. We evaluate and illustrate the main features of our approach through different scenarios, highlighting the 6DoF haptic feedback and the use of containers.

Modal Fourier Wavefront Reconstruction Using FPGA Technology

Atmospheric turbulence introduces optical aberration into wavefronts arriving at ground-based telescopes. The wavefront reconstruction using fast Fourier transforms (FFT) and spatial filtering is computationally tractable and sufficiently accurate in large Shack-Hartmann-based adaptive optics systems (up to 10,000 actuators). A first design has been developed using a Graphical Processing Units (GPU) platform. However, an increase in telescope size requires significant computational power. For this reason other hardware technologies must be taken into account during the development of a specific processor. In this paper, an improvement on the efficiency of the above methods is proposed based on Field Programmable Gate Array (FPGA) technology. To the best of our knowledge, there are not other wavefront reconstruction implementations based on FPGA technology. The basic advantages of this technology are its flexible architecture and extremely highperformance signal processing capability which are captured through parallelism. The implemented phase recoverer is, consequently, faster than the CPU or the GPU solution. Furthermore, the implemented design incorporating this technology meets current and future adaptive optics image processing frame rate requirements. Keywords: Adaptive optics, FPGA, Phase recovery, Fast fourier transform, DSP

Ultrasound frame rate requirements for cardiac elastography: Experimental and in vivo results

Cardiac elastography using radiofrequency echo signals can provide improved 2D strain information compared to B-mode image data, provided data are acquired at sufficient frame rates. In this paper, we evaluate ultrasound frame rate requirements for unbiased and robust estimation of tissue displacements and strain. Both tissue-mimicking phantoms under cyclic compressions at rates that mimic the contractions of the heart and in vivo results are presented. Sinusoidal compressions were applied to the phantom at frequencies ranging from 0.5 to 3.5 cycles/sec, with a maximum deformation of 5% of the phantom height. Local displacements and strains were estimated using both a two-step one-dimensional and hybrid two-dimensional cross-correlation method. Accuracy and repeatability of local strains were assessed as a function of the ultrasound frame rate based on signal-to-noise ratio values. The maximum signal-to-noise ratio obtained in a uniformly elastic phantom is 20 dB for both a 1.26 Hz and a 2 Hz compression frequency when the radiofrequency echo acquisition is at least 12 Hz and 20 Hz respectively. However, for compression frequencies of 2.8 Hz and 4 Hz the maximum signal-to-noise ratio obtained is around 16 dB even for a 40 Hz frame rate. Our results indicate that unbiased estimation of displacements and strain require ultrasound frame rates greater than ten times the compression frequency, although a frame rate of about two times the compression frequency is sufficient to estimate the compression frequency imparted to the tissue-mimicking phantom. In vivo results derived from short-axis views of the heart acquired from normal human volunteers also demonstrate this frame rate requirement for elastography.

Luminescence Properties and Morphology of ZnSe:Te Films

High-speed imaging applications, such as time-resolved X-ray diffraction, require detectors with high frame rates ranging from hundreds to thousands of frames per second. New high-resolution CCD imagers capable of operating at the required frame rates have been developed; however, the current X-ray to light converters are the performance limiting factor in such applications. Here we report on the development of a structured ZnSe:Te scintillator that promises to provide extraordinarily high scintillation efficiency, emission at 640 nm (ideally suited for CCD sensors), high density of 5.4 g/cm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> , and a fast decay time of ~3 mus to ~50 mus with no afterglow, which permits high speed imaging without the problem of ghosting due to persistence. Furthermore, the non-hygroscopic and non-toxic nature of the ZnSe:Te scintillator, along with its stability of response over a wide range of temperatures and extremely high levels of radiation, makes it an ideal material for radiation detection in general and for synchrotron applications in particular. At RMD, we have fabricated microcolumnar ZnSe:Te films measuring ~25 mum to 85 mum in thickness using co-evaporation of ZnSe and ZnTe on suitable substrates. These films show columnar structure with columns ranging from 0.2 mum to 5 mum in diameter. The scintillation light produced by the radiation interaction is channeled within the microcolumns by the mechanism of total internal reflection, thereby providing very high spatial resolution, even when films are made thick to achieve high X-ray absorption.

Frame rate requirement for tissue Doppler imaging in different phases of cardiac cycle: radial and longitudinal functions

Tissue Doppler imaging (TDI) has been suggested for quantitative analysis of regional myocardial function. Myocardial movement included different mechanical phases with different duration and tissue velocity profiles need to high sampling rate in the acquisition of tissue velocity imaging for phases with shorter duration. The aim of this study is determining of frame rate requirement for myocardial tissue velocity imaging for longitudinal and radial functions separately. Tissue velocity imaging recorded from 29 healthy volunteers by use of the apical and para-sternal views. Off-line analysis performed for extracting tissue velocity profiles of the myocardial longitudinal and radial functions. The frequency and subsequently the frame rate calculated separately for all LV segments during two consequent cardiac cycles. Segmental distribution of the time intervals measured in all cardiac phases and the minimum frame rate requirement calculated for each segment. We found significant differences between radial and longitudinal functions (P < 0.001) except early diastolic phases. The presented normal frame rate values for LV segments may useful for accurate studies of myocardial longitudinal and radial functions in different cardiac phases. We conclude that data sampling at a rate of at least 105 and 118 frames per second need for longitudinal and radial functions respectively.

FPGA-based Real-time Optical Flow Algorithm Design and Implementation

Optical flow algorithms are difficult to apply to robotic vision applications in practice because of their extremely high computational and frame rate requirements. In most cases, traditional general purpose processors and sequentially executed software cannot compute optical flow in real time. In this paper, a tensor-based optical flow algorithm is developed and implemented using field programmable gate array (FPGA) technology. The resulting algorithm is significantly more accurate than previously published FPGA results and was specifically developed to be implemented using a pipelined hardware structure. The design can process 640 × 480 images at 64 fps, which is fast enough for most real-time robot navigation applications. This design has low resource requirements, making it easier to fit into small embedded systems. Error analysis on a synthetic image sequence is given to show its effectiveness. The algorithm is also tested on a real image sequence to show its robustness and limitations. The resulting limitations are analyzed and an improved scheme is then proposed. It is then shown that the performance of the design could be substantially improved with sufficient hardware resources.

3.4: Improvement of Motion Image Quality by High Frame Rate

The required frame rate for the best motion image quality against blur and jerkiness is presented. To avoid those impairments, psychophysical evaluation had been applied. The result shows that 120fps performs good improvement comparing to 60fps while the best improvement where evaluation is saturated is found at around 240fps.

Evaluating the visual fidelity of physically based animations

For many systems that produce physically based animations, plausibility rather than accuracy is acceptable. We consider the problem of evaluating the visual quality of animations in which physical parameters have been distorted or degraded, either unavoidably due to real-time frame-rate requirements, or intentionally for aesthetic reasons. To date, no generic means of evaluating or predicting the fidelity, either physical or visual, of the dynamic events occurring in an animation exists. As a first step towards providing such a metric, we present a set of psychophysical experiments that established some thresholds for human sensitivity to dynamic anomalies, including angular, momentum and spatio-temporal distortions applied to simple animations depicting the elastic collision of two rigid objects. In addition to finding significant acceptance thresholds for these distortions under varying conditions, we identified some interesting biases that indicate non-symmetric responses to these distortions (e.g., expansion of the angle between post-collision trajectories was preferred to contraction and increases in velocity were preferred to decreases). Based on these results, we derived a set of probability functions that can be used to evaluate the visual fidelity of a physically based simulation. To illustrate how our results could be used, two simple case studies of simulation levels of detail and constrained dynamics are presented.