High Frame Research Articles

A 92 fps and 2.56 mJ/Frame Computing-In-Memory-Based Human Pose Estimation Accelerator With Resource-Efficient Macro for Mobile Devices

Human pose estimation (HPE) is a promising solution for accurately understanding the state and context of human actions in virtual reality (VR). A high frame rate with low-power HPE processing is required for a realistic user interaction experience in battery-limited mobile devices. The proposed HPE accelerator is a computing-in-memory (CIM) based accelerator that computes depth-wise separable convolution (DWSC) of a lightweight HPE network. Three key features contribute to a resource-efficient CIM accelerator: 1) Dual-mode CIM computes DWSC with a reconfigurable homogenous architecture, resulting in 2.68 times higher throughput than previous analog CIMs. 2) Effective layeraware unrolling performs bit-parallel computation on dualmode CIM with fewer ADC operations, achieving 46 times higher throughput than before. 3) Adaptive fused inter-macro balancing improves latency balance in layer fusion execution, leading to a 57.0 % higher frame rate than before. The proposed HPE accelerator is implemented in 28nm CMOS technology. It achieves higher computation resource utilization and operates HPE with a low energy-delay product of 27.6 uJ7s in mobile VR devices.

Deformation-encoding Deep Learning Transformer for High-Frame-Rate Cardiac Cine MRI.

Purpose To develop a deep learning model for increasing cardiac cine frame rate while maintaining spatial resolution and scan time. Materials and Methods A transformer-based model was trained and tested on a retrospective sample of cine images from 5840 patients (mean age, 55 years ± 19 [SD]; 3527 male patients) referred for clinical cardiac MRI from 2003 to 2021 at nine centers; images were acquired using 1.5- and 3-T scanners from three vendors. Data from three centers were used for training and testing (4:1 ratio). The remaining data were used for external testing. Cines with downsampled frame rates were restored using linear, bicubic, and model-based interpolation. The root mean square error between interpolated and original cine images was modeled using ordinary least squares regression. In a prospective study of 49 participants referred for clinical cardiac MRI (mean age, 56 years ± 13; 25 male participants) and 12 healthy participants (mean age, 51 years ± 16; eight male participants), the model was applied to cines acquired at 25 frames per second (fps), thereby doubling the frame rate, and these interpolated cines were compared with actual 50-fps cines. The preference of two readers based on perceived temporal smoothness and image quality was evaluated using a noninferiority margin of 10%. Results The model generated artifact-free interpolated images. Ordinary least squares regression analysis accounting for vendor and field strength showed lower error (P < .001) with model-based interpolation compared with linear and bicubic interpolation in internal and external test sets. The highest proportion of reader choices was "no preference" (84 of 122) between actual and interpolated 50-fps cines. The 90% CI for the difference between reader proportions favoring collected (15 of 122) and interpolated (23 of 122) high-frame-rate cines was -0.01 to 0.14, indicating noninferiority. Conclusion A transformer-based deep learning model increased cardiac cine frame rates while preserving both spatial resolution and scan time, resulting in images with quality comparable to that of images obtained at actual high frame rates. Keywords: Functional MRI, Heart, Cardiac, Deep Learning, High Frame Rate Supplemental material is available for this article. © RSNA, 2024.

A study of the latest updates of the readout system for the hybird-pixel detector at HEPS

The High Energy Photon Source (HEPS) represents a fourth-generation light source. This facility has made unprecedented advancements in accelerator technology, necessitating the development of new detectors to satisfy physical requirements such as single-photon resolution, large dynamic range, and high frame rates. Since 2016, the Institute of High Energy Physics has introduced the first user-experimental hybrid pixel detector, progressing to the fourth-generation million-pixel detector designed for challenging conditions, with the dual-threshold single-photon detector HEPS-Beijing PIXel (HEPS-BPIX) set as the next-generation target. HEPS-BPIX will employ the entirely new Application-Specific Integrated Circuit (ASIC) BP40 for pixel information readout. Data flow will be managed and controlled through readout electronics based on a two-tier Field-Programmable Gate Array (FPGA) system: the Front-End Electronics (FEE) and the Input-Output Board (IOB) handle the fan-out for 12 ASICs, and the μ4FCP is tasked with processing serial data on high-speed links, transferring pixel-level data to the back-end RTM and μTCA chassis, or independently outputting through a network port, enabling remote control of the entire detector. The new HEPS-BPIX firmware has undergone a comprehensive redesign and update to meet the electronic characteristics of the new chip and to improve the overall performance of the detector. We provide an overview of the core subunits of HEPS-BPIX, emphasizing the readout system, evaluating the new hardware and firmware, and highlighting some of its innovative features and characteristics.

47‐2: Invited Paper: Modulated Analog Driving and Evaluation of Image Quality

Consumer demand for higher resolution, higher frame rate, and deeper color displays is driving the data rates of the connected video systems to the limits of affordability. At current high‐end displays, like 4K gaming at 360 Hz frame rate or 8K TV's at &gt;120 Hz frame rate, conventional data transports require parallel data‐pairs plus equalization circuits to deliver the required video data. These accommodations increase cost, power and complexity.HYPHY's revolutionary ‘modulated analog' (also referred to as “sampled Analog”) video transport is the solution to this impasse, demonstrably delivering 10X more video payload per “wire‐hertz” than traditional methods, delivering equal or better image quality.To emphasize the advantages of this method, HYPHY has developed an image and video quality measurement method that demonstrates the image quality of Modulated Analog to be at least on par with traditional display driving.

80‐3: Distinguished Paper: 47.5 inch 8K Inkjet Printing AMOLED MNT with Local Boosting GOA Design

In this paper, we presented a 47.5 inch 8K2K AMOLED MNT employing a LTPS backplane, top emissive inkjet printing (IJP) OLED devices and gate driver on array (GOA) supporting independent 32*9 zone control and internal compensation driving. It is so far the first prototype AMOLED MNT integrating inkjet printing process and local high frame rate (up to 960Hz) driving. Thanks to the high resolution, local frame rate data processing and superior IJP OLED EL, it shows a brand new approach to deliver a premium display.

62‐2: High Frame Rate Scanning Backlight System for Privacy Display with Active Retarder

The demand for higher frame rates and resolutions on display panels remains strong as the gaming market has expanded in recent years. To make new use of these display panels, we have developed a privacy display in combination with active retarder. We made a prototype using high frame rate scanning backlight system and achieved the 4% level for the crosstalk ratio.

P‐32: Dual Gate Amorphous Silicon Thin Film Transistor Technology for High Brightness and High Frame Rate Outdoor Display Panels

In this paper, we present the development of a dual‐gate (DG) architecture for hydrogenated amorphous silicon thin film transistors (a‐Si:H TFTs). The aim is to address the challenges of low on‐state current (Ion) and light‐induced degradation in traditional single‐gate (SG) TFTs. The experimental results demonstrate that the DG‐TFT exhibits significantly higher Ion compared to the SG‐TFT, and the improvement depends on the thickness of the top gate insulator. Additionally, the threshold voltage (Vth) can be adjusted by applying the top gate voltage, allowing for a full‐ swing modulation coefficient of approximately 0.2. To ensure long‐term stability, a light‐blocking layer has been incorporated as the top gate electrode, enabling the DG‐ TFT to maintain its initial properties even under intense illumination conditions of up to 28,000 nits.

Pixel-wise programmability enables dynamic high-SNR cameras for high-speed microscopy

High-speed wide-field fluorescence microscopy has the potential to capture biological processes with exceptional spatiotemporal resolution. However, conventional cameras suffer from low signal-to-noise ratio at high frame rates, limiting their ability to detect faint fluorescent events. Here, we introduce an image sensor where each pixel has individually programmable sampling speed and phase, so that pixels can be arranged to simultaneously sample at high speed with a high signal-to-noise ratio. In high-speed voltage imaging experiments, our image sensor significantly increases the output signal-to-noise ratio compared to a low-noise scientific CMOS camera (~2–3 folds). This signal-to-noise ratio gain enables the detection of weak neuronal action potentials and subthreshold activities missed by the standard scientific CMOS cameras. Our camera with flexible pixel exposure configurations offers versatile sampling strategies to improve signal quality in various experimental conditions.

Development and Characterization of Hemoglobin Microbubbles for Acoustic Blood Oxygen Level Dependent Imaging.

Oxygen levels in tissues and organs are crucial for their normal functioning, and approaches to monitor them non-invasively have wide biological and clinical applications. In this study, we developed a method of acoustically detecting oxygenation using contrast-enhanced ultrasound (CEUS) imaging. Our approach involved the use of specially designed hemoglobin-based microbubbles (HbMBs) that reversibly bind to oxygen and alter the state-dependent acoustic response. We confirmed that the bioactivity of hemoglobin remained intact after the microbubble shell was formed, and we did not observe any significant loss of heme. We conducted passive cavitation detection (PCD) experiments to confirm whether the acoustic properties of HbMBs vary based on the level of oxygen present. The experiments involved driving the HbMBs with a 1.1 MHz focused ultrasound transducer. Through the PCD data collected, we observed significant differences in the subharmonic and harmonic responses of the HbMBs when exposed to an oxygen-rich environment versus an oxygen-depleted one. We used a programmable ultrasound system to capture high-frame rate B mode videos of HbMBs in both oxy and deoxy conditions at the same time in a two-chambered flow phantom and observed that the mean pixel intensity of deoxygenated HbMB was greater than in the oxygenated state using B-mode imaging. Finally, we demonstrated that HbMBs can circulate in vivo and are detectable by a clinical ultrasound scanner. To summarize, our results indicate that CEUS imaging with HbMB has the potential to detect changes in tissue oxygenation and could be a valuable tool for clinical purposes in monitoring regional blood oxygen levels.

Non-systematic noise reduction framework for ToF camera

Time-of-flight (ToF) cameras enable a diverse range of applications due to their high frame rate, high resolution, and low cost. However, these cameras suffer from non-systematic noise during the acquisition of high-quality depth images, severely affecting their range accuracy. In this paper, we propose a non-systematic noise reduction framework named “DCS2Noise” to address this issue. This framework comprises a three-stage denoising strategy, involving noise standardization, deep learning-based differential correlation sampling (DCS) denoising and further enhancement with pairwise noise suppression. This framework directly captures and denoises DCS images during the ToF imaging process, making it more suitable for non-systematic noise reduction in ToF cameras. Compared to traditional methods, our approach significantly reduces the root mean squared error (RMSE) and improves the noise reduction ratio, peak signal to noise ratio (PSNR), and structural similarity index measure (SSIM). We believe that this study provides new insights into understanding noise in ToF cameras and offers effective references for reducing non-systematic noise in three-dimensional measuring instruments.

Predictive value of Cmmi-MHR combined with thromboelastography parameters in acute cerebral infarction.

Cerebral infarction is a common neurological disease with high rates of morbidity, mortality, and recurrence, posing a great threat to human life and health. Cerebral infarction is the second leading cause of death in the world and the leading cause of long-term disability in humans. The results of the third national retrospective sampling survey on causes of death in 2008 showed that cerebral infarction has become the leading cause of death in China and its mortality rate is 4-5 times that of European and American countries. Therefore, this article proposed a study on the predictive value of Cmmi-MHR combined with thromboelastography parameters that was performed for acute cerebral infarction. This paper mainly proposed a high frame rate imaging technology and analyzed its algorithm. In this article, in the experimental part, an in-depth analysis of the predictive value of the Monocyte-to-high-density lipoprotein cholesterol ratio (MHR) combined with thromboelastography parameters was performed for acute cerebral infarction. The final experimental results showed that HDL (OR = 1.695%, P-trend = 0.049) had a probability of death within 90days of hospitalization (OR = 0.81, 95% CI = 1.06-3.11, P-trend = 0.523). There were no significant differences in mortality rate after 90days. Regardless of adjusting for confounders such as age, gender, and NIHSS score, there was no significant difference in the risk of MHR or monocyte count within 90days of hospitalization. The conclusion indicates that the combination of Cmmi-MHR and thromboelastography parameters provides a new perspective and method for the diagnosis and treatment of cerebral infarction, and provides important support for personalized treatment and management of cerebral infarction.

A two‐dimensional, reach‐scale implementation of space‐time image velocimetry (STIV) and comparison to particle image velocimetry (PIV)

AbstractImage‐based algorithms have become a powerful tool for estimating flow velocities in rivers. In this study, we generalize the space‐time image velocimetry (STIV) framework for reach‐scale application rather than along a cross section. The new algorithm provides information on both the magnitude and orientation of velocity vectors, and we refer to the algorithm as two‐dimensional STIV, or 2D‐STIV. The workflow involves setting up a grid, using centreline tangent vectors as initial estimates of flow direction, and then extracting space‐time images (STIs) along search lines radiating from each grid node. The autocorrelation function is used to infer the inclination of streak lines present in STIs, which represents the advection of water surface features. Information on flow direction is obtained by evaluating various candidate search lines and identifying that which yields the highest velocity. This search can be performed exhaustively or via optimization. We applied the new 2D‐STIV algorithm to three test cases, one simulated data set and two natural channels, and compared image‐derived velocities to modelled or measured values. We also applied two established particle image velocimetry (PIV) algorithms to the same data sets. 2D‐STIV performed as well as the two PIV algorithms for simulated images. For a natural river with distinct water surface features, 2D‐STIV was effective for much of the channel but also led to a more patchy, irregular velocity field than the two PIV algorithms. For a site lacking obvious surface features, exhaustive 2D‐STIV led to velocity estimates uncorrelated with field data while the optimization‐based version produced erratic flow directions. 2D‐STIV also required greater image sequence durations, higher frame rates, and generally longer computational run times. Overall, ensemble PIV was the most reliable algorithm.

Perceptions of Hybrid Lighting for Virtual Reality

Virtual reality (VR) applications are computationally demanding due to high frame rate requirements, which precludes large numbers of realtime lights from being used. In this paper, we explore a hybrid lighting approach that combines the benefits of realtime and baked lights based on the importance of each source. First, we demonstrate that the hybrid approach affords better frames per second than realtime and mixed lighting. We then present the results of an online paired-comparison study, in which participants (n = 60) compared videos of the four lighting conditions (baked, mixed, realtime, and hybrid) in terms of preference. Our results indicate that the hybrid lighting approach is better than realtime lighting in terms of graphical performance and also yields better perceptions of quality for scenes with a small to moderately large number of lights.

Ultrasound vector mapping of common femoral artery blood flow in normal and early-stage stenosis

It is assumed that the wall shear stress (WSS), which determines the function of the endothelium, is constant along the arterial bed. The assessment of turbulence, blood flow velocity in the arterial system in healthy (13 patients) and in patients with the initial form of atherosclerosis in the femoral artery (42 patients) was discussed. The study quantified blood flow in the common femoral artery using V Flow with visualization of blood flow with a high frame rate. The results obtained in the femoral arteries were evaluated by the wall shear rate, velocity profile and oscillation index (OSI). It was shown that the average value of WSSmean in the femoral artery in healthy and in patients with stenosis &lt;30–35% is 0.9± 0.4 – 0.91±0.4 Pa and does not significantly differ. The wall thickness in the common femoral artery in patients with the initial form of atherosclerosis was 0.9–1.1 mm, and in healthy patients 0.8–0.9 mm. The correlation between the parameters was evaluated by nonparametric analysis of Kendall’s Taub. It was revealed that there is no correlation between WSSmean and blood flow velocity (Vs) in both healthy and patients with the initial form of atherosclerosis.

Radar-optical fusion detection of UAV based on improved YOLOv7-tiny

This study presents a radar-optical fusion detection method for unmanned aerial vehicles (UAVs) in maritime environments. Radar and camera technologies are integrated to improve the detection capabilities of the platforms. The proposed method involves generating regions of interest (ROI) by projecting radar traces onto optical images through matrix transformation and geometric centroid registration. The generated ROI are matched with YOLO detection boxes using the intersection-over-union (IoU) algorithm, enabling radar-optical fusion detection. A modified algorithm, called SPN-YOLOv7-tiny, is developed to address the challenge of detecting small UAV targets that are easily missed in images. In this algorithm, the convolutional layers in the backbone network are replaced with a space-to-depth convolution, and a small object detection layer is added. In addition, the loss function was replaced with a normalized weighted distance loss function. Experimental results demonstrate that compared to the original YOLOv7-tiny method, SPN-YOLOv7-tiny achieves an improved mAP@0.5 (mean average precision at an IoU threshold of 0.5) from 0.852 to 0.93, while maintaining a high frame rate of 135.1 frames per second. Moreover, the proposed radar-optical fusion detection method achieves an accuracy of 96.98%, surpassing the individual detection results of the radar and camera. The proposed method effectively addresses the detection challenges posed by closely spaced overlapping targets on a radar chart.

Time Efficient Ultrasound Localization Microscopy Based on A Novel Radial Basis Function 2D Interpolation.

Ultrasound localization microscopy (ULM) allows for the generation of super-resolved (SR) images of the vasculature by precisely localizing intravenously injected microbubbles. Although SR images may be useful for diagnosing and treating patients, their use in the clinical context is limited by the need for prolonged acquisition times and high frame rates. The primary goal of our study is to relax the requirement of high frame rates to obtain SR images. To this end, we propose a new time-efficient ULM (TEULM) pipeline built on a cutting-edge interpolation method. More specifically, we suggest employing Radial Basis Functions (RBFs) as interpolators to estimate the missing values in the 2-dimensional (2D) spatio-temporal structures. To evaluate this strategy, we first mimic the data acquisition at a reduced frame rate by applying a down-sampling (DS = 2, 4, 8, and 10) factor to high frame rate ULM data. Then, we up-sample the data to the original frame rate using the suggested interpolation to reconstruct the missing frames. Finally, using both the original high frame rate data and the interpolated one, we reconstruct SR images using the ULM framework steps. We evaluate the proposed TEULM using four in vivo datasets, a Rat brain (dataset A), a Rat kidney (dataset B), a Rat tumor (dataset C) and a Rat brain bolus (dataset D), interpolating at the in-phase and quadrature (IQ) level. Results demonstrate the effectiveness of TEULM in recovering vascular structures, even at a DS rate of 10 (corresponding to a frame rate of sub-100Hz). In conclusion, the proposed technique is successful in reconstructing accurate SR images while requiring frame rates of one order of magnitude lower than standard ULM.

Swift-Eye: Towards Anti-blink Pupil Tracking for Precise and Robust High-Frequency Near-Eye Movement Analysis with Event Cameras.

Eye tracking has shown great promise in many scientific fields and daily applications, ranging from the early detection of mental health disorders to foveated rendering in virtual reality (VR). These applications all call for a robust system for high-frequency near-eye movement sensing and analysis in high precision, which cannot be guaranteed by the existing eye tracking solutions with CCD/CMOS cameras. To bridge the gap, in this paper, we propose Swift-Eye, an offline precise and robust pupil estimation and tracking framework to support high-frequency near-eye movement analysis, especially when the pupil region is partially occluded. Swift-Eye is built upon the emerging event cameras to capture the high-speed movement of eyes in high temporal resolution. Then, a series of bespoke components are designed to generate high-quality near-eye movement video at a high frame rate over kilohertz and deal with the occlusion over the pupil caused by involuntary eye blinks. According to our extensive evaluations on EV-Eye, a large-scale public dataset for eye tracking using event cameras, Swift-Eye shows high robustness against significant occlusion. It can improve the IoU and F1-score of the pupil estimation by 20% and 12.5% respectively, compared with the second-best competing approach, when over 80% of the pupil region is occluded by the eyelid. Lastly, it provides continuous and smooth traces of pupils in extremely high temporal resolution and can support high-frequency eye movement analysis and a number of potential applications, such as mental health diagnosis, behaviour-brain association, etc. The implementation details and source codes can be found at https://github.com/ztysdu/Swift-Eye.

Efficient Radius Search for Adaptive Foveal Sizing Mechanism in Collaborative Foveated Rendering Framework

Collaborative Foveated Rendering (CFR) is the latest collaborative rendering framework proposed to enable high frame rate VR applications on mobile devices. Compared with the strategies adopted in conventional collaborative rendering, the pixel-based Adaptive Foveal Sizing (AFS) mechanism in CFR offers a more flexible and intelligent workload trade-off by predicting the radius. However, the performance of the AFS mechanism in actual deployment depends on its adaptability to two factors, including the Sudden Environmental Variations (SEV) and the Random Discrete Latency (RDL). Guaranteeing the performance of the AFS mechanism by adapting to these two factors is of great significance to guaranteeing users' immersive experience. <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">This paper identifies the existence of the SEV and RDL phenomenon in the AFS mechanism for the first time, and contributes the first method that offers the effective and real-time AFS mechanism implementation for the practical deployment, namely the Efficient Radius Search (ERS).</b> The ERS method efficiently searches the largest radius online that controls the rendering workload within the foveated layer just below the offline baked threshold, thereby achieving the immediate response to SEV and reducing the oscillating frame rendering latency led by RDL. <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Through the experiments on 3 VR applications and 4 mobile devices, the resulting 2.44× to 9.07× higher frame rate precision compared with the state-of-the-art method demonstrate the superiority of the ERS method.</b>

47.5‐inch 8K inkjet printing AMOLED monitor with local boosting gate driver on array design

AbstractIn this paper, we presented a 47.5 in. 8K2K AMOLED MNT employing a low temperature poly‐silicon (LTPS) backplane, top emissive inkjet printing (IJP) OLED devices, and gate driver on array (GOA) supporting independent 32 * 9 zone control and internal compensation driving. By combining printed RGB units with a common evaporative blue stack, we proposed the hybrid tandem IJP devices that improved both device efficiency and operation load balance. In the local boosting GOA design, a novel architecture for independent display partition control was developed. The GOA cascading and timing scheme was carefully considered by reusing several clock signals to support both TFT internal compensation and display zone control. It is so far the first prototype AMOLED MNT integrating inkjet printing process and local high frame rate (up to 960 Hz) driving. Thanks to the 8K2K high resolution, local frame rate data processing, and IJP OLED EL reaching HDR brightness up to 580 nits with superior optical performance, it shows a brand‐new approach to deliver a premium display.

Research on the Deblurring Algorithm for High Frame Rate Cameras

With the rapid development of technology, high frame rate cameras have been widely used in various fields. However, due to the characteristics of high frame rate cameras, the corresponding deblur algorithm has become the focus of this research. This paper aims to propose an efficient and accurate idea to improve the imaging quality and application performance of high frame rate cameras. However, the current idea still has some limitations, and it is hoped that future studies can realize the algorithm optimization and improve the practicality and performance of the algorithm to provide better support for the application of high frame rate cameras.