High Frame Rate Camera Research Articles

Research on the Deblurring Algorithm for High Frame Rate Cameras

With the rapid development of technology, high frame rate cameras have been widely used in various fields. However, due to the characteristics of high frame rate cameras, the corresponding deblur algorithm has become the focus of this research. This paper aims to propose an efficient and accurate idea to improve the imaging quality and application performance of high frame rate cameras. However, the current idea still has some limitations, and it is hoped that future studies can realize the algorithm optimization and improve the practicality and performance of the algorithm to provide better support for the application of high frame rate cameras.

Video-based analysis of the blink reflex in Parkinson’s disease patients

We developed a video-based tool to quantitatively assess the Glabellar Tap Reflex (GTR) in patients with idiopathic Parkinson’s disease (iPD) as well as healthy age-matched participants. We also video-graphically assessed the effect of dopaminergic medication on the GTR in iPD patients, as well as the frequency and blinking duration of reflex and non-reflex blinks. The Glabellar Tap Reflex is a clinical sign seen in patients e.g. suffering from iPD. Reliable tools to quantify this sign are lacking. Methods: We recorded the GTR in 11 iPD patients and 12 healthy controls (HC) with a consumer-grade camera at a framerate of at least 180 images/s. In these videos, reflex and non-reflex blinks were analyzed for blink count and blinking duration in an automated fashion. Results: With our setup, the GTR can be extracted from high-framerate cameras using landmarks of the MediaPipe face algorithm. iPD patients did not habituate to the GTR; dopaminergic medication did not alter that response. iPD patients’ non-reflex blinks were higher in frequency and higher in blinking duration (width at half prominence); dopaminergic medication decreased the median frequency (Before medication—HC: p < 0.001, After medication—HC: p = 0.0026) and decreased the median blinking duration (Before medication—HC: p = 0.8594, After medication—HC: p = 0.6943)—both in the direction of HC. Conclusion: We developed a quantitative, video-based tool to assess the GTR and other blinking-specific parameters in HC and iPD patients. Further studies could compare the video data to electromyogram (EMG) data for accuracy and comparability, as well as evaluate the specificity of the GTR in patients with other neurodegenerative disorders, in whom the GTR can also be present. Significance: The video-based detection of the blinking parameters allows for unobtrusive measurement in patients, a safer and more comfortable option.

A customized drone for ocean and atmospheric measurements and its performances

Drones- Unmanned Aerial vehicles (UAVs)- have become more significant across the world for a wide range of commercial and terrestrial defense applications in recent years. The National Institute of Ocean Technology (NIOT)- Ministry of Earth Sciences (MOES)- is exploring UAVs with a specific focus on maritime applications. NIOT has customized a heavy lift category drone, operable in marine environments and capable of withstanding coastal wind conditions of up to 40 kmph. It can lift and carry an instrumentation payload of 10 kg, conduct ocean data collection, and even take seawater samplings. The 10 kg payload of the UAV might hold a Conductivity Temperature Depth (CTD) sensor, a programmable sea water sampler, and a multi-parameter sensor (MPS) for ocean data collections, and a Light Detection and Ranging (LiDAR) device integrated with a high frame rate camera system for coastal mapping and digital elevation model (DEM) developmental applications. The customized hexacopter UAV is capable of withstanding winds of up to 10 m/s and features a waterproof IPX7 thruster with a maximum thrust of 153 N per axis. The UAV system is also interfaced with a Global Positioning System (GPS), a barometric pressure sensor, a compass, a highly accurate gyroscope, a 15 MP surveillance camera, and an accelerometer sensor connected to a reliable cube orange flight controller module with a redundant 32-bit controller through a serial peripheral interface (SPI). The drone structure and frame is composed of carbon fiber composites to provide an excellent weight-to-strength ratio. This paper presents the outcomes of an initial field test, carried out to ensure the drone’s suitability for various marine applications with intended payloads ranging from 5 - 10 kg weight, including coastal demonstrations and ocean data collections performed at Nellore (Andra Pradesh) and Chennai (Tamil Nadu) coastal waters. Highlights A drone is customised to with stand elevated wind conditions of coastal regions Design is fully flexible meeting the ease of transport from one location to other Highly stable mechanical design to sustain winds of up to 45 KMPH Redundant flight control mechanism is evolved to assure fail safe operations of drone Drone built in with redundant GPS module to avoid scenario like NO Geo-reference at any time Design is type certified and UIN registered with DGCA - India

IN SITU MELT TEMPERATURE ANALYSIS DURING HIGH PRESSURE GAS ATOMIZATION OF LIQUID METALS

Imaging of the melt plume during high pressure gas atomization (HPGA) using consumer digital single lens reflex (DSLR) equipment provides useful information about the process. Color imaging and high spatial resolution can be a useful adjunct to the more widely reported imaging using specialist high frame rate cameras. With knowledge of the camera's color response curves, the ratio of the signals in the red, green, and blue channels can be used to make spatially resolved temperature estimates of the material within the melt plume. Moreover, by combining these temperature estimates, which depend only upon intensity ratios, with the actual intensity of the optical signal, we propose that it is possible to obtain estimates of the relative surface area of the melt within the plume. This in turn can be related to the local melt fragmentation rate within the atomization plume.

Deployment and assessment of a deep learning model for real-time detection of anal precancer with high frame rate high-resolution microendoscopy

Anal cancer incidence is significantly higher in people living with HIV as HIV increases the oncogenic potential of human papillomavirus. The incidence of anal cancer in the United States has recently increased, with diagnosis and treatment hampered by high loss-to-follow-up rates. Novel methods for the automated, real-time diagnosis of AIN 2+ could enable "see and treat" strategies, reducing loss-to-follow-up rates. A previous retrospective study demonstrated that the accuracy of a high-resolution microendoscope (HRME) coupled with a deep learning model was comparable to expert clinical impression for diagnosis of AIN 2+ (sensitivity 0.92 [P = 0.68] and specificity 0.60 [P = 0.48]). However, motion artifacts and noise led to many images failing quality control (17%). Here, we present a high frame rate HRME (HF-HRME) with improved image quality, deployed in the clinic alongside a deep learning model and evaluated prospectively for detection of AIN 2+ in real-time. The HF-HRME reduced the fraction of images failing quality control to 4.6% by employing a high frame rate camera that enhances contrast and limits motion artifacts. The HF-HRME outperformed the previous HRME (P < 0.001) and clinical impression (P < 0.0001) in the detection of histopathologically confirmed AIN 2+ with a sensitivity of 0.91 and specificity of 0.87.

The fluid-structure interaction response of composite auxetic re-entrant honeycomb structures to underwater impact

In this paper, the effects of the gradient form, peak pressure and decaying time on the fluid-structure interaction of composite auxetic re-entrant honeycomb structures was experimentally and numerically investigated. The specimens for this experiment were manufactured by hot-press molding method. The acrylic transparent tube was utilized to accommodate water medium, pistons and specimen during experiment. The exponential decaying shock wave was simulated by accelerating the fly plate to impact the piston based on the one-stage light gas gun system. The incident velocity of the fly plate and process of the fluid-structure interaction response were monitored by high frame rate camera. In addition, the velocity of the shock wave in water was measured by transducers installed at different measuring points. The voltage signal of the dynamic overpressure collected by transducers was amplified by the charge amplifier and recorded by the oscilloscope. On the other hand, finite element software Abaqus/Explicit was applied to simulate the cavitation process of water domain and dynamic response of specimens. The experimental results and numerical results were in good agreement. The results indicated that the increase of the thickness of the fly plate had a certain effect on the propagation velocity of the shock wave, and the increase of the peak pressure would make the cavitation generation position close to the fluid-structure interface and make the cavitation duration longer. And the average structure which has a minimum transformation of 8.12 mm shows better underwater impact resistance than the gradient structure.

Rescaling invariance and anomalous energy transport in a small vertical column of grains.

It is well known that energy dissipation and finite size can deeply affect the dynamics of granular matter, often making usual hydrodynamic approaches problematic. Here we report on the experimental investigation of a small model system, made of ten beads constrained into a 1D geometry by a narrow vertical pipe and shaken at the base by a piston excited by a periodic wave. Recording the beads motion with a high frame rate camera allows to investigate in detail the microscopic dynamics and test hydrodynamic and kinetic models. Varying the energy, we explore different regimes from fully fluidized to the edge of condensation, observing good hydrodynamic behavior down to the edge of fluidization, despite the small system size. Density and temperature fields for different system energies can be collapsed by suitable space and time rescaling, and the expected constitutive equationholds very well when the particle diameter is considered. At the same time, the balance between dissipated and fed energy is not well described by commonly adopted dependence due to the up-down symmetry breaking. Our observations, supported by the measured particle velocity distributions, show a different phenomenological temperature dependence, which yields equationsolutions in agreement with experimental results.

Investigation on the multimodal failure characteristics of cement mortar under uniaxial compression loading

The physical condition and durability of cement-based structures can easily decrease after years of operation due to exposure to severe scenarios associated with internal defects, aggressive usage, and continuous load changes. Therefore, accurately assessing such infrastructures is essential for ensuring safety and serviceability and preventing hazards. In this study, an experimental investigation of the multimodal failure characteristics of cement mortar with various sand contents at different uniaxial compressive loading rates was conducted. Specifically, the multimodal failure characteristics include three main categories: mechanical properties (e.g., compressive strength), one-dimensional signals (e.g., micro-vibration and electromagnetic radiation), and two-dimensional videos/images (e.g., crack propagation trajectory). A geophysical acquisition and a high frame rate camera were employed to simultaneously record the micro-vibration, electromagnetic radiation and crack propagation videos of the cement specimens in a uniaxial compression test. Based on the computer vision technique and the feature pyramid network (FPN), an automatic pixel-level crack segmentation method was proposed to extract cracks of each critical moment in the fracture video. Then, the fractal dimension and crack area were calculated to quantitatively depict the crack propagation trajectory. Finally, the stress drop, micro-vibration, electromagnetic radiation, and crack propagation trajectory characteristics were analysed and discussed. The experimental results indicate the following. (1) The FPN offers a practical way to identify cracks in structures of cement-based materials at the pixel-level. (2) The micro-vibration, electromagnetic radiation, and crack propagation trajectory exhibit good consistency. In particular, the area and fractal dimension of cracks positively correlate with the stress drop of cement under compression loading. (3) The peak and cumulative energy values of micro-vibration and electromagnetic radiation tend to increase as the sand content increases when subjected to the same loading rate.

SuperFast: 200× Video Frame Interpolation via Event Camera.

Traditional frame-based video frame interpolation (VFI) methods rely on the linear motion assumption and brightness invariance assumption, which may lead to fatal errors confronting the scenarios with high-speed motions. To tackle the above challenge, inspired by the advantages of event cameras on asynchronously recording brightness changes at each pixel, we propose a Fast-Slow joint synthesis framework for event-enhanced high-speed video frame interpolation, named SuperFast, in this paper, which can generate high frame rate (5000 FPS, 200× faster) video from the input low frame rate (25 FPS) video and the corresponding event stream. In our framework, the task is divided into two sub-tasks, i.e., video frame interpolation for the contents with and without high-speed motions, which are tackled by two corresponding branches, i.e., the fast synthesis pathway and the slow synthesis pathway. The fast synthesis pathway leverages a spiking neural network to encode the input event stream, and combines boundary frames to generate intermediate results through synthesis and refinement, targeting on contents with high-speed motions. The slow synthesis pathway stacks the two input boundary frames and the event stream to synthesize intermediate results, focusing on relatively slow-motion contents. Finally, a fusion module with a comparison loss is utilized to generate the final video frame interpolation results. We also build a hybrid visual acquisition system containing an event camera and a high frame rate camera, and collect the first 5000 FPS High-Speed Event-enhanced Video frame Interpolation (THU[Formula: see text]) dataset. To evaluate the performance of our proposed framework, we have conducted experiments on our THU[Formula: see text] dataset and the existing HS-ERGB dataset. Experimental results demonstrate that our proposed framework can achieve state-of-the-art 200× video frame interpolation performance under high-speed motion scenarios.

Discontinuous Yielding of Fe E420 under High Strain Rate Loading

The discontinuous yielding behaviour of Fe E420 steel was experimentally investigated through tensile tests conducted on laminated sheets. In order to assess the response under both quasi-static and dynamic loading conditions, uniaxial tensile tests were performed according to ASTM E8 standards, and dynamic tests were conducted on the Hopkinson bar using a specially designed flat specimen. Three nominal strain rates (2×10−4, 500, and 1000) s−1 were considered. The effect of heat treatment on the material was also considered. For this purpose, the proposed experimental campaign was carried out on both the as-is material and on one treated by water quenching. All dynamic tests were performed with the direct tensile split Hopkinson pressure bar and were recorded via a high frame rate camera. The post failure fracture surfaces were investigated using scanning electron microscopy technique.

Optimization of Computational Resources for Real-Time Product Quality Assessment Using Deep Learning and Multiple High Frame Rate Camera Sensors

Human eyes generally perform product defect inspection in Indonesian industrial production lines; resulting in low efficiency and a high margin of error due to eye tiredness. Automated quality assessment systems for mass production can utilize deep learning connected to cameras for more efficient defect detection. However, employing deep learning on multiple high frame rate cameras (HFRC) causes the need for much computation and decreases deep learning performance, especially in the real-time inspection of moving objects. This paper proposes optimizing computational resources for real-time product quality assessment on moving cylindrical shell objects using deep learning with multiple HFRC Sensors. Two application frameworks embedded with several deep learning models were compared and tested to produce robust and powerful applications to assess the quality of production results on rotating objects. Based on the experiment results using three HFRC Sensors, a web-based application with tensorflow.js framework outperformed desktop applications in computation. Moreover, MobileNet v1 delivers the highest performance compared to other models. This result reveals an opportunity for a web-based application as a lightweight framework for quality assessment using multiple HFRC and deep learning.

In-Motion Railroad Tie Deflection Measurement via Ultrasonic Airborne Sonar and Computer Vision Techniques

ABSTRACT It is known in the railroad maintenance engineering community that the deflection of railroad ties is an indicator of the quality of the tie–ballast interface, whose deterioration may cause dangerous train derailments. A new technology is proposed to reconstruct the full-field deflection profile of railroad ties in-motion by means of non-contact ultrasonic testing and computer vision techniques. The sensing layout consists of an array of air-coupled capacitive transducers (operated in pulse-echo sonar-based ranging mode) and a high frame-rate camera, rigidly connected to the main frame of a moving train car. The acquisition system is programmed such that the synchronized waveforms and images are collected and saved as train car moves. In the processing stage, a supervised machine learning-based image classification approach is developed to demarcate the tie boundaries. For this purpose, the Speeded-Up Robust Features (SURF) and Bag of Visual Words (BOVW) algorithms are employed to encode images into condensed feature vectors, which are subsequently fed into the Support Vector Machine (SVM) to train a classifier. The relative deflections of the identified ties are eventually computed by tracking the time-of-flight of the reflected waves from the surfaces flagged as tie. An image processing technique is also developed to estimate the spatial resolution of the tracking system, required to reconstruct the full-field deflection profile of the scanned ties. The importance of such a technique is stressed if the test run is performed without any dedicated positioning system. The proposed ‘tie sonar’ system was prototyped and used to reconstruct the deflection profile of the ties scanned during a series of test runs conducted at slow (walking) speed at the Rail Defect Testing Facility (RDTF) of UC San Diego as well as a BNSF yard in San Diego, CA, with a realistic train load. Further developments of this system should include a performance evaluation at higher speeds (e.g., revenue speed).

Abnormal Phenomena and Mathematical Model of Fluid Imbibition in a Capillary Tube

At present, the imbibition behavior in tight rocks has been attracted increasing attention since spontaneous imbibition plays an important role in unconventional oil and gas development, such as increasing swept area and enhancing recovery rate. However, it is difficult to describe the imbibition behavior through imbibition experiment using tight rock core. To characterize the imbibition behavior, imbibition and drainage experiments were conducted among water, oil, and gas phases in a visible circular capillary tube. The whole imbibition process is monitored using a microfluidic platform equipped with a high frame rate camera. This study conducts two main imbibition experiments, namely liquid-displacing-air and water-displacing-oil experiments. The latter is a spontaneous imbibition that the lower-viscosity liquid displaces the higher-viscosity liquid. For the latter, the tendency of imbibition rate with time does not match with previous model. The experimental results indicate that it is unreasonable to take no account of the effect of accumulated liquid flowing out of the capillary tube on imbibition, especially in the imbibition experiments where the lower-viscosity liquid displaces the higher-viscosity liquid. A mathematical model is established by introducing an additional force to describe the imbibition behavior in capillary tube, and the model shows a good prediction effect on the tendency of imbibition rate with time. This study discovers and analyzes the effect of additional force on imbibition, and the analysis has significances to understand the imbibition behavior in tight rocks.

Fast and pure phase-shifting off-axis holographic microscopy with a digital micromirror device.

We present a phase-shifting digital holographic microscopy technique, where a digital micromirror device enables to perform a precise phase-only shift of the reference wave. By coupling the beam into a monomode fiber, we obtain a laser mode with a constant phase shift, equally acting on all pixels of the hologram. This method has the advantage of being relatively simple and compatible with high frame rate cameras, which makes it of great interest for the observation of fast phenomena. We demonstrate the validity of the technique in an off-axis configuration by imaging living paramecia caudata.

The Thresher: Lucky imaging without the waste

ABSTRACT In traditional lucky imaging (TLI), many consecutive images of the same scene are taken with a high frame-rate camera, and all but the sharpest images are discarded before constructing the final shift-and-add image. Here, we present an alternative image analysis pipeline – The Thresher – for these kinds of data, based on online multi-frame blind deconvolution. It makes use of all available data to obtain the best estimate of the astronomical scene in the context of reasonable computational limits; it does not require prior estimates of the point-spread functions in the images, or knowledge of point sources in the scene that could provide such estimates. Most importantly, the scene it aims to return is the optimum of a justified scalar objective based on the likelihood function. Because it uses the full set of images in the stack, The Thresher outperforms TLI in signal-to-noise ratio; as it accounts for the individual-frame PSFs, it does this without loss of angular resolution. We demonstrate the effectiveness of our algorithm on both simulated data and real Electron-Multiplying CCD images obtained at the Danish 1.54-m telescope (hosted by ESO, La Silla). We also explore the current limitations of the algorithm, and find that for the choice of image model presented here, non-linearities in flux are introduced into the returned scene. Ongoing development of the software can be viewed at https://github.com/jah1994/TheThresher.

Utilizing Pencil Beam Scan Dynamics and a Scintillation Screen to produce 3D Dose Distribution of Proton Beams

This study describes a method for producing volumetric dose profiles of proton beams from 2D slices of optical scintillation images. The method relies on a high frame rate camera acquisition (100 frames per second), the spot scanning capabilities of current proton pencil beam scanning systems, and a water equivalent scintillation screen. The acquired slices are corrected for optical blurring and ionization quenching and stacked to produce volumetric dose distribution. The volumetric optical dose profile had a pass rate of 98.3% for 2%/2mm local gamma analysis, suggesting the method can accurately measure dose profiles. The method can potentially image all clinical proton beams for pencil beam scanning systems and can extend to imaging patient plans, with further verification.

Visual-Feedback-Based Frame-by-Frame Synchronization for 3000 fps Projector–Camera Visual Light Communication

This paper proposes a novel method for synchronizing a high frame-rate (HFR) camera with an HFR projector, using a visual feedback-based synchronization algorithm for streaming video sequences in real time on a visible-light communication (VLC)-based system. The frame rates of the camera and projector are equal, and their phases are synchronized. A visual feedback-based synchronization algorithm is used to mitigate the complexities and stabilization issues of wire-based triggering for long-distance systems. The HFR projector projects a binary pattern modulated at 3000 fps. The HFR camera system operates at 3000 fps, which can capture and generate a delay signal to be given to the next camera clock cycle so that it matches the phase of the HFR projector. To test the synchronization performance, we used an HFR projector–camera-based VLC system in which the proposed synchronization algorithm provides maximum bandwidth utilization for the high-throughput transmission ability of the system and reduces data redundancy efficiently. The transmitter of the VLC system encodes the input video sequence into gray code, which is projected via high-definition multimedia interface streaming in the form of binary images 590 × 1060. At the receiver, a monochrome HFR camera can simultaneously capture and decode 12-bit 512 × 512 images in real time and reconstruct a color video sequence at 60 fps. The efficiency of the visual feedback-based synchronization algorithm is evaluated by streaming offline and live video sequences, using a VLC system with single and dual projectors, providing a multiple-projector-based system. The results show that the 3000 fps camera was successfully synchronized with a 3000 fps single-projector and a 1500 fps dual-projector system. It was confirmed that the synchronization algorithm can also be applied to VLC systems, autonomous vehicles, and surveillance applications.

Mono Camera-Based Optical Vehicular Communication for an Advanced Driver Assistance System

This technical paper proposes a new waveform combining the low-rate and high-rate data streams to detect the region-of-interest signal in a high-mobility environment using optical camera communication. The proposed technique augments the bit rate of the low-rate stream; consequently, the link setup time is reduced and the requirement of low frame rate camera is eliminated. Additionally, both the low-rate and high-rate data streams in the proposed bi-level pulse position modulation are decoded with a unique adaptive thresholding mechanism with a high frame rate camera. We also propose a vehicle localization scheme to assist the drivers in maintaining a safe following distance that can significantly reduce the frequency of accidents. Moreover, two neural networks are proposed to detect the light-emitting diodes (LEDs) for localization and communication, and to estimate the road curvature from different rear LED shapes of the forwarding vehicle, respectively. The system is implemented, and its performance is analyzed in Python 3.7. The implementation results show that the proposed system is able to achieve 75% localization accuracy, a 150 bps low-rate stream, and a 600 bps high-rate stream over a range of 25 m with a commercial 30 fps camera.

Induction of Microstreaming by Nonspherical Bubble Oscillations in an Acoustic Levitation System.

When located near biological barriers, oscillating microbubbles may increase cell membrane permeability, allowing for drug and gene internalization. Experimental observations suggest that the temporary permeabilization of these barriers may be due to shear stress that is exerted on cell tissues by cavitation microstreaming. Cavitation microstreaming is the generation of vortex flows which arise around oscillating ultrasound microbubbles. To produce such liquid flows, bubble oscillations must deviate from purely spherical oscillations and include either a translational instability or shape modes. Experimental studies of bubble-induced flows and shear stress on nearby surfaces are often restricted in their scope due to the difficulty of capturing shape deformations of microbubbles in a stable and controllable manner. We describe the design of an acoustic levitation chamber for the study of symmetry-controlled nonspherical oscillations. Such control is performed by using a coalescence technique between two approaching bubbles in a sufficiently intense ultrasound field. The control of nonspherical oscillations opens the way to a controlled cavitation microstreaming of a free surface-oscillating microbubble. High-frame rate cameras allow investigating quasi-simultaneously the nonspherical bubble dynamics at the acoustic timescale and the liquid flow at a lower timescale. It is shown that a large variety of fluid patterns may be obtained and that they are correlated to the modal content of the bubble interface. We demonstrate that even the high-order shape modes can create large-distance fluid patterns if the interface dynamics contain several modes, highlighting the potential of nonspherical oscillations for targeted and localized drug delivery.

Continuous-wave planar laser induced fluorescence with a fast camera.

We present planar, laser induced fluorescence (PLIF) measurements of the velocity-resolved distribution function of ions in a plasma using a modulated, narrow linewidth, continuous-wave laser. Plasma emission is acquired with a high frame rate camera, and the laser light is spread into a thin sheet so that an entire plane of the plasma is imaged at each interrogation wavelength. Fourier analysis is conducted on each pixel of the images to separate the modulated fluorescent emission from the background light. Argon ion temperatures and bulk flow maps are reported in a helicon plasma source, and standard single-point LIF measurements provide validation of the PLIF measurement.