Low Frame Research Articles

Self-referencing for quasi shot-noise-limited widefield transient microscopy

Many applications of ultrafast and nonlinear optical microscopy require the measurement of small differential signals over large fields-of-view. Widefield configurations drastically reduce the acquisition time; however, they suffer from the low frame rates of two-dimensional detectors, which limit the modulation frequency, making the measurement sensitive to excess laser noise. Here we introduce a self-referenced detection configuration for widefield differential imaging. Employing regions of the field of view with no differential signal as references, we cancel probe fluctuations and increase the signal-to-noise ratio by an order of magnitude reaching noise levels only a few percent above the shot noise limit. We anticipate broad applicability of our method to transient absorption, stimulated Raman scattering and photothermal-infrared microscopies.

Sensor synchronized DIC: A robust approach to linear and nonlinear modal analysis using low frame rate cameras

Digital image correlation (DIC) is a non-contacting, optical method for measuring full-field displacements. Recently, there has been interest in DIC as a tool for experimental modal analysis. Currently DIC necessitates the use of high-speed cameras that satisfy the bandwidth requirements for the system under investigation. High-speed cameras, however, tend to be expensive, bulky, and low-resolution, thus limiting their application. In contrast, low-speed cameras tend to be cheaper, smaller, and higher-resolution. In this work an alternative approach that utilizes low-speed cameras is developed. The approach functions by synchronizing image capture against a high bandwidth reference sensor. This enables images to be precisely captured at prescribed moments in a waveform. A basic algorithm is developed for the synchronization, including an uncertainty analysis to estimate phase errors arising from the approach. The method is then demonstrated by using low-speed cameras to estimate the full-field amplitude response of a thin, cantilevered plate. This is achieved by alternating image capture between extremums during a swept-sine test. The excitation level is then increased to elicit a highly nonlinear response from the plate. Measurements are repeated to demonstrate that the approach works even when the system exhibits large harmonic distortions.

Simulating a turbulent video affected by spatiotemporally varying blur and tilt using temporal cross correlation of intermodal Zernike coefficients

Videos captured in long-distance horizontal imaging through the atmosphere suffer from dynamic spatiotemporal movements and blur caused by the air turbulence. Simulations of atmospheric turbulence in such videos, which have been conducted in the past, are difficult to compute. Our goal in this research is to develop an effective simulation algorithm of videos affected by atmospheric turbulence characterized by spatiotemporally varying blur and tilt, when supplied with a given image. We accomplish this via extending an already established method that simulates atmospheric turbulence in a single image, by incorporating turbulence properties in the time domain that include both the tilts and blurring effects. This study also extends our previous work that simulated turbulence, but did not consider the space-varying property of the blur. This is done by employing the relationship between turbulence image distortions and the intermodal correlations of the Zernike coefficients in time and space, and also via analyzing the spatiotemporal matrix that represents the spatial correlation of movements between different frames. The proposed method can facilitate the production of simulations, given turbulence properties that include turbulence strength, object distance, and height. The simulation is applied to videos with low and high frame rates, and the differences between them are analyzed. The proposed method can prove useful when generating machine-learning algorithms that apply to videos affected by atmospheric turbulence, which require large labeled video datasets (with controlled turbulence and imaging parameters) for training.

MAM[formula omitted]SLAM: Towards underwater-robust multi-agent visual SLAM

Some underwater applications involve deploying multiple underwater Remotely Operated Vehicles in a common area. Such applications require the localization of these vehicles, not only with respect to each other but also with respect to a previously unknown environment. To this end, this work presents MAM3SLAM, a new fully centralized multi-agent and multi-map monocular Visual Simultaneous Localization And Mapping (VSLAM) framework. Multi-agent evaluation metrics are introduced to provide an extensive evaluation of MAM3SLAM compared to the state-of-the-art multi-agent VSLAM on four two-agent scenarios: one standard airborne dataset and three new underwater datasets recorded in a pool and the sea. The results show that MAM3SLAM is robust to underwater visual conditions and tracking failures, outperforms the other evaluated methods in estimating the individual and relative poses of the agents and in collaborative mapping accuracy. MAM3SLAM successfully estimates the individual and relative localization of the agents with an error lower than 5 cm on three out of the four test sequences, and is twice as accurate as competing multi-agent works in challenging visual conditions with frequent visual dropouts, poor textures, low framerate and fast motion. MAM3SLAM’s source code is made available, as are the underwater datasets.

Dose-area product reduction through a practice implementing low frame rate fluoroscopy and increased collimation during single-vessel percutaneous coronary interventions

Abstract This study aims to investigate the impact of lowering fluoroscopic frame rate and increasing collimation on dose area product (DAP) during single-vessel percutaneous coronary interventions. Two patient groups that underwent such procedures were retrospectively reviewed: one treated with a standard practice employing fluoroscopy at 15 frames/s and usual collimation (standard group), and the other with a reduced practice employing fluoroscopy at 7.5 frames/s and increased collimation (reduced group). DAP, patient’s effective dose (ED), fluoroscopy time, and number of cineangiography (cine) frames were compared between groups. Multilinear analysis was performed, with DAP as a primary endpoint, to assess the impact of reduced practice in comparison to that of other variables deemed as key descriptors of radiation, patient, disease, or procedure. Relative to the standard group, the reduced group showed a significant decrease of ~37% in DAP (91.46 ± 45.08 Gy•cm2 vs. 145.23 ± 59.67 Gy•cm2, P < 0.0001) and of ~39% in ED (18.70 ± 9.00 mSv vs. 30.61 ± 12.72 mSv, P < 0.0001). While fluoroscopic time was not statistically different between groups (reduced: 10.3 ± 5.6 min vs. standard: 11.1 ± 5.3 min, P = 0.09), number of cine frames was significantly higher in the reduced (1162 ± 304 vs. 936 ± 363, P < 0.0001) due to interoperator experience variability. Multilinear analysis demonstrated that the adoption of the reduced practice was the most significant contributor to the observed DAP decrease. After adjusting for potential confounders, a reduction of ~39% in DAP was found.

Real-time, depth-resolved, in vivo multiphoton fluorescence lifetime imaging microscopy of agricultural herbicide treatments in plants

The development of effective and safe agricultural treatments requires sub-cellular insight of the biochemical effects of treatments in living tissue in real-time. Industry-standard mass spectroscopic imaging lacks real-time in vivo capability. As an alternative, multiphoton fluorescence lifetime imaging microscopy (MPM-FLIM) allows for 3D sub-cellular quantitative metabolic imaging but is often limited to low frame rates. To resolve relatively fast effects (e.g., photosynthesis inhibiting treatments), high-frame-rate MPM-FLIM is needed. In this paper, we demonstrate and evaluate a high-speed MPM-FLIM system, “Instant FLIM”, as a time-resolved 3D sub-cellular molecular imaging system in highly scattering, living plant tissues. We demonstrate simultaneous imaging of cellular autofluorescence and crystalline agrochemical crystals within plant tissues. We further quantitatively investigate the herbicidal effects of two classes of agricultural herbicide treatments, photosystem II inhibiting herbicide (Basagran) and auxin-based herbicide (Arylex), and successfully demonstrate the capability of the MPM-FLIM system to measure biological changes over a short time with enhanced imaging speed. Results indicate that high-frame-rate 3D MPM-FLIM achieves the required fluorescence lifetime resolution, temporal resolution, and spatial resolution to be a useful tool in basic plant cellular biology research and agricultural treatment development.

SlowFastFormer for 3D human pose estimation

3D human pose estimation in videos aims at locating the human joints in the 3D space given a temporal sequence. Motion information and skeleton context are two significant elements for pose estimation in videos. In this paper, we propose a SlowFastFormer (slow-fast transformer) network where two branches with different input rates are composed to encode these two different kinds of context. For the slow branch, skeleton context is well learned at a higher frame rate. For the fast branch, motion information is captured at a lower frame rate. Through these two branches, different kinds of context are encoded separately. We fuse these two branches at a later stage to fully utilize the skeleton context and motion information. Afterwards, a blending module is developed to promote the message exchange among multiple branches. In the blending stage, different kinds of context information are exchanged and feature representation is enhanced consequently. Lastly, a hierarchical supervision scheme is tailored where predictions of different levels are inferred in a progressive manner. Our approach achieves competitive performance with lower computation complexity on several benchmarks, i.e., Human3.6M, MPI-INF-3DHP and HumanEva-I.

High-speed noncoherent underwater acoustic communication using permutation alphabets and sliding hypothesis-tree decoding.

A framework for joint synchronization and decoding is presented, named sliding hypothesis-tree decoding (SHDT), which targets communication applications in adverse underwater acoustic channels. The framework is exemplified by constructing a robust noncoherent link with spectral efficiency 0.149-0.222 bit/s/Hz using Polar coding and super permutation frequency shift keying. The link method is evaluated with randomized time-Doppler shifts through replay simulation of recorded UWA channels, where the five benchmark channels of the public benchmark Watermark and two channels from the Baltic Sea are used; these simulations show that a low frame error rate is achievable in all seven channels. The constructed link method is further evaluated in an adverse acoustic horizontal shallow-water channel using a high-mobility setup; the link method is able to perform synchronization, and a decoder performance similar to the replay simulations is observed. SHDT allows the construction of link methods with a performance limited by the information decoding rather than the time-Doppler synchronization, making it applicable to communication with lower-rate methods in non-line-of-sight scenarios.

Towards vision-based structural modal identification at low frame rate using blind source separation

With increasing availability of cost-effective and high-resolution cameras, their use as a non-contact sensing tool has rapidly progressed for structural health monitoring. The cameras offer unique capabilities to provide full-field measurement with high spatial density at low cost. However, extracting high-density temporal data is challenging, as a high-speed camera increases the monitoring cost with high-rate data processing. Recently, motion magnification (MM) has shown significant success in analyzing low-amplitude motion of structural systems. However, previous studies observed that MM methodology performs poorly at low frame rates for modal identifications. In this paper, the influence of low frame rate on phased-based motion magnification (PMM) has been investigated. A novel technique is proposed by combining PMM with zero mean-normalization cross-correlation tracker to determine vibrational responses, and then the spatial Wigner-Ville spectrum-based time-frequency blind source separation method is explored for modal identification using the extracted vibrational responses obtained from the video data. The experimental data of a lumped mass experimental model and a steel bridge is used to test the accuracy of the proposed method. The original and motion-magnified image response data is compared with accelerometer data for modal identification. The proposed method is able to extract the modal parameters with high accuracy for motion-magnified images, even for low frame rates.

Synthetic interpolated DSA for radiation exposure reduction via gamma variate contrast flow modeling: a retrospective cohort study

BackgroundDigital subtraction angiography (DSA) yields high cumulative radiation dosages (RD) delivered to patients. We present a temporal interpolation of low frame rate angiograms as a method to reduce cumulative RDs.MethodsPatients undergoing interventional evaluation and treatment of cerebrovascular vasospasm following subarachnoid hemorrhage were retrospectively identified. DSAs containing pre- and post-intervention runs capturing the full arterial, capillary, and venous phases with at least 16 frames each were selected. Frame rate reduction (FRR) of the original DSAs was performed to 50%, 66%, and 75% of the original frame rate. Missing frames were regenerated by sampling a gamma variate model (GVM) fit to the contrast response curves to the reduced data. A formal reader study was performed to assess the diagnostic accuracy of the “synthetic” studies (sDSA) compared to the original DSA.ResultsThirty-eight studies met inclusion criteria (average RD 1,361.9 mGy). Seven were excluded for differing views, magnifications, or motion. GVMs fit to 50%, 66%, and 75% FRR studies demonstrated average voxel errors of 2.0 ± 2.5% (mean ± standard deviation), 6.5 ± 1.5%, and 27 ± 2%, respectively for anteroposterior projections, 2.0 ± 2.2%, 15.0 ± 3.1%, and 14.8 ± 13.0% for lateral projections, respectively. Reconstructions took 0.51 s/study. Reader studies demonstrated an average rating of 12.8 (95% CI 12.3−13.3) for 75% FRR, 12.7 (12.2−13.2) for 66% FRR and 12.0 (11.5−12.5) for 50% FRR using Subjective Image Grading Scale. Kendall’s coefficient of concordance resulted in W = 0.506.ConclusionFRR by 75% combined with GVM reconstruction does not compromise diagnostic quality for the assessment of cerebral vasculature.Relevance statementUsing this novel algorithm, it is possible to reduce the frame rate of DSA by as much as 75%, with a proportional reduction in radiation exposure, without degrading imaging quality.Key points• DSA delivers some of the highest doses of radiation to patients.• Frame rate reduction (FRR) was combined with bolus tracking to interpolate intermediate frames.• This technique provided a 75% FRR with preservation of diagnostic utility as graded by a formal reader study for cerebral angiography performed for the evaluation of cerebral vasospasm.• This approach can be applied to other types of angiography studies.Graphical

Indoor Mapping with Entertainment Devices: Evaluating the Impact of Different Mapping Strategies for Microsoft HoloLens 2 and Apple iPhone 14 Pro.

Due to their low cost and portability, using entertainment devices for indoor mapping applications has become a hot research topic. However, the impact of user behavior on indoor mapping evaluation with entertainment devices is often overlooked in previous studies. This article aims to assess the indoor mapping performance of entertainment devices under different mapping strategies. We chose two entertainment devices, the HoloLens 2 and iPhone 14 Pro, for our evaluation work. Based on our previous mapping experience and user habits, we defined four simplified indoor mapping strategies: straight-forward mapping (SFM), left-right alternating mapping (LRAM), round-trip straight-forward mapping (RT-SFM), and round-trip left-right alternating mapping (RT-LRAM). First, we acquired triangle mesh data under each strategy with the HoloLens 2 and iPhone 14 Pro. Then, we compared the changes in data completeness and accuracy between the different devices and indoor mapping applications. Our findings show that compared to the iPhone 14 Pro, the triangle mesh accuracy acquired by the HoloLens 2 has more stable performance under different strategies. Notably, the triangle mesh data acquired by the HoloLens 2 under the RT-LRAM strategy can effectively compensate for missing wall and floor surfaces, mainly caused by furniture occlusion and the low frame rate of the depth-sensing camera. However, the iPhone 14 Pro is more efficient in terms of mapping completeness and can acquire a complete triangle mesh more quickly than the HoloLens 2. In summary, choosing an entertainment device for indoor mapping requires a combination of specific needs and scenes. If accuracy and stability are important, the HoloLens 2 is more suitable; if efficiency and completeness are important, the iPhone 14 Pro is better.

A framework to facilitate development and testing of image‐based river velocimetry algorithms

AbstractImage‐based methods have compelling, demonstrated potential for characterizing flow fields in rivers, but algorithms like particle image velocimetry (PIV) must be further tested and improved to enable more effective use of these techniques. This paper presents a framework designed for this exact purpose: Simulating Hydraulics and Images for Velocimetry Evaluation and Refinement (SHIVER). The approach involves coupling a hydrodynamic model with a synthetic particle generator to advect particles between frames, as dictated by local velocity vectors and thus construct a plausible image sequence specific to the reach of interest. The resulting time series can then be used as input to a velocimetry algorithm to compare image‐derived estimates with known (modelled) velocities to perform an exhaustive, spatially distributed accuracy assessment. As an example application of SHIVER, we examined the effects of interrogation area (IA) size, frame rate, flow velocity, and image sequence duration on the performance of a standard PIV algorithm. This analysis indicated that image‐derived velocities were generally in close agreement with those from the flow model (root mean square error <10% and mean bias <3%), except when small IAs were coupled with low frame rates. Velocity estimates were most accurate for the lowest modelled discharge ( at baseflow) and became less reliable as the mean flow velocity increased ( for an intermediate discharge and at bankfull). Accuracy was essentially independent of image sequence duration, implying that long occupations might not be necessary. Errors were concentrated along channel margins, where PIV‐based velocities tended to be greater than those from the flow model. Small IAs led to underpredictions of velocity, while larger IAs led to overpredictions. SHIVER is highly modular and could be updated to make use of different hydrodynamic models or image simulators. The framework could also facilitate more thorough sensitivity analyses and comparison of various velocimetry algorithms.

Data-Driven Detection of Video Forgery for Secured Microcontrollers: Dataset and Method

The microcontrollers in a camera often capture videos at a low frame rate due to limited processing capability. To satisfy the requirement of high quality of service, low-frame-rate videos are often forged as the high-frame-rate ones by the Frame Rate Up-Conversion (FRUC) operation. Therefore, detecting the existence of FRUC has become a necessary job for secured microcontrollers. In this paper, we propose a data-driven detection to identify whether a video is forged by FRUC. The core of detection is the creation of a large-scale video dataset VifFRUC (Videos forged by FRUC). Various types of forged videos can continue to be added into VifFRUC, making the detection more universal and robust. To match with VifFRUC, we have also designed a neural network, which trains a number of Long Short Term Memory (LSTM) units in parallel to learn the data-driven detection. The parallel LSTM structure of network can continually adapt to the newly added FRUC methods in VifFRUC. Extensive experiments on VifFRUC demonstrate the effectiveness of data-driven detection for FRUC, resulting in the security improvement of microcontrollers.

Arbitrary Timestep Video Frame Interpolation with Time-Dependent Decoding

Given an observed low frame rate video, video frame interpolation (VFI) aims to generate a high frame rate video, which has smooth video frames with higher frames per second (FPS). Most existing VFI methods often focus on generating one frame at a specific timestep, e.g., 0.5, between every two frames, thus lacking the flexibility to increase the video’s FPS by an arbitrary scale, e.g., 3. To better address this issue, in this paper, we propose an arbitrary timestep video frame interpolation (ATVFI) network with time-dependent decoding. Generally, the proposed ATVFI is an encoder–decoder architecture, where the interpolation timestep is an extra input added to the decoder network; this enables ATVFI to interpolate frames at arbitrary timesteps between input frames and to increase the video’s FPS at any given scale. Moreover, we propose a data augmentation method, i.e., multi-width window sampling, where video frames can be split into training samples with multiple window widths, to better leverage training frames for arbitrary timestep interpolation. Extensive experiments were conducted to demonstrate the superiority of our model over existing baseline models on several testing datasets. Specifically, our model trained on the GoPro training set achieved 32.50 on the PSNR metric on the commonly used Vimeo90k testing set.

Adaptive Sparse Memory Networks for Efficient and Robust Video Object Segmentation.

Recently, memory-based networks have achieved promising performance for video object segmentation (VOS). However, existing methods still suffer from unsatisfactory segmentation accuracy and inferior efficiency. The reasons are mainly twofold: 1) during memory construction, the inflexible memory storage mechanism results in a weak discriminative ability for similar appearances in complex scenarios, leading to video-level temporal redundancy, and 2) during memory reading, matching robustness and memory retrieval accuracy decrease as the number of video frames increases. To address these challenges, we propose an adaptive sparse memory network (ASM) that efficiently and effectively performs VOS by sparsely leveraging previous guidance while attending to key information. Specifically, we design an adaptive sparse memory constructor (ASMC) to adaptively memorize informative past frames according to dynamic temporal changes in video frames. Furthermore, we introduce an attentive local memory reader (ALMR) to quickly retrieve relevant information using a subset of memory, thereby reducing frame-level redundant computation and noise in a simpler and more convenient manner. To prevent key features from being discarded by the subset of memory, we further propose a novel attentive local feature aggregation (ALFA) module, which preserves useful cues by selectively aggregating discriminative spatial dependence from adjacent frames, thereby effectively increasing the receptive field of each memory frame. Extensive experiments demonstrate that our model achieves state-of-the-art performance with real-time speed on six popular VOS benchmarks. Furthermore, our ASM can be applied to existing memory-based methods as generic plugins to achieve significant performance improvements. More importantly, our method exhibits robustness in handling sparse videos with low frame rates.

Beyond Appearance: Multi-Frame Spatio-Temporal Context Memory Networks for Efficient and Robust Video Object Segmentation.

Current video object segmentation approaches primarily rely on frame-wise appearance information to perform matching. Despite significant progress, reliable matching becomes challenging due to rapid changes of the object's appearance over time. Moreover, previous matching mechanisms suffer from redundant computation and noise interference as the number of accumulated frames increases. In this paper, we introduce a multi-frame spatio-temporal context memory (STCM) network to exploit discriminative spatio-temporal cues in multiple adjacent frames by utilizing a multi-frame context interaction module (MCI) for memory construction. Based on the proposed MCI module, a sparse group memory reader is developed to enable efficient sparse matching during memory reading. Our proposed method is generic and achieves state-of-the-art performance with real-time speed on benchmark datasets such as DAVIS and YouTube-VOS. In addition, our model exhibits robustness to sparse videos with low frame rates.

Neuromorphic Imaging With Joint Image Deblurring and Event Denoising.

Neuromorphic imaging reacts to per-pixel brightness changes of a dynamic scene with high temporal precision and responds with asynchronous streaming events as a result. It also often supports a simultaneous output of an intensity image. Nevertheless, the raw events typically involve a large amount of noise due to the high sensitivity of the sensor, while capturing fast-moving objects at low frame rates results in blurry images. These deficiencies significantly degrade human observation and machine processing. Fortunately, the two information sources are inherently complementary - events with microsecond-level temporal resolution, which are triggered by the edges of objects recorded in a latent sharp image, can supply rich motion details missing from the blurry one. In this work, we bring the two types of data together and introduce a simple yet effective unifying algorithm to jointly reconstruct blur-free images and noise-robust events in an iterative coarse-to-fine fashion. Specifically, an event-regularized prior offers precise high-frequency structures and dynamic features for blind deblurring, while image gradients serve as a kind of faithful supervision in regulating neuromorphic noise removal. Comprehensively evaluated on real and synthetic samples, such a synergy delivers superior reconstruction quality for both images with severe motion blur and raw event streams with a storm of noise, and also exhibits greater robustness to challenging realistic scenarios such as varying levels of illumination, contrast and motion magnitude. Meanwhile, it can be driven by much fewer events and holds a competitive edge at computational time overhead, rendering itself preferable as available computing resources are limited. Our solution gives impetus to the improvement of both sensing data and paves the way for highly accurate neuromorphic reasoning and analysis.

Evaluation of Polar-Coded Noncoherent Acoustic Underwater Communication

This article investigates several variants of noncoherent multiple-frequency-shift keying (MFSK) that uses Polar coding and evaluates their applicability to robust underwater acoustic underwater communication. A simplified data model is assumed, yielding a low-complexity MFSK symbol estimator, and symbol likelihoods are calculated using sparse pilot symbols. Results presented from field tests in the Stockholm archipelago indicate that a Polar code will decode with a low frame error rate (FER) if not signal-to-noise ratio (SNR) limited and that its performance is superior to convolutional coding. The reliability of short frames, with 0.16–0.5 s transmission time, and long frames, with spectral efficiency up to 0.1 b/s/Hz, are evaluated in the five replay channels of the Watermark benchmark over varying SNR and SNR per bit, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$E_{b}/N_{0}$</tex-math></inline-formula> . Synchronization is solved with a sliding hypothesis approach and the results show a FER <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\lesssim 1 \%$</tex-math></inline-formula> for <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">all</i> Watermark channels, where the robustness is significantly improved by either increasing the frequency separation of tones or lowering the code rate and extending the frame length. The performance compares favorably to methods found in the literature, particularly for the less-benign channels of the benchmark.

Retracted: Research on Low Frame Rate Video Compression Algorithm in the Context of New Media

Retracted: Research on Low Frame Rate Video Compression Algorithm in the Context of New Media

Observing inter-well and intra-well oscillations in buckled nanomechanical systems enabled by image processing

The scanning electron microscope (SEM) recordings of dynamic nano-electromechanical systems (NEMS) are difficult to analyze due to the noise caused by low frame rate, insufficient resolution, and blurriness induced by applied electric potentials. Here, we develop an image processing platform enhanced by the physics of the underlying system to track the motion of buckling NEMS structures in the presence of high noise levels. The algorithm is composed of an image filter, two data filters, and a nonlinear regression model, which utilizes the expected form of the physical solution. The method was applied to the recordings of a NEMS beam about 150 nm wide, undergoing intra- and inter-well post-buckling states with a transition rate of approximately 0.5 Hz. The algorithm can track the dynamical motion of the NEMS and capture the dependency of deflection amplitude on the compressive force on the beam. With the help of the proposed algorithm, the transition from inter-well to intra-well motion is clearly resolved for buckling NEMS imaged under SEM.