Shutter Camera Research Articles

Multi-visual-inertial system: Analysis, calibration, and estimation

In this paper, we study state estimation of multi-visual-inertial systems (MVIS) and develop sensor fusion algorithms to optimally fuse an arbitrary number of asynchronous inertial measurement units (IMUs) or gyroscopes and global and/or rolling shutter cameras. We are especially interested in the full calibration of the associated visual-inertial sensors, including the IMU/camera intrinsics and the IMU-IMU/camera spatiotemporal extrinsics as well as the image readout time of rolling-shutter cameras (if used). To this end, we develop a new analytic combined IMU integration with inertial intrinsics—termed ACI3—to pre-integrate IMU measurements, which is leveraged to fuse auxiliary IMUs and/or gyroscopes alongside a base IMU. We model the multi-inertial measurements to include all the necessary inertial intrinsic and IMU-IMU spatiotemporal extrinsic parameters, while leveraging IMU-IMU rigid-body constraints to eliminate the necessity of auxiliary inertial poses and thus reducing computational complexity. By performing observability analysis of MVIS, we prove that the standard four unobservable directions remain—no matter how many inertial sensors are used, and also identify, for the first time, degenerate motions for IMU-IMU spatiotemporal extrinsics and auxiliary inertial intrinsics. In addition to extensive simulations that validate our analysis and algorithms, we have built our own MVIS sensor rig and collected over 25 real-world datasets to experimentally verify the proposed calibration against the state-of-the-art calibration method Kalibr. We show that the proposed MVIS calibration is able to achieve competing accuracy with improved convergence and repeatability, which is open sourced to better benefit the community.

ChromaFlash: Snapshot Hyperspectral Imaging Using Rolling Shutter Cameras

Hyperspectral imaging captures scene information across narrow, contiguous bands of the electromagnetic spectrum. Despite its proven utility in industrial and biomedical applications, its ubiquity has been limited by bulky form factors, slow capture times, and prohibitive costs. In this work, we propose a generalized approach to snapshot hyperspectral imaging that only requires a standard rolling shutter camera and wavelength-adjustable lighting. The crux of this approach entails using the rolling shutter as a spatiotemporal mask, varying incoming light quicker than the camera's frame rate in order for the captured image to contain rows of pixels illuminated at different wavelengths. An image reconstruction pipeline then converts this coded image into a complete hyperspectral image using sparse optimization. We demonstrate the feasibility of this approach by deploying a low-cost system called ChromaFlash, which uses a smartphone's camera for image acquisition and a series of LEDs to change the scene's illumination. We evaluated ChromaFlash through simulations on two public hyperspectral datasets and assessed its spatial and spectral accuracy across various system parameters. We also tested the real-world performance of our prototype by capturing diverse scenes under varied ambient lighting conditions. In both experiments, ChromaFlash outperformed state-of-the-art techniques that use deep learning to convert RGB images into hyperspectral ones, achieving snapshot performance not demonstrated by prior attempts at accessible hyperspectral imaging.

Demonstration of symbol timing synchronization for rolling shutter-based optical camera communication

Optical camera communication (OCC) provides license-free communication channels without radio waves. The rolling shutter mechanism is expected to enhance the capacity of wireless links in OCC. However, signal erasure occurs due to waiting times between frames in consecutive signal transmission. To address this issue, we propose a symbol timing synchronization for OCC using a rolling shutter camera. The receiver camera synchronizes to the transmitter with additional waiting time which is computed based on the defined preamble signals. We analyzed the performance to clarify the theoretical limit. The feasibility of the proposed idea was confirmed with experimental results.

Spectroscopic investigation of early light emission from anode-initiated surface flashover in vacuum

Time-resolved optical emission spectroscopy is used to explore the early development of anode-initiated flashover in vacuum. An experimental apparatus for collecting flashover self-luminosity is introduced, which enables spatially resolved light collection from regions adjacent to the anode and cathode triple junctions. The emitted light is spectrally and temporally resolved utilizing an imaging spectrograph and a fast electronic shutter camera. The first light detectable from flashovers across both Rexolite and polytetrafluoroethylene (PTFE) insulators begins less than 10 ns prior to the flashover gap's impedance collapse and consists of faint broadband feature in primarily visible wavelengths. This suggests luminosity due to excitation in the solid, possibly occurring as a result of field emitted electrons. In the few nanoseconds leading up to and including the impedance collapse, the broadband spectrum grows in intensity and extends into the UV, eventually accompanied by a few emission lines of the insulator material. This is strong evidence that the early stages of anode-initiated flashover include surface layer breakdown of the insulator, as a contrast from cathode-initiated flashover, which is predominately an above-surface process. Spectra accumulated over longer exposures of PTFE flashovers indicate that, in the first few hundred nanoseconds after the impedance collapse, ions from the insulator material constitute the majority of the emission lines. Later, neutral and ionic metal species from the electrode contribute with similar prominence as well as molecular bands from diatomic carbon. Finally, a comparison is provided of the results of these studies to other spectroscopic investigations of vacuum flashover from the literature.

Design and Implementation of a Hybrid Optical Camera Communication System for Indoor Applications.

Optical wireless communication is a promising emerging technology that addresses the limitations of radio-frequency-based wireless technologies. This study presents a new hybrid modulation method for optical camera communication (OCC), which integrates two waveforms transmitted from a single transmitter light-emitting diode (LED) and receives data through two rolling shutter camera devices on the receiver side. Then, a smart camera with a high-resolution image sensor captures the high-frequency signal, and a low-resolution image sensor from a smartphone camera captures the low-frequency signal. Based on this hybrid scheme, two data streams are transmitted from a single LED, which reduces the cost of the indoor OCC device compared with transmitting two signals from two different LEDs. In the proposed scheme, rolling-shutter orthogonal frequency-division multiplexing is used for the high-frequency signals, and M-ary frequency-shift keying is used for the low-frequency signals in the time domain. This proposed scheme is compatible with smartphone and USB cameras. By controlling the OCC parameters, the hybrid scheme can be implemented with high performance for a communication distance of 10 m.

Integrated particle image velocimetry and fluid–structure interaction analysis for patient-specific abdominal aortic aneurysm studies

BackgroundUnderstanding the hemodynamics of an abdominal aortic aneurysm (AAA) is crucial for risk assessment and treatment planning. This study introduces a low-cost, patient-specific in vitro AAA model to investigate hemodynamics using particle image velocimetry (PIV) and flow-simulating circuit, validated through fluid–structure interaction (FSI) simulations.MethodsIn this study, 3D printing was employed to manufacture a flexible patient-specific AAA phantom using a lost-core casting technique. A pulsatile flow circuit was constructed using off-the-shelf components. A particle image velocimetry (PIV) setup was built using an affordable laser source and global shutter camera, and finally, the flow field inside the AAA was analyzed using open-source software. Fluid–structure interaction (FSI) simulations were performed to enhance our understanding of the flow field, and the results were validated by PIV analysis. Both steady-state and transient flow conditions were investigated.ResultsOur experimental setup replicated physiological conditions, analyzing arterial wall deformations and flow characteristics within the aneurysm. Under constant flow, peak wall deformations and flow velocities showed deviations within − 12% to + 27% and − 7% to + 5%, respectively, compared to FSI simulations. Pulsatile flow conditions further demonstrated a strong correlation (Pearson coefficient 0.85) in flow velocities and vectors throughout the cardiac cycle. Transient phenomena, particularly the formation and progression of vortex structures during systole, were consistently depicted between experimental and numerical models.ConclusionsBy bridging high-fidelity experimental observations with comprehensive computational analyses, this study underscores the potential of integrated methodologies in enhancing our understanding of AAA pathophysiology. The convergence of realistic AAA phantoms, precise PIV measurements at affordable cost point, and validated FSI models heralds a new paradigm in vascular research, with significant implications for personalized medicine and bioengineering innovations.

Rolling Shutter Camera: Modeling, Optimization and Learning

Rolling Shutter Camera: Modeling, Optimization and Learning

Rotating machinery speed extraction through smartphone video acquisition from a radial viewpoint

This article investigates the viability of using a smartphone as a cost-effective tool for rotational speed extraction. The proposed method exploits the geometrical image deformations induced by a smartphone’s rolling shutter camera, which reads pixel lines sequentially with a delay equal to the rolling shutter period. This characteristic period is the basis for the developed methodology and allows measuring constant and varying speeds above the Nyquist limit related to the camera’s frame rate. With respect to the rotating shaft, the smartphone is pointed to the side surface of the shaft (radial viewpoint). A method is proposed to measure the speed from the deformation of a zebra pattern as a consequence of the sequential readout of a rolling shutter camera. A mathematical model, which also takes into account a possible misalignment between the shaft and the camera’s rolling shutter reading direction, is developed and validated on an in-house test rig. It is shown that the proposed method is robust against changing lighting conditions and a misalignment between the shaft and the smartphone camera. Normalized Root-Mean-Square Errors (NRMSE) of about 3% (or less) are reached for the smartphone aligned with the shaft with peaks in the percentage error of about 5% to 8%. Nevertheless, even for a misalignment of 60°, the NRMSE is only 4% with variations in the percentage error to about 10%.The proposed method shows promising results with room for improvements, making it possible to develop a smartphone application to measure a shaft’s rotational speed in the future.

Online Self-Calibration for Visual-Inertial Navigation: Models, Analysis, and Degeneracy

As sensor calibration plays an important role in visual-inertial sensor fusion, this article performs an in-depth investigation of online self-calibration for robust and accurate visual-inertial state estimation. To this end, we first conduct complete observability analysis for visual-inertial navigation systems (VINS) with full calibration of sensing parameters, including inertial measurement unit (IMU)/camera intrinsics and IMU-camera spatial-temporal extrinsic calibration, along with readout time of rolling shutter (RS) cameras (if used). We study different inertial model variants containing intrinsic parameters that encompass most commonly used models for low-cost inertial sensors. With these models, the observability analysis of linearized VINS with full sensor calibration is performed. Our analysis theoretically proves the intuition commonly assumed in the literature—that is, VINS with full sensor calibration has four unobservable directions, corresponding to the system's global yaw and position, while all sensor calibration parameters are observable given fully excited motions. Moreover, we, for the first time, identify degenerate motion primitives for IMU and camera intrinsic calibration, which, when combined, may produce complex degenerate motions. We compare the proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">online</i> self-calibration on commonly used IMUs against the state-of-art <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">offline</i> calibration toolbox Kalibr, showing that the proposed system achieves better consistency and repeatability. Based on our analysis and experimental evaluations, we also offer practical guidelines to effectively perform online IMU-camera self-calibration in practice.

Long-distance indoor optical camera communication using side-emitting fibers as distributed transmitters.

We present a design approach for a long-distance optical camera communication (OCC) system using side-emitting fibers as distributed transmitters. We demonstrate our approach feasibility by increasing the transmission distance by two orders up to 40 m compared to previous works. Furthermore, we explore the effect of the light-emitting diode (LED) modulation frequency and rolling shutter camera exposure time on inter-symbol interference and its effective mitigation. Our proposed OCC-fiber link meets the forward-error-correction (FEC) limit of 3.8 · 10-3 of bit error rate (BER) for up to 35 m (with BER= 3.35 · 10-3) and 40 m (with BER=1.13 · 10-3) using 2-mm and 3-mm diameter side-emitting fibers, respectively. Our results at on-off keying modulation frequencies of 3.54 kHz and 5.28 kHz pave the way to moderate-distance outdoor and long-distance indoor highly-reliable applications in the Internet of Things and OCC using side-emitting fiber-based distributed transmitters.

Extreme detectable vibration frequency limited by rolling shutter camera imaging of laser speckles.

The row scanning mechanism of a rolling shutter camera can be used to infer high-frequency information from a low-frame-rate video. Combining the high intensity of laser speckle and high row-sampling rate of a rolling shutter, extreme detectable vibration frequency limited by rolling shutter camera imaging is experimentally demonstrated. Using a commercially available industrial camera at a frame rate of 70 fps, a vibration signal with a frequency of 14.285 kHz is extracted that corresponds to an inter-row sampling period of 35 µs and a sampling frequency of 28.57 kHz. Connected component and centroid alignment algorithms are used to extract the inter-row vibration displacement. The parameters that limit the highest and lowest detectable frequencies are discussed.

A Review on Image Sensor Communication and Its Applications to Vehicles

Image sensor communication (ISC), also known as optical camera communication, is a form of visible light communication that utilizes image sensors rather than a single photodiode, for data reception. ISC offers spatial separation properties and robustness to ambient noise, making it suitable for outdoor applications such as intelligent transportation systems (ITSs). This review analyzes the research trends in ISC, specifically concerning its application in ITSs. Our focus is on various ISC receivers, including rolling shutter cameras, global shutter high-speed cameras, optical communication image sensors, and event cameras. We analyze how each of these receivers is being utilized in ISC vehicular applications. In addition, we highlight the use of ISC in range estimation techniques and the ability to achieve simultaneous communication and range estimation. By examining these topics, we aim to provide a comprehensive overview of the role of ISC technology in ITSs and its potential for future development.

Experimental Tests and Simulations on Correction Models for the Rolling Shutter Effect in UAV Photogrammetry

Many unmanned aerial vehicles (UAV) host rolling shutter (RS) cameras, i.e., cameras where image rows are exposed at slightly different times. As the camera moves in the meantime, this causes inconsistencies in homologous ray intersections in the bundle adjustment, so correction models have been proposed to deal with the problem. This paper presents a series of test flights and simulations performed with different UAV platforms at varying speeds over terrain of various morphologies with the objective of investigating and possibly optimising how RS correction models perform under different conditions, in particular as far as block control is concerned. To this aim, three RS correction models have been applied in various combinations, decreasing the number of fixed ground control points (GCP) or exploiting GNSS-determined camera stations. From the experimental tests as well as from the simulations, four conclusions can be drawn: (a) RS affects primarily horizontal coordinates and varies notably from platform to platform; (b) if the ground control is dense enough, all correction models lead practically to the same mean error on checkpoints; however, some models may cause large errors in elevation if too few GCP are used; (c) in most cases, a specific correction model is not necessary since the affine deformation caused by RS can be adequately modelled by just applying the extended Fraser camera calibration model; (d) using GNSS-assisted block orientation, the number of necessary GCP is strongly reduced.

DeflickerCycleGAN: learning to detect and remove flickers in a single image.

Eliminating the flickers in digital images captured by rolling shutter cameras is a fundamental and important task in computer vision applications. The flickering effect in a single image stems from the mechanism of asynchronous exposure of rolling shutters employed by cameras equipped with CMOS sensors. In an artificial lighting environment, the light intensity captured at different time intervals varies due to the fluctuation of the AC-powered grid, ultimately leading to the flickering artifact in the image. Up to date, there are few studies related to single image deflickering. Further, it is even more challenging to remove flickers without a priori information, e.g., camera parameters or paired images. To address these challenges, we propose an unsupervised framework termed DeflickerCycleGAN, which is trained on unpaired images for end-to-end single image deflickering. Besides the cycle-consistency loss to maintain the similarity of image contents, we meticulously design another two novel loss functions, i.e., gradient loss and flicker loss, to reduce the risk of edge blurring and color distortion. Moreover, we provide a strategy to determine whether an image contains flickers or not without extra training, which leverages an ensemble methodology based on the output of two previously trained markovian discriminators. Extensive experiments on both synthetic and real datasets show that our proposed DeflickerCycleGAN not only achieves excellent performance on flicker removal in a single image but also shows high accuracy and competitive generalization ability on flicker detection, compared to that of a well-trained classifier based on ResNet50.

Dual-Shutter Optical Vibration Sensing.

Visual vibrometry is a highly useful tool for remote capture of audio, as well as the physical properties of materials, human heart rate, and more. While visually-observable vibrations can be captured directly with a high-speed camera, minute imperceptible object vibrations can be optically amplified by imaging the displacement of a speckle pattern created by shining a laser beam on the vibrating surface. In this paper, we propose a novel method for sensing vibrations at high speeds (up to 63kHz), for multiple scene sources at once, using sensors rated for only 130Hz operation. Our method relies on simultaneously capturing the scene with two cameras equipped with rolling and global shutter sensors, respectively. The rolling shutter camera captures distorted speckle images that encode the high-speed object vibrations. The global shutter camera captures undistorted reference images of the speckle pattern, helping to decode the source vibrations. We demonstrate our method by capturing vibration caused by audio sources (e.g., speakers, human voice, and musical instruments) and analyzing the vibration modes of a tuning fork.

Conception of a High-Level Perception and Localization System for Autonomous Driving

This paper describes the conception of a high level, compact, scalable, and long autonomy perception and localization system for autonomous driving applications. Our benchmark is composed of a high resolution lidar (128 channels), a stereo global shutter camera, an inertial navigation system, a time server, and an embedded computer. In addition, in order to acquire data and build multi-modal datasets, this system embeds two perception algorithms (RBNN detection, DCNN detection) and one localization algorithm (lidar-based localization) to provide real-time advanced information such as object detection and localization in challenging environments (lack of GPS). In order to train and evaluate the perception algorithms, a dataset is built from 10,000 annotated lidar frames from various drives carried out under different weather conditions and different traffic and population densities. The performances of the three algorithms are competitive with the state-of-the-art. Moreover, the processing time of these algorithms are compatible with real-time autonomous driving applications. By providing directly accurate advanced outputs, this system might significantly facilitate the work of researchers and engineers with respect to planning and control modules. Thus, this study intends to contribute to democratizing access to autonomous vehicle research platforms.

Spatial-Temporal Measurement Alignment of Rolling Shutter Camera and LiDAR

This letter proposes a novel method to fuse the asynchronized outputs from a rolling shutter camera and a spinning LiDAR mounted on a moving vehicle. Compared with traditional methods only relying on intrinsic/extrinsic calibration, the proposed method takes the ego-motion, rolling shutter distortion, and occlusion into the fusion model. In essence, the method estimates the temporal offset between LiDAR and the camera by minimizing the reprojection error of the row values between consecutive iterations. Hence, it can precisely project the 3-D LiDAR points into the image pixels from a rolling shutter camera moving at high speed, which is critical for a multisensory autonomous driving system. Field tests based on a vehicle prototype verify the performance enhancement of the proposed method, particularly in high-speed conditions. This method is especially helpful for engineering practices on low-level sensor fusion of LiDAR and rolling shutter camera.

Observability Analysis and Keyframe-Based Filtering for Visual Inertial Odometry With Full Self-Calibration

Camera–inertial measurement unit (IMU) sensor fusion has been extensively studied in recent decades. Numerous observability analysis and fusion schemes for motion estimation with self-calibration have been presented. However, it has been uncertain whether the intrinsic parameters of both the camera and the IMU are observable under general motion. To answer this question, by using the Lie derivatives, we first prove that for a rolling shutter (RS) camera–IMU system, all the intrinsic and extrinsic parameters, camera time offset, and readout time of the RS camera are observable with an unknown landmark. To our knowledge, we are the first to present such a proof. Next, to validate this analysis and to solve the drift issue of a structureless filter during standstills, we develop a keyframe-based sliding window filter (KSWF) for odometry and self-calibration, which works with a monocular RS camera or stereo RS cameras. Though the keyframe concept is widely used in vision-based sensor fusion, to our knowledge, the KSWF is the first of its kind to support self-calibration. Our simulation and real data tests have validated that it is possible to fully calibrate the camera–IMU system using the observations of opportunistic landmarks under diverse motion. Real data tests confirmed previous allusions that keeping landmarks in the state vector can remedy the drift in standstill and showed that the keyframe-based scheme is an alternative solution.

Ctrl-VIO: Continuous-Time Visual-Inertial Odometry for Rolling Shutter Cameras

In this letter, we propose a probabilistic continuous-time visual-inertial odometry (VIO) for rolling shutter cameras. The continuous-time trajectory formulation naturally facilitates the fusion of asynchronized high-frequency IMU data and motion-distorted rolling shutter images. To prevent intractable computation load, the proposed VIO is sliding-window and keyframe-based. We propose to probabilistically marginalize the control points to keep the constant number of keyframes in the sliding window. Furthermore, the line exposure time difference (line delay) of the rolling shutter camera can be online calibrated in our continuous-time VIO. To extensively examine the performance of our continuous-time VIO, experiments are conducted on publicly-available WHU-RSVI, TUM-RSVI, and SenseTime-RSVI rolling shutter datasets. The results demonstrate the proposed continuous-time VIO significantly outperforms the existing state-of-the-art VIO methods.

Tunable Laser Absorption Imaging for 2D Gas Measurement With an Electronic Rolling Shutter Camera

High spatial resolution two-dimensional (2D) imaging of gas flow is essential for better understanding and characterization of a combustion field. We propose a method for 2D gas measurement with high spatial resolution by combining tunable laser absorption spectroscopy and an imager with an electronic rolling shutter through their common feature of time scanning. The method was demonstrated by oxygen measurement using a 764nm vertical cavity surface emitting laser and a low-cost CMOS camera. 2D imaging of the oxygen concentration distribution were obtained at a frame rate of 15fps with a horizontal spatial resolution of about <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$28\mu \text{m}$ </tex-math></inline-formula> . This method enjoys simple structure, low cost, high spatial resolution and shows potential in the 2D and 3D reconstruction of the combustion field.