Rolling Shutter Image Research Articles

Wavy water-to-air optical camera communication system using rolling shutter image sensor and long short term memory neural network.

We propose and experimentally demonstrate for the first time up to the authors' knowledge a wide field-of-view (FOV) water-to-air optical transmission using rolling-shutter (RS) based optical camera communication (OCC). Here, we evaluate the proposed OCC system without water ripple and with different percentage increases of water ripple. Long short term memory neural network (LSTM-NN) is utilized to mitigate the wavy water turbulence induced link outage and to decode 4-level pulse-amplitude-modulation (PAM4) RS pattern by meeting the pre-forward error correction bit-error-rate (pre-FEC BER = 3.8 × 10-3). We also evaluate the FOVs of the proposed water-to-air RS-based OCC system. This can be implemented by using different angular rotations of the camera. Experimental results show that the proposed OCC system can support ±70°, ± 30°, and ±30° rotations around the z-, y- and x-directions, respectively when operated at 6 kbit/s and decoded using LSTM-NN.

Rolling shutter speckle plethysmography for quantitative cardiovascular monitoring.

We propose a new speckle-based plethysmography technique, termed rolling shutter speckle plethysmography (RSSPG), which can quantitatively measure the velocity and volume fluctuations of blood flow during the cardiac cycle. Our technique is based on the rolling shutter speckle imaging, where the short row-by-row time differences in the rolling shutter image sensors are used to measure the temporal decorrelation behavior of vertically elongated speckles from a single image capture. Temporal analysis of the speckle field provides rich information regarding the dynamics of the scattering media, such as both the dynamic scattering fraction and the speckle decorrelation time. Using a sequence of images, RSSPG can monitor fluctuations in the blood flow dynamics while separating velocity and volume changes in blood vessels and obtaining high-quality plethysmography waveforms compared to regular photoplethysmography. We demonstrate the quantitative RSSPG based on accurate fitting of the speckle dynamics model, as well as the qualitative RSSPG based on simple row-by-row correlation (RIC) calculation for fast and robust analysis. Based on exploratory in vivo experiment, we show that RSSPG can reliably measure pulsatile waveforms and heart rate variations in various conditions, potentially providing physiologically relevant information for cardiovascular monitoring.

A star identification algorithm for rolling shutter exposure based on Hough transform

The star identification algorithm usually identifies stars by angular distance matching. However, under high dynamic conditions, the rolling shutter effect distorts the angular distances between the measured and true star positions, leading to plethoric false matches and requiring complex and time-consuming verification for star identification. Low identification rate hinders the application of low-noise and cost-effective rolling shutter image sensors. In this work, we first study a rolling shutter distortion model of angular distances between stars, and then propose a novel three-stage star identification algorithm to identify distorted star images captured by the rolling shutter star sensor. The first stage uses a modified grid algorithm with adaptive error tolerance and an expanded pattern database to efficiently eliminate spurious matches. The second stage performs angular velocity estimation based on Hough transform to verify the matches that follow the same distortion pattern. The third stage applies a rolling shutter error correction method for further verification. Both the simulation and night sky image test demonstrate the effectiveness and efficiency of our algorithm under high dynamic conditions. The accuracy of angular velocity estimation method by Hough transform is evaluated and the root mean square error is below 0.5 (°)/s. Our algorithm achieves a 95.7% identification rate at an angular velocity of 10 (°)/s, which is much higher than traditional algorithms.

Point feature correction based rolling shutter modeling for EKF-based visual-inertial odometry

The application of consumer-grade rolling-shutter (RS) cameras in visual inertial odometry (VIO) systems deserves attention. In previous works, VIO systems with RS cameras usually adopt different dimensions of RS modeling to represent the camera motion. Although the complex camera motion expression improves accuracy, adding more variables greatly increases the computation load. Therefore, considering the limitations of camera motion modeling, this paper introduces a point feature correction strategy to improve the performance of the system in terms of both speed and accuracy. The optimized camera poses and high-frequency inertial measurement unit (IMU) are employed to correct the point features in the RS image to new pixel coordinates in the global shutter view (in the middle of the RS image readout time) by epipolar transfer. At the same time, we handle the zero velocity and abnormal cases of RS modeling to improve the robustness of the system. Furthermore, the readout time of the RS camera is self-calibrated online during the state augmentation stage and can quickly converge and stabilize to near the true value. The readout time of the RS camera is locally observable, except for two degenerate motions. Experimental results demonstrate that the proposed method outperforms the state-of-the-art methods on a public dataset in terms of accuracy and computational cost. We port our algorithm to an Android-based mobile phone and our system outputs real-time trajectories on the mobile phone. Then, we use the motion capture system to obtain the ground truth phone pose to test the performance of our algorithm.

Achieving Successful VLC Signal Reception Using A Rolling Shutter Image Sensor While Driving at 40 km/h

Achieving Successful VLC Signal Reception Using A Rolling Shutter Image Sensor While Driving at 40 km/h

High Sampling Rate Smartphone-PPG via Built-in Rolling Shutter Image Sensor

Recent advancement of CMOS camera image sensor (CIS) on smartphone brings a significant improvement to the IoT-based mobile healthcare technology in the form of CIS-photoplethysmography (CPPG). Nevertheless, most of the available smartphone is equipped with a limited sampling rate (Fs), typically 30 frame per second (fps), thus often resulting in a distorted CPPG signal acquisition. This distorted signal is hard to be utilized for different types of advanced photoplethysmography (PPG)-derived physiological analysis, and only useful in a simple pulse rate monitoring system. In this article, the rolling-shutter camera mechanism has been exploited to extract CPPG data points from CIS-pixel rows, thus allowing high-Fs CPPG signal extraction from a common built-in, low-fps smartphone CIS. Multiple experiments were conducted to prove the reliability of rolling-shutter CPPG (RSCPPG) signal. First, we conduct iterative experiments with different CIS parameters to find their correlation to the acquired RSCPPG signal quality. Results indicate that the short exposure time produces a high-SNR CPPG signal up to 25 ± 2.38 dB, and highly correlated signal morphology (average <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${r}$ </tex-math></inline-formula> = 0.95) compared to the reference PPG signal. Then, we also demonstrated the proposed RSCPPG algorithm allowing a high CPPG data sampling with Fs = 150 Hz (that is ≥ 5 times CIS fps). Finally, a feasibility study has been conducted on multiple features extracted from RSCPPG that are potentially implemented for further physiological analysis application. These findings suggest that the proposed RSCPPG algorithm is a reliable bio-signal acquisition technique in smartphone-based healthcare technology.

Ctrl-VIO: Continuous-Time Visual-Inertial Odometry for Rolling Shutter Cameras

In this letter, we propose a probabilistic continuous-time visual-inertial odometry (VIO) for rolling shutter cameras. The continuous-time trajectory formulation naturally facilitates the fusion of asynchronized high-frequency IMU data and motion-distorted rolling shutter images. To prevent intractable computation load, the proposed VIO is sliding-window and keyframe-based. We propose to probabilistically marginalize the control points to keep the constant number of keyframes in the sliding window. Furthermore, the line exposure time difference (line delay) of the rolling shutter camera can be online calibrated in our continuous-time VIO. To extensively examine the performance of our continuous-time VIO, experiments are conducted on publicly-available WHU-RSVI, TUM-RSVI, and SenseTime-RSVI rolling shutter datasets. The results demonstrate the proposed continuous-time VIO significantly outperforms the existing state-of-the-art VIO methods.

Display Light Panel and Rolling Shutter Image Sensor Based Optical Camera Communication (OCC) Using Frame-Averaging Background Removal and Neural Network

Optical wireless communication (OWC) has emerged as a complementary or alternative technology to the radio-frequency (RF) communication. OWC based on image sensor, which is also called optical camera communication (OCC) has attracted much attention from industrial and academic societies. Here, we discuss several recent OCC technologies. We propose and demonstrate a light emitting diode (LED) light panel and rolling shutter image sensor based OCC system using frame-averaging background removal (FABR) technique, Z-score normalization, and neural network (NN). Here, a driver circuit for the LED display panel based on a bipolar-junction-transistor (BJT) and a metal-oxide-semiconductor field-effect-transistor (MOSFET) is also discussed. It can provide low enough driving frequency from a few Hz to kHz and high bias current for the LED light panel. Experimental results show that the proposed scheme can mitigate the inter-symbol interference (ISI) observed in the rolling shutter pattern produced by the high noise-ratio (NR) of the display contents.

Depth Estimation Under Motion With Single Pair Rolling Shutter Stereo Images

Many methods have been proposed to process stereo matching for rolling shutter image pairs, they treat all pixels from an image pair in an identical way and require additional estimation approaches to estimate motion states of the camera. However, pixels from a rolling shutter image pair naturally have diverse baseline lengths, and motion estimation methods are unstable for one instantaneous image pair input. In this letter, we present a rolling shutter stereo depth estimation pipeline, which can robustly estimate motion states and depth maps by alternating the estimation of the depth maps and refining the motion states from coarse image levels to fine image levels. What is more, we design a novel cost volume building method for rolling shutter image pairs, which adapts depth candidates to the change of baseline lengths for all pixels. We further demonstrate the usability of the proposed method by constructing a new platform, building an outdoor evaluation dataset, and comparing it with baseline methods.

Deep end-to-end rolling shutter rectification.

CMOS sensors employ a row-wise acquisition mechanism while imaging a scene, which can result in undesired motion artifacts known as rolling shutter (RS) distortions in the captured image. Existing single image RS rectification methods attempt to account for these distortions by using either algorithms tailored for a specific class of scenes that warrants information of intrinsic camera parameters or a learning-based framework with known ground truth motion parameters. In this paper, we propose an end-to-end deep neural network for the challenging task of single image RS rectification. Our network consists of a motion block, a trajectory module, a row block, an RS rectification module, and an RS regeneration module (which is used only during training). The motion block predicts the camera pose for every row of the input RS distorted image, while the trajectory module fits estimated motion parameters to a third-order polynomial. The row block predicts the camera motion that must be associated with every pixel in the target, i.e., RS rectified image. Finally, the RS rectification module uses motion trajectory and the output of a row block to warp the input RS image to arrive at a distortion-free image. For faster convergence during training, we additionally use an RS regeneration module that compares the input RS image with the ground truth image distorted by estimated motion parameters. The end-to-end formulation in our model does not constrain the estimated motion to ground truth motion parameters, thereby successfully rectifying the RS images with complex real-life camera motion. Experiments on synthetic and real datasets reveal that our network outperforms prior art both qualitatively and quantitatively.

Rolling Shutter Homography and its Applications.

In this article we study the adaptation of the concept of homography to Rolling Shutter (RS) images. This extension has never been clearly adressed despite the many roles played by the homography matrix in multi-view geometry. We first show that a direct point-to-point relationship on a RS pair can be expressed as a set of 3 to 8 atomic 3x3 matrices depending on the kinematic model used for the instantaneous-motion during image acquisition. We call this group of matrices the RS Homography. We then propose linear solvers for the computation of these matrices using point correspondences. Finally, we derive linear and closed form solutions for two famous problems in computer vision in the case of RS images: image stitching and plane-based relative pose computation. Extensive experiments with both synthetic and real data from public benchmarks show that the proposed methods outperform state-of-art techniques.

The WHU Rolling Shutter Visual-Inertial Dataset

The vast majority of modern consumer cameras employ a rolling shutter (RS) mechanism which has a price and electronic advantage to global shutter (GS). However, in geometric computer vision applications such as visual simultaneous localization and mapping (VSLAM), performances of accuracy and robustness are usually deteriorated due to the rolling shutter effect when using the RS cameras. This paper introduced the Wuhan University Rolling Shutter Visual-Inertial (WHU-RSVI) synthetic dataset for evaluating VSLAM and VI-SLAM (visual-inertial SLAM) methods in which RS cameras or IMU data are typically used. The proposed synthetic dataset contains RS images, time-synchronized GS images, inertial measurement unit (IMU) measurements, and accurate ground truth. It provides camera images with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$640\times 480 $ </tex-math></inline-formula> resolution at 30 Hz and IMU measurements from 90 Hz to 14400 Hz. The cubic B-spline curves are used to model the motion of trajectories. Based on the known trajectories, an image of each pose can be rendered, and the corresponding IMU measurement model is then established. The dataset provides realistic images and IMU measurements by modeling the sensor noise in RGB and IMU data. Two trajectories with three sequences of different motion speeds (i. e., slow, medium and fast corresponding to different rolling shutter effects) are contained in the proposed dataset. Herein, the proposed dataset can be applied to compare the impact of different rolling shutter effects on a specific method.

Robustified Structure from Motion with rolling-shutter camera using straightness constraint

We propose a 3-step method for structure and motion computation from two or more images taken by a one or multiple moving rolling shutter cameras. This work is motivated by the realization that existing reconstruction methods using rolling shutter images do not give satisfactory results or even fail in many configurations due to singularities and degenerate configurations. The first contribution consists in decoupling the rotate ego motion from the remaining parameters by adding a constraint on image curves basing on the a priori knowledge that they correspond to world 3D straight lines with unknown directions. Straight lines frequently appear in man-made environments such as urban or indoor scenes. After introducing the parameterization of a curve projected from a 3D straight line observed by a moving camera using three rolling shutter projection models, we show how to linearly extract angular velocity of each camera by using detected curves. Then we develop a linear method to recover the translational velocities and the motion between the cameras using point-matches, after compensating effects of angular velocity on each image. The second contribution consists in a novel point based bundle adjustment for rolling shutter cameras (C-RSBA) which does not consider a static row index during structure and motion optimization contrarily to existing methods. This enables to refine the parameters obtained thanks to the straightness constraint by avoiding degenerate configurations, thus outperforming existing RSBA methods. The approach was evaluated on both synthetic and real data.

Design and Implementation of a Novel Compatible Encoding Scheme in the Time Domain for Image Sensor Communication.

This paper presents a modulation scheme in the time domain based on On-Off-Keying and proposes various compatible supports for different types of image sensors. The content of this article is a sub-proposal to the IEEE 802.15.7r1 Task Group (TG7r1) aimed at Optical Wireless Communication (OWC) using an image sensor as the receiver. The compatibility support is indispensable for Image Sensor Communications (ISC) because the rolling shutter image sensors currently available have different frame rates, shutter speeds, sampling rates, and resolutions. However, focusing on unidirectional communications (i.e., data broadcasting, beacons), an asynchronous communication prototype is also discussed in the paper. Due to the physical limitations associated with typical image sensors (including low and varying frame rates, long exposures, and low shutter speeds), the link speed performance is critically considered. Based on the practical measurement of camera response to modulated light, an operating frequency range is suggested along with the similar system architecture, decoding procedure, and algorithms. A significant feature of our novel data frame structure is that it can support both typical frame rate cameras (in the oversampling mode) as well as very low frame rate cameras (in the error detection mode for a camera whose frame rate is lower than the transmission packet rate). A high frame rate camera, i.e., no less than 20 fps, is supported in an oversampling mode in which a majority voting scheme for decoding data is applied. A low frame rate camera, i.e., when the frame rate drops to less than 20 fps at some certain time, is supported by an error detection mode in which any missing data sub-packet is detected in decoding and later corrected by external code. Numerical results and valuable analysis are also included to indicate the capability of the proposed schemes.

Vision-aided inertial navigation with rolling-shutter cameras

In this paper, we focus on the problem of pose estimation using measurements from an inertial measurement unit and a rolling-shutter (RS) camera. The challenges posed by RS image capture are typically addressed by using approximate, low-dimensional representations of the camera motion. However, when the motion contains significant accelerations (common in small-scale systems) these representations can lead to loss of accuracy. By contrast, we here describe a different approach, which exploits the inertial measurements to avoid any assumptions on the nature of the trajectory. Instead of parameterizing the trajectory, our approach parameterizes the errors in the trajectory estimates by a low-dimensional model. A key advantage of this approach is that, by using prior knowledge about the estimation errors, it is possible to obtain upper bounds on the modeling inaccuracies incurred by different choices of the parameterization’s dimension. These bounds can provide guarantees for the performance of the method, and facilitate addressing the accuracy–efficiency tradeoff. This RS formulation is used in an extended-Kalman-filter estimator for localization in unknown environments. Our results demonstrate that the resulting algorithm outperforms prior work, in terms of accuracy and computational cost. Moreover, we demonstrate that the algorithm makes it possible to use low-cost consumer devices (i.e. smartphones) for high-precision navigation on multiple platforms.

An Automatic Flicker Detection Method for Embedded Camera Systems

The last few years have seen explosive growth in the market for inexpensive cameras embedded in mobile electronic systems such as cell phones. These embedded cameras often use CMOS image sensors with an electronic rolling shutter. Rolling shutter image sensors are susceptible to highly objectionable artifacts caused by flickering AC lighting fixtures. Flicker artifacts can easily be avoided if the AC power frequency is known. A technique for automatically detecting flicker is presented.