Rolling Shutter Distortion Research Articles

Fly-scan high-throughput coded ptychographic microscopy via active micro-vibration and rolling-shutter distortion correction.

Recent advancements in ptychography have demonstrated the potential of coded ptychography (CP) for high-resolution optical imaging in a lensless configuration. However, CP suffers imaging throughput limitations due to scanning inefficiencies. To address this, we propose what we believe is a novel 'fly-scan' scanning strategy utilizing two eccentric rotating mass (ERM) vibration motors for high-throughput coded ptychographic microscopy. The intrinsic continuity of the 'fly-scan' technique effectively eliminates the scanning overhead typically encountered during data acquisition. Additionally, its randomized scanning trajectory considerably reduces periodic artifacts in image reconstruction. We also developed what we believe to be a novel rolling-shutter distortion correction algorithm to fix the rolling-shutter effects. We built up a low-cost, DIY-made prototype platform and validated our approach with various samples including a resolution target, a quantitative phase target, a thick potato sample and biospecimens. The reported platform may offer a cost-effective and turnkey solution for high-throughput bio-imaging.

Rolling Shutter Distortion Modeling and Corrective Algorithm for Agility Improvement of Star Sensors

Rolling Shutter Distortion Modeling and Corrective Algorithm for Agility Improvement of Star Sensors

A star identification algorithm for rolling shutter exposure based on Hough transform

The star identification algorithm usually identifies stars by angular distance matching. However, under high dynamic conditions, the rolling shutter effect distorts the angular distances between the measured and true star positions, leading to plethoric false matches and requiring complex and time-consuming verification for star identification. Low identification rate hinders the application of low-noise and cost-effective rolling shutter image sensors. In this work, we first study a rolling shutter distortion model of angular distances between stars, and then propose a novel three-stage star identification algorithm to identify distorted star images captured by the rolling shutter star sensor. The first stage uses a modified grid algorithm with adaptive error tolerance and an expanded pattern database to efficiently eliminate spurious matches. The second stage performs angular velocity estimation based on Hough transform to verify the matches that follow the same distortion pattern. The third stage applies a rolling shutter error correction method for further verification. Both the simulation and night sky image test demonstrate the effectiveness and efficiency of our algorithm under high dynamic conditions. The accuracy of angular velocity estimation method by Hough transform is evaluated and the root mean square error is below 0.5 (°)/s. Our algorithm achieves a 95.7% identification rate at an angular velocity of 10 (°)/s, which is much higher than traditional algorithms.

Fast Rolling Shutter Correction in the Wild.

This paper addresses the problem of rolling shutter correction (RSC) in uncalibrated videos. Existing works remove rolling shutter (RS) distortion by explicitly computing the camera motion and depth as intermediate products, followed by motion compensation. In contrast, we first show that each distorted pixel can be implicitly rectified back to the corresponding global shutter (GS) projection by rescaling its optical flow. Such a point-wise RSC is feasible with both perspective and non-perspective cases without the pre-knowledge of the camera used. Besides, it allows a pixel-wise varying direct RS correction (DRSC) framework that handles locally varying distortion caused by various sources, such as camera motion, moving objects, and even highly varying depth scenes. More importantly, our approach is an efficient CPU-based solution that enables undistorting RS videos in real-time (40fps for 480p). We evaluate our approach across a broad range of cameras and video sequences, including fast motion, dynamic scenes, and non-perspective lenses, demonstrating the superiority of our proposed approach over state-of-the-art methods in both effectiveness and efficiency. We also evaluated the ability of the RSC results to serve for downstream 3D analysis, such as visual odometry and structure-from-motion, which verifies preference for the output of our algorithm over other existing RSC methods.

Spatial-Temporal Measurement Alignment of Rolling Shutter Camera and LiDAR

This letter proposes a novel method to fuse the asynchronized outputs from a rolling shutter camera and a spinning LiDAR mounted on a moving vehicle. Compared with traditional methods only relying on intrinsic/extrinsic calibration, the proposed method takes the ego-motion, rolling shutter distortion, and occlusion into the fusion model. In essence, the method estimates the temporal offset between LiDAR and the camera by minimizing the reprojection error of the row values between consecutive iterations. Hence, it can precisely project the 3-D LiDAR points into the image pixels from a rolling shutter camera moving at high speed, which is critical for a multisensory autonomous driving system. Field tests based on a vehicle prototype verify the performance enhancement of the proposed method, particularly in high-speed conditions. This method is especially helpful for engineering practices on low-level sensor fusion of LiDAR and rolling shutter camera.

Error Correction of Rolling Shutter Effect for Star Sensor Based on Angular Distance Invariance Using Single Frame Star Image

When the high dynamic star sensor works in rolling shutter exposure mode, the star image distortion will be caused by the rotation of the star sensor and the time delay between each row of the sensor chip during the imaging process, which increases the centroid positioning error of the star sensor. Therefore, we propose a rolling shutter distortion correction method based on angular distance information of a single frame star image. Using the variation law of star centroid position in single frame star image and the angular distance information between stars, we construct the error compensation model of rolling shutter distortion. Then, the simulation results show that when the angular velocity of the star sensor is less than 3°/s, and the maximum angular distance error after compensation does not exceed 0.0024°. Finally, the field experimental results show that the angular distance error is equally reduced by 90.38% under the typical working condition of angular velocity 3°/s. These results provide a new idea to solve the rolling shutter effect by using a single frame star image, which is conducive to further improve the measurement accuracy of the star sensor under dynamic conditions.

Rolling-Shutter-stereo-aware motion estimation and image correction

The vast majority of modern consumer-grade cameras employ the Rolling Shutter (RS) mechanism, which often produces geometrically distorted images due to inter-row delay during imaging, that needs to be properly handled. While several models and solutions have been proposed for the motion estimation problem of the monocular RS camera, the corresponding motion estimation model and algorithm have not been developed for the standard RS stereo setup. This is primarily due to the temporal-dynamic nature of the RS camera, which results in scanline-varying camera poses. This paper fills in this gap by developing two tractable RS stereo models under constant velocity motion and constant acceleration motion respectively, which only need to simply scale the optical flow vector. We further propose two corresponding RS stereo motion estimation algorithms to accurately compute the relative pose of the RS stereo camera between two consecutive frames. As a result of utilizing these RS stereo models associated with the parallel visual rays, our proposed methods offer better disambiguation of the translation and rotation, hence leading to significantly improved performance compared with the Global Shutter (GS) stereo model. Furthermore, we develop an RS stereo image correction method to remove RS distortions and recover high-quality GS images by conducting motion compensation. Experimental results on both simulated data and real RS stereo images show the superiority of our proposed models and methods.

Simultaneous Video Stabilization and Rolling Shutter Removal.

Due to the delay in the row-wise exposure and the lack of stable support when a photographer holds a CMOS camera, video jitter and rolling shutter distortion are closely coupled degradations in the captured videos. However, previous methods have rarely considered both phenomena and usually treat them separately, with stabilization approaches that are unable to handle the rolling shutter effect and rolling shutter removal algorithms that are incapable of addressing motion shake. To tackle this problem, we propose a novel method that simultaneously stabilizes and rectifies a rolling shutter shaky video. The key issue is to estimate both inter-frame motion and intra-frame motion. Specifically, for each pair of adjacent frames, we first estimate a set of spatially variant inter-frame motions using a neighbor-motion-aware local motion model, where the classical mesh-based model is improved by introducing a new constraint to enhance the neighbor motion consistency. Then, different from other 2D rolling shutter removal methods that assume the pixels in the same row have a single intra-frame motion, we build a novel mesh-based intra-frame motion calculation model to cope with the depth variation in a mesh row and obtain more faithful estimation results. Finally, temporal and spatial motion constraints and an adaptive weight assignment strategy are considered together to generate the optimal warping transformations for different motion situations. Experimental results demonstrate the effectiveness and superiority of the proposed method when compared with other state-of-the-art methods.

Deep end-to-end rolling shutter rectification.

CMOS sensors employ a row-wise acquisition mechanism while imaging a scene, which can result in undesired motion artifacts known as rolling shutter (RS) distortions in the captured image. Existing single image RS rectification methods attempt to account for these distortions by using either algorithms tailored for a specific class of scenes that warrants information of intrinsic camera parameters or a learning-based framework with known ground truth motion parameters. In this paper, we propose an end-to-end deep neural network for the challenging task of single image RS rectification. Our network consists of a motion block, a trajectory module, a row block, an RS rectification module, and an RS regeneration module (which is used only during training). The motion block predicts the camera pose for every row of the input RS distorted image, while the trajectory module fits estimated motion parameters to a third-order polynomial. The row block predicts the camera motion that must be associated with every pixel in the target, i.e., RS rectified image. Finally, the RS rectification module uses motion trajectory and the output of a row block to warp the input RS image to arrive at a distortion-free image. For faster convergence during training, we additionally use an RS regeneration module that compares the input RS image with the ground truth image distorted by estimated motion parameters. The end-to-end formulation in our model does not constrain the estimated motion to ground truth motion parameters, thereby successfully rectifying the RS images with complex real-life camera motion. Experiments on synthetic and real datasets reveal that our network outperforms prior art both qualitatively and quantitatively.

A Stacked Back Side-Illuminated Voltage Domain Global Shutter CMOS Image Sensor with a 4.0 μm Multiple Gain Readout Pixel †.

A backside-illuminated complementary metal-oxide-semiconductor (CMOS) image sensor with 4.0 μm voltage domain global shutter (GS) pixels has been fabricated in a 45 nm/65 nm stacked CMOS process as a proof-of-concept vehicle. The pixel components for the photon-to-voltage conversion are formed on the top substrate (the first layer). Each voltage signal from the first layer pixel is stored in the sample-and-hold capacitors on the bottom substrate (the second layer) via micro-bump interconnection to achieve a voltage domain GS function. The two sets of voltage domain storage capacitor per pixel enable a multiple gain readout to realize single exposure high dynamic range (SEHDR) in the GS operation. As a result, an 80dB SEHDR GS operation without rolling shutter distortions and motion artifacts has been achieved. Additionally, less than −140dB parasitic light sensitivity, small noise floor, high sensitivity and good angular response have been achieved.

A two-step approach for the correction of rolling shutter distortion in UAV photogrammetry

The use of consumer grade unmanned aerial vehicles (UAV) is becoming more and more ubiquitous in photogrammetric applications. A large proportion of consumer grade UAVs are equipped with CMOS image sensor and rolling shutter. When imaging with a rolling shutter camera, the image sensor is exposed line by line, which can introduce additional distortions in image space since the UAV navigates at a relatively high speed during aerial acquisitions. In this paper, we propose (1) an approach to calibrate the readout time of rolling shutter camera, (2) a two-step method to correct the image distortion introduced by this effect. The two-step method makes assumption that during exposure, the change of camera orientation is negligible with respect to the change of camera position, which is often the case when camera is fixed on a stabilized mount. Firstly, the camera velocity is estimated from the results of an initial bundle block adjustment; then, one camera pose per scan-line of the image sensor is recovered and image observations are corrected. To evaluate the performance of the proposed method, four datasets of block and corridor configurations are acquired with the DJI Mavic 2 Pro and its original Hasselbald L1D-20c camera. The proposed method is implemented in MicMac, a free, open-source photogrammetric software; comparisons are carried out with other two mainstream software, AgiSoft MetaShape and Pix4D, which also have the functionality of rolling shutter effect correction. For block configuration datasets, the three software give comparable results. AgiSoft Metashape and Pix4D are sensitive to the flight configuration and encounter difficulties when processing datasets in corridor configurations. The proposed method shows good robustness both in block and corridor configurations, and is the only method that works in corridor configuration. After the application of the rolling shutter effect correction, the 3D accuracy is improved by 30–60% in block configuration and 15–25% in corridor configuration. A further improvement can be expected if a precise dating of image is available or if the camera positions can be directly extracted from GNSS data.

An Affine Motion Model for Removing Rolling Shutter Distortions

Rolling shutter distortions degrade the quality of videos captured by hand-held cameras. This letter proposes an affine motion model for removing rolling shutter distortions. The model represents the image motion during image capture as a sequence of affine transformations and computes the composition of these affine transformations precisely. Because an affine transformation can be represented by a neural network with one layer of linear neurons, the motion model can be represented by a multilayer neural network of linear neurons. Thus, the backpropagation algorithm can be used to improve the efficiency of the optimization process that estimates the model parameters. The proposed model is calibration-free. It is more general than other rolling shutter motion models because it only assumes that the velocity of the image during image acquisition is piecewise constant. Experimental results demonstrate that the model is more accurate than two state-of-the-art models and that the model parameters can be estimated efficiently.

Efficient Rolling Shutter Distortion Removal using Hierarchical Block-based Motion Estimation

This paper reports an efficient algorithm for accurate rolling shutter distortion removal. A hierarchical global motion estimation approach for a group of blocks reduces the level of computation by three orders of magnitude. In addition, the motion of each scanline is determined accurately by averaging two candidates obtained through cubic spline interpolation. The experimental results show that the proposed method produces accurate motion information with significant computation reduction and corrects the rolling shutter distortion effectively.

Rolling shutter distortion removal based on curve interpolation

In videos shot by rolling shutter cameras, rapid camera or object motion may cause obvious image distortions. This paper proposes a rectification method that removes rolling shutter distortions by performing curve interpolation for each pixel. This method does not require camera calibration, and it can handle camera and object motion simultaneously. The method only assumes that the motion is smooth. If camera motion alone causes distortions, a variation of this method requiring less computation can be used. Experiments on synthetic datasets and real videos show that the methods can remove rolling shutter distortions effectively.

Suppressing rolling-shutter distortion of CMOS image sensors by motion vector detection

This paper focuses on the rolling shutter distortion of CMOS image sensor coming from its unique readout mechanism as the main cause for image degradation when there are fast-moving objects. This paper proposes a post image processing scheme based on motion vector detection to suppress the rolling shutter distortion. Motion vector detection is performed based on an optical flow method at a reasonable computational complexity. A practical implementation scheme is also described.