Electronic Rolling Shutter Research Articles

A laser field synchronous scanning imaging system for underwater long-range detection

A novel underwater optical imaging system is proposed to improve underwater image degradation, which is severely deteriorated due to absorption and scattering under artificial illumination. The light source employs a line laser with 520 nm wavelength and a mirror galvanometer that controls the one-dimension scanning of the laser beam to form a planar illumination light field. The camera adopts an sCMOS image sensor with an electronic rolling shutter exposure pattern, compressing the camera’s instantaneous field of view(IFOV) into a narrow strip by reducing the exposure time. Meanwhile, the system controls the light source to irradiate the IFOV of the camera only synchronously, which significantly compressed the overlapping volume of the light path between the camera and the light source so that the backscattered light captured by the camera was dramatically reduced. Experimental results at different turbidity and distances in pool environments show that the laser field synchronous scanning imaging system obtains excellent performance in contrast ratio (CR) and contrast signal-to-noise ratio (CSNR). In the turbidity experiment, the peak CR of the target image improved by a factor of 3.88, and the peak CSNR improved by a factor of 2.33 in 19.6 FTU. In the distance experiment, the peak CR of the target image improved by a factor of 5.57, and the peak CSNR improved by a factor of 1.67 at a distance of 13 m, achieving a clear imaging range of 15 m. This system substantially diminishes the impact of light backscattering on imaging quality, which effectively facilitates further development of the underwater task and renders the system highly applicable in underwater imaging.

Variable Angular Rate Measurement for a Spacecraft Based on the Rolling Shutter Mode of a Star Tracker

Angular rate is a piece of useful information for the attitude control of a spacecraft. The star tracker as a space optical sensor can be used to measure the angular rate of a spacecraft. In this paper, a novel approach is proposed to improve the measurement accuracy of the angular rate during spacecraft rotation. The electronic rolling shutter (RS) imaging mode of the complementary metal-oxide semiconductor (CMOS) image sensor in a star tracker is applied to obtain much higher sampling frequency for reducing the change of the angular rate between the sampling interval. The optic flow vector on the imaging plane is approximated within the second order using three successive star images to reflect the nonlinear effect from the variable angular rate. The experiment is performed to demonstrate the advantage of the new approach for variable angular rate measurement.

Differential SfM and image correction for a rolling shutter stereo rig

• A simple and flexible generalized RS stereo differential SfM algorithm over two consecutive frames. • Two effective and effcient RS-stereo non-linear optimization techniques based on the maximum likelihood criterion. • An RS-stereo image correction method to remove the inaccuracies induced by the RS effect. • The proposed model and method are universal and tractable. Most modern consumer-grade cameras are equipped with an electronic rolling shutter (RS), leading to image distortions when the camera moves during image acquisition. We explore the first structure and motion estimation problem of a dynamic generalized RS stereo camera. Such a general configuration is commonplace in robots and autonomous driving applications. We propose a tractable RS stereo differential structure from motion (SfM) algorithm, taking into account the RS effect during consecutive imaging, which effectively compensates for the RS-stereo image distortion by a linear scaling operation on each optical flow. We further propose embedding the cheirality into RANSAC and develop a robust RS-stereo-aware full-motion estimation framework. We demonstrate that the RS stereo motion and depth map refined by our non-linear optimization schemes within the maximum likelihood criterion can be used for image correction to recover high-quality global shutter (GS) stereo images. Moreover, using the proposed generalized RS stereo differential SfM pipeline, the corrected images produce an accurate 3D scene structure as the ground-truth structure. Extensive experiments on both synthetic and real RS stereo data demonstrate the effectiveness of our model and method in various configurations.

A wide-field and high-resolution lensless compound eye microsystem for real-time target motion perception

Optical measurement systems suffer from a fundamental tradeoff between the field of view (FOV), the resolution and the update rate. A compound eye has the advantages of a wide FOV, high update rate and high sensitivity to motion, providing inspiration for breaking through the constraint and realizing high-performance optical systems. However, most existing studies on artificial compound eyes are limited by complex structure and low resolution, and they focus on imaging instead of precise measurement. Here, a high-performance lensless compound eye microsystem is developed to realize target motion perception through precise and fast orientation measurement. The microsystem splices multiple sub-FOVs formed by long-focal subeyes, images targets distributed in a panoramic range into a single multiplexing image sensor, and codes the subeye aperture array for distinguishing the targets from different sub-FOVs. A wide-field and high resolution are simultaneously realized in a simple and easy-to-manufacture microelectromechanical system (MEMS) aperture array. Moreover, based on the electronic rolling shutter technique of the image sensor, a hyperframe update rate is achieved by the precise measurement of multiple time-shifted spots of one target. The microsystem achieves an orientation measurement accuracy of 0.0023° (3σ) in the x direction and 0.0028° (3σ) in the y direction in a cone FOV of 120° with an update rate ~20 times higher than the frame rate. This study provides a promising approach for achieving optical measurements with comprehensive high performance and may have great significance in various applications, such as vision-controlled directional navigation and high-dynamic target tracking, formation and obstacle avoidance of unmanned aerial vehicles.

Tunable Laser Absorption Imaging for 2D Gas Measurement With an Electronic Rolling Shutter Camera

High spatial resolution two-dimensional (2D) imaging of gas flow is essential for better understanding and characterization of a combustion field. We propose a method for 2D gas measurement with high spatial resolution by combining tunable laser absorption spectroscopy and an imager with an electronic rolling shutter through their common feature of time scanning. The method was demonstrated by oxygen measurement using a 764nm vertical cavity surface emitting laser and a low-cost CMOS camera. 2D imaging of the oxygen concentration distribution were obtained at a frame rate of 15fps with a horizontal spatial resolution of about <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$28\mu \text{m}$ </tex-math></inline-formula> . This method enjoys simple structure, low cost, high spatial resolution and shows potential in the 2D and 3D reconstruction of the combustion field.

Compressive temporal imaging using a rolling shutter camera array.

In this work, we present a novel camera array that exploits the electronic rolling shutter to achieve high speed compressive temporal imaging. Traditional compressive temporal imaging makes use of mechanical coded apertures, despite implementation and calibration challenges. Instead, we propose to model the inherent spatial and temporal coding provided by the distinctive rolling shutter sampling from each camera of the array as a compressive temporal imaging system matrix. Thus, we can recover a high speed video from a set of snapshots from the camera array by using compressive sensing reconstruction algorithms. We present both simulation and experimental results for a 4-camera array system with different orientation angles, reconstructing up to 56 high-speed sub-frames from a set of simultaneously triggered snapshots from the array, achieving a compression rate of up to 14X.

Luminescence lifetime imaging using a cellphone camera with an electronic rolling shutter

Luminescence lifetime imaging systems are typically complex and expensive instruments targeting bench-top applications. We present a low-cost approach for video frame rate luminescence lifetime imaging on the microsecond scale based on a cellphone camera. The luminescence lifetime is mapped by measuring the phase shift of optical fringes captured by a camera with an electronic rolling shutter at 30 frames per second. Luminescence lifetime imaging is validated by observing a platinum(II)-5,10,15,20-tetrakis-(2,3,4,5,6-pentafluorphenyl)-porphyrin (PtTFPP)/polystyrene film with a non-uniform lifetime distribution. The experimental results indicate a lifetime resolution of 580 ns when a lifetime of 22 µs is measured. This device is portable, low in cost, and shows potential in point-of-care diagnostic and environment applications that measure long-lifetime luminescence.

A SYSTEM FOR MONITORING OF UAV CAMERA ORIENTATION: DESIGN AND INITIAL ANALYSIS

Abstract. Many unmanned aerial vehicles (UAV) that are used for aerial mapping are equipped with consumer-grade digital cameras, which use CMOS (Complementary metal–oxide–semiconductor) image sensors. Majority of these sensors capture images using an electronic rolling shutter, which can cause distortions on the image if either the camera or the captured objects are moving. This phenomenon is usually ignored in aerial mapping with UAVs in practice. However, there is a lack of published research papers that would prove the effect can be neglected. In this paper, we present the design of the system for monitoring UAV camera orientation. Furthermore, the calibration process to get correct and reliable readings is described. The initial analysis of the data is focused on assessing the accuracy that can be achieved using the proposed system. The main component of our system is a MEMS (Microelectromechanical system) gyroscope. It was selected for its low weight and size, low price and high sampling rates which are all very beneficial characteristics for a system, mounted on a UAV. In a paper, a working prototype is presented that uses the selected MEMS gyroscope connected to a single-board computer. The presented initial analysis of collected data shows, that the system would be capable to indirectly detect the image distortions, caused by camera orientation changes during exposure, in the range of typical ground sample distance (GSD).

Shuttering methods and the artifacts they produce

When exposure times were measured in minutes, the opening and closing of the shutter was essentially instantaneous. As more sensitive films and brighter optics became available, exposure times decreased, the travel time of the shutter mechanism became increasingly significant, and artifacts became visible. Perhaps the best-known shutter artifacts are the spatio-temporal distortions associated with photographing moving subjects using a focal-plane shutter or sequential electronic sampling of pixels (electronic rolling shutter). However, the shutter mechanism also can cause banding with flickering light sources and strange artifacts in out-of-focus regions (bokeh); it can even impact resolution. This paper experimentally evaluates and discusses the artifacts caused by leaf, focal plane, electronic first curtain, and fully electronic sequential-readout shuttering.

帘幕式CMOS全局曝光成像技术

帘幕式CMOS全局曝光成像技术

Measuring frequency of one-dimensional vibration with video camera using electronic rolling shutter

Cameras offer a unique capability of collecting high density spatial data from a distant scene of interest. They can be employed as remote monitoring or inspection sensors to measure vibrating objects because of their commonplace availability, simplicity, and potentially low cost. A defect of vibrating measurement with the camera is to process the massive data generated by camera. In order to reduce the data collected from the camera, the camera using electronic rolling shutter (ERS) is applied to measure the frequency of one-dimensional vibration, whose frequency is much higher than the speed of the camera. Every row in the image captured by the ERS camera records the vibrating displacement at different times. Those displacements that form the vibration could be extracted by local analysis with sliding windows. This methodology is demonstrated on vibrating structures, a cantilever beam, and an air compressor to identify the validity of the proposed algorithm. Suggestions for applications of this methodology and challenges in real-world implementation are given at last.

Novel approach to improve the attitude update rate of a star tracker

The star tracker is widely used in attitude control systems of spacecraft for attitude measurement. The attitude update rate of a star tracker is important to guarantee the attitude control performance. In this paper, we propose a novel approach to improve the attitude update rate of a star tracker. The electronic Rolling Shutter (RS) imaging mode of the complementary metal-oxide semiconductor (CMOS) image sensor in the star tracker is applied to acquire star images in which the star spots are exposed with row-to-row time offsets, thereby reflecting the rotation of star tracker at different times. The attitude estimation method with a single star spot is developed to realize the multiple attitude updates by a star image, so as to reach a high update rate. The simulation and experiment are performed to verify the proposed approaches. The test results demonstrate that the proposed approach is effective and the attitude update rate of a star tracker is increased significantly.

Measuring the Angular Velocity of a Propeller with Video Camera Using Electronic Rolling Shutter

Noncontact measurement for rotational motion has advantages over the traditional method which measures rotational motion by means of installing some devices on the object, such as a rotary encoder. Cameras can be employed as remote monitoring or inspecting sensors to measure the angular velocity of a propeller because of their commonplace availability, simplicity, and potentially low cost. A defect of the measurement with cameras is to process the massive data generated by cameras. In order to reduce the collected data from the camera, a camera using ERS (electronic rolling shutter) is applied to measure angular velocities which are higher than the speed of the camera. The effect of rolling shutter can induce geometric distortion in the image, when the propeller rotates during capturing an image. In order to reveal the relationship between the angular velocity and the image distortion, a rotation model has been established. The proposed method was applied to measure the angular velocities of the two-blade propeller and the multiblade propeller. The experimental results showed that this method could detect the angular velocities which were higher than the camera speed, and the accuracy was acceptable.

Flicker removal for CMOS wide dynamic range imaging based on alternating current component analysis

This paper presents a novel wide dynamic range system to reduce flickering artifact produced by a high speed electronic rolling shutter in a CMOS image sensor. The proposed algorithm removes flicker artifact in the shortexposure image using a correlation between the pair of two differently exposed images to extend the dynamic range. Since the proposed method overcomes the electronic shutter problem in a conventional wide dynamic range imaging system, it can remove the flicker artifact under an alternating current light source such as a fluorescent bulb.

An implementation method based on ERS imaging mode for sun sensor with 1 kHz update rate and 1″ precision level

Stringent attitude determination accuracy through a high bandwidth is required for the development of the advanced space technologies, such as earth observation and laser communication. In this work, we presented a novel proposal for a digital sun sensor with high accuracy, large Field of View (FOV) and ultra-high data update rate. The Electronic Rolling Shutter (ERS) imaging mode of an APS CMOS detector was employed and an "amplifier factor" was introduced to improve the data update rate significantly. Based on the idea of the multiplexing detector, a novel mask integrated with two kinds of aperture patterns was also introduced to implement its distinctive performance of high precision and large FOV. Test results show that the ERS based sun sensor is capable of achieving the data update rate of 1 kHz and precision of 1.1″ (1σ) within a 105° × 105° FOV. The digital sun sensor can play an important role in precise attitude determination and provide a broader application for high accuracy satellites.

Feasibility of employing a smartphone as the payload in a photogrammetric UAV system

Smartphones can be operated in a 3G network environment at any time or location, and they also cost less than existing photogrammetric UAV systems, providing high-resolution images and 3D location and attitude data from a variety of built-in sensors. This study aims to assess the feasibility of using a smartphone as the payload for a photogrammetric UAV system. To carry out the assessment, a smartphone-based photogrammetric UAV system was developed and utilized to obtain image, location, and attitude data under both static and dynamic conditions. The accuracy of the location and attitude data obtained and sent by this system was then evaluated. The smartphone images were converted into ortho-images via image triangulation, which was carried out both with and without consideration of the interior orientation (IO) parameters determined by camera calibration. In the static experiment, when the IO parameters were taken into account, the triangulation results were less than 1.28pixels (RMSE) for all smartphone types, an improvement of at least 47% compared with the case when IO parameters were not taken into account. In the dynamic experiment, on the other hand, the accuracy of smartphone image triangulation was not significantly improved by considering IO parameters. This was because the electronic rolling shutter within the complementary metal-oxide semiconductor (CMOS) sensor built into the smartphone and the actuator for the voice coil motor (VCM)-type auto-focusing affected by the vibration and the speed of the UAV, which is likely to have a negative effect on image-based digital elevation model (DEM) generation. However, considering that these results were obtained using a single smartphone, this suggests that a smartphone is not only feasible as the payload for a photogrammetric UAV system but it may also play a useful role when installed in existing UAV systems.

Analysis and Compensation of Rolling Shutter Effect

Due to the sequential-readout structure of complementary metal-oxide semiconductor image sensor array, each scanline of the acquired image is exposed at a different time, resulting in the so-called electronic rolling shutter that induces geometric image distortion when the object or the video camera moves during image capture. In this paper, we propose an image processing technique using a planar motion model to address the problem. Unlike previous methods that involve complex 3-D feature correspondences, a simple approach to the analysis of inter- and intraframe distortions is presented. The high-resolution velocity estimates used for restoring the image are obtained by global motion estimation, BEzier curve fitting, and local motion estimation without resort to correspondence identification. Experimental results demonstrate the effectiveness of the algorithm.

An Automatic Flicker Detection Method for Embedded Camera Systems

The last few years have seen explosive growth in the market for inexpensive cameras embedded in mobile electronic systems such as cell phones. These embedded cameras often use CMOS image sensors with an electronic rolling shutter. Rolling shutter image sensors are susceptible to highly objectionable artifacts caused by flickering AC lighting fixtures. Flicker artifacts can easily be avoided if the AC power frequency is known. A technique for automatically detecting flicker is presented.