Shutter Camera Research Articles

PHOTOGRAMMETRIC ACCURACY AND MODELING OF ROLLING SHUTTER CAMERAS

Unmanned aerial vehicles (UAVs) are becoming increasingly popular in professional mapping for stockpile analysis, construction site monitoring, and many other applications. Due to their robustness and competitive pricing, consumer UAVs are used more and more for these applications, but they are usually equipped with rolling shutter cameras. This is a significant obstacle when it comes to extracting high accuracy measurements using available photogrammetry software packages. In this paper, we evaluate the impact of the rolling shutter cameras of typical consumer UAVs on the accuracy of a 3D reconstruction. Hereto, we use a beta-version of the Pix4Dmapper 2.1 software to compare traditional (non rolling shutter) camera models against a newly implemented rolling shutter model with respect to both the accuracy of geo-referenced validation points and to the quality of the motion estimation. Multiple datasets have been acquired using popular quadrocopters (DJI Phantom 2 Vision+, DJI Inspire 1 and 3DR Solo) following a grid flight plan. For comparison, we acquired a dataset using a professional mapping drone (senseFly eBee) equipped with a global shutter camera. The bundle block adjustment of each dataset shows a significant accuracy improvement on validation ground control points when applying the new rolling shutter camera model for flights at higher speed (8m=s). Competitive accuracies can be obtained by using the rolling shutter model, although global shutter cameras are still superior. Furthermore, we are able to show that the speed of the drone (and its direction) can be solely estimated from the rolling shutter effect of the camera.

Fast structured illumination microscopy using rolling shutter cameras

Spatial light modulators (SLM) update in a synchronous manner, whereas the data readout process in fast structured illumination systems is usually done using a rolling shutter camera with asynchronous readout. In structured illumination microscopy (SIM), this leads to synchronization problems causing a speed limit for fast acquisition. In this paper we present a configuration to overcome this limit by exploiting the extremely fast SLM display and dividing it into several segments along the direction of the rolling shutter of the sCMOS camera and displaying multiple SLM frames per camera acquisition. The sCMOS runs in continuous rolling shutter mode and the SLM keeps the readout-line always inside a dark region presenting different SIM patterns before and after the readout/start-exposure line.Using this approach, we reached a raw frame rate of 714 frames per second (fps) resulting in a two-beam SIM acquisition rate of 79 fps with a region of interest (ROI) of 16.5 × 16.5 μm2.

On the Mathematical Foundations of Computational Photography

Flutter shutter (coded exposure) cameras allow to extend indefinitely the exposure time for uniform motion blurs. Recently, Tendero et al. (SIAM J Imaging Sci 6(2):813---847, 2013) proved that for a fixed known velocity v the gain of a flutter shutter in terms of root means square error (RMSE) cannot exceeds a 1.1717 factor compared to an optimal snapshot. The aforementioned bound is optimal in the sense that this 1.1717 factor can be attained. However, this disheartening bound is in direct contradiction with the recent results by Cossairt et al. (IEEE Trans Image Process 22(2), 447---458, 2013). Our first goal in this paper is to resolve mathematically this discrepancy. An interesting question was raised by the authors of reference (IEEE Trans Image Process 22(2), 447---458, 2013). They state that the "gain for computational imaging is significant only when the average signal level J is considerably smaller than the read noise variance $$\sigma _r^2$$?r2" (Cossairt et al., IEEE Trans Image Process 22(2), 447---458, 2013, p. 5). In other words, according to Cossairt et al. (IEEE Trans Image Process 22(2), 447---458, 2013) a flutter shutter would be more efficient when used in low light conditions e.g., indoor scenes or at night. Our second goal is to prove that this statement is based on an incomplete camera model and that a complete mathematical model disproves it. To do so we propose a general flutter shutter camera model that includes photonic, thermal (The amplifier noise may also be mentioned as another source of background noise, which can be included w.l.o.g. in the thermal noise) and additive [The additive (sensor readout) noise may contain other components such as reset noise and quantization noise. We include them w.l.o.g. in the readout.] (sensor readout, quantification) noises of finite variances. Our analysis provides exact formulae for the mean square error of the final deconvolved image. It also allows us to confirm that the gain in terms of RMSE of anyflutter shutter camera is bounded from above by 1.1776 when compared to an optimal snapshot. The bound is uniform with respect to the observation conditions and applies for any fixed known velocity. Incidentally, the proposed formalism and its consequences also apply to e.g., the Levin et al. motion-invariant photography (ACM Trans Graphics (TOG), 27(3):71:1---71:9, 2008; Method and apparatus for motion invariant imag- ing, October 1 2009. US Patent 20,090,244,300, 2009) and variant (Cho et al. Motion blur removal with orthogonal parabolic exposures, 2010). In short, we bring mathematical proofs to the effect of contradicting the claims of Cossairt et al. (IEEE Trans Image Process 22(2), 447---458, 2013). Lastly, this paper permits to point out the kind of optimization needed if one wants to turn the flutter shutter into a useful imaging tool.

Unsynchronized structured light

Various Structured Light (SL) methods are used to capture 3D range images, where a number of binary or continuous light patterns are sequentially projected onto a scene of interest, while a digital camera captures images of the illuminated scene. All existing SL methods require the projector and camera to be hardware or software synchronized, with one image captured per projected pattern. A 3D range image is computed from the captured images. The two synchronization methods have disadvantages, which limit the use of SL methods to niche industrial and low quality consumer applications. Unsynchronized Structured Light (USL) is a novel SL method which does not require synchronization of pattern projection and image capture. The light patterns are projected and the images are captured independently, at constant, but possibly different, frame rates. USL synthesizes new binary images as would be decoded from the images captured by a camera synchronized to the projector, reducing the subsequent computation to standard SL. USL works both with global and rolling shutter cameras. USL enables most burst-mode-capable cameras, such as modern smartphones, tablets, DSLRs, and point-and-shoots, to function as high quality 3D snapshot cameras. Beyond the software, which can run in the devices, a separate SL Flash, able to project the sequence of patterns cyclically, during the acquisition time, is needed to enable the functionality.

Homogeneous codes for energy-efficient illumination and imaging

Programmable coding of light between a source and a sensor has led to several important results in computational illumination, imaging and display. Little is known, however, about how to utilize energy most effectively, especially for applications in live imaging. In this paper, we derive a novel framework to maximize energy efficiency by "homogeneous matrix factorization" that respects the physical constraints of many coding mechanisms (DMDs/LCDs, lasers, etc. ). We demonstrate energy-efficient imaging using two prototypes based on DMD and laser illumination. For our DMD-based prototype, we use fast local optimization to derive codes that yield brighter images with fewer artifacts in many transport probing tasks. Our second prototype uses a novel combination of a low-power laser projector and a rolling shutter camera. We use this prototype to demonstrate never-seen-before capabilities such as (1) capturing live structured-light video of very bright scenes---even a light bulb that has been turned on; (2) capturing epipolar-only and indirect-only live video with optimal energy efficiency; (3) using a low-power projector to reconstruct 3D objects in challenging conditions such as strong indirect light, strong ambient light, and smoke; and (4) recording live video from a projector's---rather than the camera's---point of view.

Effect of surrounding gas condition on surface integrity in micro-drilling of SiC by ns pulsed laser

The influence of the surrounding gas conditions on the surface integrity in the micro-drilling of silicon carbide was experimentally investigated using ns pulsed laser of 266 nm wavelength. Moreover, micro-machining characteristics were observed using high-speed shutter and video cameras in the micro-drilling of silicon carbide. The size and intensity of the laser-induced plasma were larger, and the plasma affected area was larger and deeper in argon than that in air. Although the intensity of the plasma was lower in helium than that in other gases, the surface around the drilled hole was roughened by the spread of the plasma in the vicinity of the drilled hole. Debris was removed along the flow field generated by laser shot in the opposite direction to the laser irradiation. The gas flow behavior and the spectrum and intensity of the laser-induced plasma were influenced by the surrounding gas type and pressure. The appearance of plasma generation affected the surface integrity at the circumference of the drilled hole, and the surface integrity was improved by reducing the pressure.

A Spline-Based Trajectory Representation for Sensor Fusion and Rolling Shutter Cameras

The use of multiple sensors for ego-motion estimation is an approach often used to provide more accurate and robust results. However, when representing ego-motion as a discrete series of poses, fus...

Measuring the Noise of Digital Imaging Sensors by Stacking Raw Images Affected by Vibrations and Illumination Flickering

This paper discusses camera noise estimation from a series of raw images of an arbitrary natural static scene, acquired with the same camera settings. Although it seems natural to characterize noise from the random time fluctuation of pixel intensity, it turns out that these fluctuations may also be caused by illumination flickering and mechanical microvibrations affecting the camera. In this context, the contributions are twofold. First, a theoretical model of image formation in the presence of illumination flickering and of vibrations is discussed. This parametric model is based on a Cox process. It is shown that illumination flickering changes the standard affine relation between noise variance and average intensity to a quadratic relation. Second, under these conditions an algorithm is proposed to estimate the main parameters governing sensor noise, namely the gain, the offset, and the readout noise. The rolling shutter effect, which potentially affects the output of any focal-plane shutter camera, is...

Online camera-gyroscope autocalibration for cell phones.

The gyroscope is playing a key role in helping estimate 3D camera rotation for various vision applications on cell phones, including video stabilization and feature tracking. Successful fusion of gyroscope and camera data requires that the camera, gyroscope, and their relative pose to be calibrated. In addition, the timestamps of gyroscope readings and video frames are usually not well synchronized. Previous paper performed camera-gyroscope calibration and synchronization offline after the entire video sequence has been captured with restrictions on the camera motion, which is unnecessarily restrictive for everyday users to run apps that directly use the gyroscope. In this paper, we propose an online method that estimates all the necessary parameters, whereas a user is capturing video. Our contributions are: 1) simultaneous online camera self-calibration and camera-gyroscope calibration based on an implicit extended Kalman filter and 2) generalization of the multiple-view coplanarity constraint on camera rotation in a rolling shutter camera model for cell phones. The proposed method is able to estimate the needed calibration and synchronization parameters online with all kinds of camera motion and can be embedded in gyro-aided applications, such as video stabilization and feature tracking. Both Monte Carlo simulation and cell phone experiments show that the proposed online calibration and synchronization method converge fast to the ground truth values.

High-speed video capture by a single flutter shutter camera using three-dimensional hyperbolic wavelets

Based on the consideration of easy achievement in modern sensors, this paper further exploits the possibility of the recovery of high-speed video (HSV) by a single flutter shutter camera. Taking into account different degrees of smoothness along the spatial and temporal dimensions of HSV, this paper proposes to use a three-dimensional hyperbolic wavelet basis based on Kronecker product to jointly model the spatial and temporal redundancy of HSV. Besides, we incorporate the total variation of temporal correlations in HSV as a prior knowledge to further enhance our reconstruction quality. We recover the underlying HSV frames from the observed low-speed coded video by solving a convex minimization problem. The experimental results on simulated and real-world videos both demonstrate the validity of the proposed method.

“Seeing” ENF: Power-Signature-Based Timestamp for Digital Multimedia via Optical Sensing and Signal Processing

Electric Network Frequency (ENF) fluctuates slightly over time from its nominal value of 50 Hz/60 Hz. The fluctuations in ENF remain consistent across the entire power grid including when measured at physically distant geographical locations. The light intensity from such indoor lighting as fluorescent lamps and incandescent bulbs, which are connected to the power mains, varies in accordance with the ENF, and the fluctuations can be recorded using visual sensors. In this paper, mechanisms using optical sensors and video cameras to record and validate the presence of ENF fluctuations in indoor lighting are presented. Spectrogram and subspace-based signal processing techniques are applied to such recordings to extract the ENF signal by estimating its instantaneous frequencies as a function of time. A high correlation is observed between the ENF fluctuations obtained from indoor lighting and that of the ENF signal captured directly from the power mains supply. A similar mechanism is then used to demonstrate the presence of ENF signals in video recordings taken in different geographical areas. Experimental results show that ENF signals are present in visual recordings made in different geographical areas and can be used as a natural timestamp for optical sensor recordings and video surveillance recordings conducted in indoor lighting environments. Robustness of ENF fluctuation traces under strong compression and CMOS rolling shutter cameras is discussed. Applications of the ENF signal analysis to tampering detection of surveillance video recordings and forensic binding of the audio and visual track of a video are also demonstrated. An analytical model based on an autoregressive process is also developed for ENF signals, and the effectiveness of using innovation sequences from the model for timestamp verification is demonstrated.

On a novel low cost high accuracy experimental setup for tomographic particle image velocimetry

This work deals with the critical aspects related to cost reduction of a Tomo PIV setup and to the bias errors introduced in the velocity measurements by the coherent motion of the ghost particles. The proposed solution consists of using two independent imaging systems composed of three (or more) low speed single frame cameras, which can be up to ten times cheaper than double shutter cameras with the same image quality. Each imaging system is used to reconstruct a particle distribution in the same measurement region, relative to the first and the second exposure, respectively. The reconstructed volumes are then interrogated by cross-correlation in order to obtain the measured velocity field, as in the standard tomographic PIV implementation. Moreover, differently from tomographic PIV, the ghost particle distributions of the two exposures are uncorrelated, since their spatial distribution is camera orientation dependent. For this reason, the proposed solution promises more accurate results, without the bias effect of the coherent ghost particles motion. Guidelines for the implementation and the application of the present method are proposed. The performances are assessed with a parametric study on synthetic experiments. The proposed low cost system produces a much lower modulation with respect to an equivalent three-camera system. Furthermore, the potential accuracy improvement using the Motion Tracking Enhanced MART (Novara et al 2010 Meas. Sci. Technol. 21 035401) is much higher than in the case of the standard implementation of tomographic PIV.

The Flutter Shutter Camera Simulator

The proposed method simulates an embedded flutter shutter camera implemented either analogically or numerically, and computes its performance. The goal of the flutter shutter is to make motion blur invertible, by a "fluttering" shutter that opens and closes on a well chosen sequence of time intervals. In the simulations the motion is assumed uniform, and the user can choose its velocity. Several types of flutter shutter codes are tested and evaluated: the original ones considered by the inventors, the classic motion blur, and finally several analog or numerical optimal codes proposed recently. In all cases the exact SNR of the deconvolved result is also computed.

Why do photo finish images look weird?

This paper deals with effects that appear on photographs of rotating objects when taken bya photo finish camera, a rolling shutter camera or a computer scanner. These effects arevery similar to Roget’s palisade illusion. A simple quantitative analysis of the images is alsoprovided. The effects are explored using a computer scanner in a way that is also suitablefor a student project activity.

Rolling shutter distortion removal based on curve interpolation

In videos shot by rolling shutter cameras, rapid camera or object motion may cause obvious image distortions. This paper proposes a rectification method that removes rolling shutter distortions by performing curve interpolation for each pixel. This method does not require camera calibration, and it can handle camera and object motion simultaneously. The method only assumes that the motion is smooth. If camera motion alone causes distortions, a variation of this method requiring less computation can be used. Experiments on synthetic datasets and real videos show that the methods can remove rolling shutter distortions effectively.

Investigation on Micro-Machining Characteristics and Phenomenon of Semiconductor Materials by Harmonics of Nd:YAG Laser

Precision micro-machining without crack and heat affected zones is required in order to use high-performance materials such as silicon and silicon carbide, and it is expected that the higher harmonics of the Nd:YAG laser can perform precision micro-machining due to its high photon energy. However, even by using the harmonics of the Nd:YAG laser, a heat affected zone is inevitable due to the plasma generation. In order to reduce the influence of plasma on the processing results, it is important to understand the generation mechanism of plasma. Therefore, the laser induced plasma in micro-drilling of silicon carbide was observed by a high-speed shutter camera, and the influence of laser wavelength and surrounding conditions on the machining characteristics were experimentally investigated. The removal depth increased with decreasing wavelength and the surrounding gas pressure. The surface integrity was improved by the combination of shorter wavelength and reduced pressure conditions. The behavior of laser induced plasma was different from the wavelength of the laser beam and the surrounding gas pressure. Under atmospheric pressure conditions, the plasma grew greatly and affected the wider surface around the drilled hole with increasing wavelength. Under reduced pressure conditions, there was little difference in plasma size by wavelength, and the affected zone around the drilled hole became relatively smaller. It became clear that a low surrounding gas pressure and shorter wavelength were important to obtain better surface integrity and highly efficient processing.

Efficient Video Rectification and Stabilisation for Cell-Phones

This article presents a method for rectifying and stabilising video from cell-phones with rolling shutter (RS) cameras. Due to size constraints, cell-phone cameras have constant, or near constant focal length, making them an ideal application for calibrated projective geometry. In contrast to previous RS rectification attempts that model distortions in the image plane, we model the 3D rotation of the camera. We parameterise the camera rotation as a continuous curve, with knots distributed across a short frame interval. Curve parameters are found using non-linear least squares over inter-frame correspondences from a KLT tracker. By smoothing a sequence of reference rotations from the estimated curve, we can at a small extra cost, obtain a high-quality image stabilisation. Using synthetic RS sequences with associated ground-truth, we demonstrate that our rectification improves over two other methods. We also compare our video stabilisation with the methods in iMovie and Deshaker.

Detection of laser induced dielectric breakdown in water using a laser doppler vibrometer

Abstract This study is focused on exploring the feasibility of an all-optic surface scanning method in determining the size and position of a submerged, laser generated, optoacoustic (OA) source. The optoacoustic effect in this case was generated when the absorption of a short electromagnetic pulse in matter caused a dielectric breakdown, a plasma emission flash and a subsequent acoustic wave. In the experiment, a laser pulse with λ = 1064 nm and 12 ns pulse length was aimed at a volume of deionized water. When the laser beam was focused by a f = 16 mm lens, a single dielectric breakdown spot occurred. When a f = 40 mm was used several breakdowns in a row were induced. The breakdowns were photographed using a double shutter camera. The acoustic wave generated by the dielectric breakdowns were detected at a point on the water surface using a laser Doppler vibrometer (LDV). First, the LDV signal was used to calculate the speed of sound with an accuracy of 10 m/s. Secondly, the location and length of the dielectric breakdown was calculated with an accuracy of 1 mm. The calculated position matched the breakdown location recorded by a camera. The results show that it is possible to use LDV surface measurements from a single spot to determine both the position and length of the OA source as well as the speed of sound in the medium. Furthermore, the LDV measurements also show a secondary peak that originates from the OA source. To unravel the origin and properties of this interesting feature, further investigations are necessary

Observation of Plasma Behavior in Micro-machining of Ceramics by Harmonics of Nd:YAG Laser

Precision micro-machining without cracks and with a narrow heat affected zone is highly required for the effective use of high-performance material. For this purpose, it is expected that the higher harmonics of Nd:YAG laser could perform the precision micro-machining by its high photon energy. However, even using harmonics of Nd:YAG laser, the heat affected zone is inevitable due to the plasma generation. In order to reduce the influence of plasma, it is important to understand the generation mechanism of plasma. Therefore, the laser induced plasma was observed by the high-speed shutter camera, and the influence of laser wavelength on machining characteristics was investigated.

Characterization of Plasma Jet Structure and Fluctuations by Statistic Processing of Photographic Images

A couple camera-computer provides a possibility of simple characterization and monitoring of plasma jets. This paper demonstrates another way of characterization of plasma flow structure by statistical processing of plasma jet images. The processing is based on an analysis of variations of local brightness in a sequence of images recorded by fast shutter camera. Resulting pictures show space distribution of plasma jet brightness fluctuations.