Rolling Shutter Research Articles

INVESTIGATION OF PHOTOTRIANGULATION ACCURACY WITH USING OF VARIOUS TECHNIQUES LABORATORY AND FIELD CALIBRATION

Abstract. Nowadays, aerial survey technology using aerial systems based on unmanned aerial vehicles (UAVs) becomes more popular. UAVs physically can not carry professional aerocameras. Consumer digital cameras are used instead. Such cameras usually have rolling, lamellar or global shutter. Quite often manufacturers and users of such aerial systems do not use camera calibration. In this case self-calibration techniques are used. However such approach is not confirmed by extensive theoretical and practical research. In this paper we compare results of phototriangulation based on laboratory, test-field or self-calibration. For investigations we use Zaoksky test area as an experimental field provided dense network of target and natural control points. Racurs PHOTOMOD and Agisoft PhotoScan software were used in evaluation. The results of investigations, conclusions and practical recommendations are presented in this article.

An Affine Motion Model for Removing Rolling Shutter Distortions

Rolling shutter distortions degrade the quality of videos captured by hand-held cameras. This letter proposes an affine motion model for removing rolling shutter distortions. The model represents the image motion during image capture as a sequence of affine transformations and computes the composition of these affine transformations precisely. Because an affine transformation can be represented by a neural network with one layer of linear neurons, the motion model can be represented by a multilayer neural network of linear neurons. Thus, the backpropagation algorithm can be used to improve the efficiency of the optimization process that estimates the model parameters. The proposed model is calibration-free. It is more general than other rolling shutter motion models because it only assumes that the velocity of the image during image acquisition is piecewise constant. Experimental results demonstrate that the model is more accurate than two state-of-the-art models and that the model parameters can be estimated efficiently.

신경망 기반 차량 이미지센서 시스템을 위한 플리커 프리 공간-PSK 변조 기법

This paper introduces a novel modulation scheme for vehicular communication in taking advantage of existing LED lights available on a car. Our proposed 2-Phase Shift Keying (2-PSK) is a spatial modulation approach in which a pair of LED light sources in a car (either rear LEDs or front LEDs) is used as a transmitter. A typical camera (i.e. low frame rate at no greater than 30fps) that either a global shutter camera or a rolling shutter camera can be used as a receiver. The modulation scheme is a part of our Image Sensor Communication proposal submitted to IEEE 802.15.7r1 (TG7r1) recently. Also, a neural network approach is applied to improve the performance of LEDs detection and decoding under the noisy situation. Later, some analysis and experiment results are presented to indicate the performance of our system

DisCo

We present DisCo, a novel display-camera communication system. DisCo enables displays and cameras to communicate with each other while also displaying and capturing images for human consumption. Messages are transmitted by temporally modulating the display brightness at high frequencies so that they are imperceptible to humans. Messages are received by a rolling shutter camera that converts the temporally modulated incident light into a spatial flicker pattern. In the captured image, the flicker pattern is superimposed on the pattern shown on the display. The flicker and the display pattern are separated by capturing two images with different exposures. The proposed system performs robustly in challenging real-world situations such as occlusion, variable display size, defocus blur, perspective distortion, and camera rotation. Unlike several existing visible light communication methods, DisCo works with off-the-shelf image sensors. It is compatible with a variety of sources (including displays, single LEDs), as well as reflective surfaces illuminated with light sources. We have built hardware prototypes that demonstrate DisCo’s performance in several scenarios. Because of its robustness, speed, ease of use, and generality, DisCo can be widely deployed in several applications, such as advertising, pairing of displays with cell phones, tagging objects in stores and museums, and indoor navigation.

Towards Kilo-Hertz 6-DoF Visual Tracking Using an Egocentric Cluster of Rolling Shutter Cameras.

To maintain a reliable registration of the virtual world with the real world, augmented reality (AR) applications require highly accurate, low-latency tracking of the device. In this paper, we propose a novel method for performing this fast 6-DOF head pose tracking using a cluster of rolling shutter cameras. The key idea is that a rolling shutter camera works by capturing the rows of an image in rapid succession, essentially acting as a high-frequency 1D image sensor. By integrating multiple rolling shutter cameras on the AR device, our tracker is able to perform 6-DOF markerless tracking in a static indoor environment with minimal latency. Compared to state-of-the-art tracking systems, this tracking approach performs at significantly higher frequency, and it works in generalized environments. To demonstrate the feasibility of our system, we present thorough evaluations on synthetically generated data with tracking frequencies reaching 56.7 kHz. We further validate the method's accuracy on real-world images collected from a prototype of our tracking system against ground truth data using standard commodity GoPro cameras capturing at 120 Hz frame rate.

Mobile-phone based visible light communication using region-grow light source tracking for unstable light source.

In order to increase the data rate of the camera-based visible light communication (VLC) system, using rolling shutter effect has been demonstrated successfully, in which the pixel rows of the complementary-metal-oxide-semiconductor (CMOS) image sensor are activated sequentially. Previous camera-based VLCs focused on using a stable LED light source, and its illumination area is positioned at the center of an image frame. In this work, we investigate the performance of a camera-based VLC with light source at different parts of an image frame. We propose and demonstrate using region-grow algorithm to track the light source. We also evaluate and discuss different scenarios when the light source is moved. Besides, a recorded > 5 kbit/s net data rate can be achieved by using only a single phosphor-based white-light LED source. Here, we demonstrate that 4.502 pixel/bit can be achieved.

Tri-linear color multi-linescan sensor with 200 kHz line rate

In this paper we present a newly developed linear CMOS high-speed line-scanning sensor realized in a 0.35μm CMOS OPTO process for line-scan with 200kHz true RGB and 600kHz monochrome line rate, respectively. In total, 60 lines are integrated in the sensor allowing for electronic position adjustment. The lines are read out in rolling shutter manner. The high readout speed is achieved by a column-wise organization of the readout chain. At full speed, the sensor provides RGB color images with a spatial resolution down to 50μm. This feature enables a variety of applications like quality assurance in print inspection, real-time surveillance of railroad tracks, in-line monitoring in flat panel fabrication lines and many more. The sensor has a fill-factor close to 100%, preventing aliasing and color artefacts. Hence the tri-linear technology is robust against aliasing ensuring better inspection quality and thus less waste in production lines.

PHOTOGRAMMETRIC ACCURACY AND MODELING OF ROLLING SHUTTER CAMERAS

Abstract. Unmanned aerial vehicles (UAVs) are becoming increasingly popular in professional mapping for stockpile analysis, construction site monitoring, and many other applications. Due to their robustness and competitive pricing, consumer UAVs are used more and more for these applications, but they are usually equipped with rolling shutter cameras. This is a significant obstacle when it comes to extracting high accuracy measurements using available photogrammetry software packages. In this paper, we evaluate the impact of the rolling shutter cameras of typical consumer UAVs on the accuracy of a 3D reconstruction. Hereto, we use a beta-version of the Pix4Dmapper 2.1 software to compare traditional (non rolling shutter) camera models against a newly implemented rolling shutter model with respect to both the accuracy of geo-referenced validation points and to the quality of the motion estimation. Multiple datasets have been acquired using popular quadrocopters (DJI Phantom 2 Vision+, DJI Inspire 1 and 3DR Solo) following a grid flight plan. For comparison, we acquired a dataset using a professional mapping drone (senseFly eBee) equipped with a global shutter camera. The bundle block adjustment of each dataset shows a significant accuracy improvement on validation ground control points when applying the new rolling shutter camera model for flights at higher speed (8m=s). Competitive accuracies can be obtained by using the rolling shutter model, although global shutter cameras are still superior. Furthermore, we are able to show that the speed of the drone (and its direction) can be solely estimated from the rolling shutter effect of the camera.

PHOTOGRAMMETRIC ACCURACY AND MODELING OF ROLLING SHUTTER CAMERAS

Unmanned aerial vehicles (UAVs) are becoming increasingly popular in professional mapping for stockpile analysis, construction site monitoring, and many other applications. Due to their robustness and competitive pricing, consumer UAVs are used more and more for these applications, but they are usually equipped with rolling shutter cameras. This is a significant obstacle when it comes to extracting high accuracy measurements using available photogrammetry software packages. In this paper, we evaluate the impact of the rolling shutter cameras of typical consumer UAVs on the accuracy of a 3D reconstruction. Hereto, we use a beta-version of the Pix4Dmapper 2.1 software to compare traditional (non rolling shutter) camera models against a newly implemented rolling shutter model with respect to both the accuracy of geo-referenced validation points and to the quality of the motion estimation. Multiple datasets have been acquired using popular quadrocopters (DJI Phantom 2 Vision+, DJI Inspire 1 and 3DR Solo) following a grid flight plan. For comparison, we acquired a dataset using a professional mapping drone (senseFly eBee) equipped with a global shutter camera. The bundle block adjustment of each dataset shows a significant accuracy improvement on validation ground control points when applying the new rolling shutter camera model for flights at higher speed (8m=s). Competitive accuracies can be obtained by using the rolling shutter model, although global shutter cameras are still superior. Furthermore, we are able to show that the speed of the drone (and its direction) can be solely estimated from the rolling shutter effect of the camera.

Design and Implementation of a Novel Compatible Encoding Scheme in the Time Domain for Image Sensor Communication.

This paper presents a modulation scheme in the time domain based on On-Off-Keying and proposes various compatible supports for different types of image sensors. The content of this article is a sub-proposal to the IEEE 802.15.7r1 Task Group (TG7r1) aimed at Optical Wireless Communication (OWC) using an image sensor as the receiver. The compatibility support is indispensable for Image Sensor Communications (ISC) because the rolling shutter image sensors currently available have different frame rates, shutter speeds, sampling rates, and resolutions. However, focusing on unidirectional communications (i.e., data broadcasting, beacons), an asynchronous communication prototype is also discussed in the paper. Due to the physical limitations associated with typical image sensors (including low and varying frame rates, long exposures, and low shutter speeds), the link speed performance is critically considered. Based on the practical measurement of camera response to modulated light, an operating frequency range is suggested along with the similar system architecture, decoding procedure, and algorithms. A significant feature of our novel data frame structure is that it can support both typical frame rate cameras (in the oversampling mode) as well as very low frame rate cameras (in the error detection mode for a camera whose frame rate is lower than the transmission packet rate). A high frame rate camera, i.e., no less than 20 fps, is supported in an oversampling mode in which a majority voting scheme for decoding data is applied. A low frame rate camera, i.e., when the frame rate drops to less than 20 fps at some certain time, is supported by an error detection mode in which any missing data sub-packet is detected in decoding and later corrected by external code. Numerical results and valuable analysis are also included to indicate the capability of the proposed schemes.

Line-scanning fiber bundle endomicroscopy with a virtual detector slit.

Coherent fiber bundles can be used to relay the image plane from the distal tip of an endomicroscope to an external confocal microscopy system. The frame rate is therefore determined by the speed of the microscope's laser scanning system which, at 10-20 Hz, may be undesirably low for in vivo clinical applications. Line-scanning allows an increase in the frame rate by an order of magnitude in exchange for some loss of optical sectioning, but the width of the detector slit cannot easily be adapted to suit different imaging conditions. The rolling shutter of a CMOS camera can be used as a virtual detector slit for a bench-top line-scanning confocal microscope, and here we extend this idea to endomicroscopy. By synchronizing the camera rolling shutter with a scanning laser line we achieve confocal imaging with an electronically variable detector slit. This architecture allows us to acquire every other frame with the detector slit offset by a known distance, and we show that subtracting this second image leads to improved optical sectioning.

RGB visible light communication using mobile-phone camera and multi-input multi-output.

Red, green, blue (RGB) light-emitting-diodes (LEDs) are used to increase the visible light communication (VLC) transmission capacity via wavelength-division-multiplexing (WDM), and the color image sensor in mobile phone is used to separate different color signals via a color filter array. However, due to the wide optical bandwidths of the color filters, there is a high spectral overlap among different channels, and a high inter-channel interference (ICI) happens. Here, we propose and demonstrate an RGB VLC transmission using CMOS image sensor with multi-input multi-output (MIMO) technique to mitigate the ICI and retrieve the three independent color channels in the rolling shutter pattern. Data pattern extinction-ratio (ER) enhancement and thresholding are deployed.

Fast structured illumination microscopy using rolling shutter cameras

Spatial light modulators (SLM) update in a synchronous manner, whereas the data readout process in fast structured illumination systems is usually done using a rolling shutter camera with asynchronous readout. In structured illumination microscopy (SIM), this leads to synchronization problems causing a speed limit for fast acquisition. In this paper we present a configuration to overcome this limit by exploiting the extremely fast SLM display and dividing it into several segments along the direction of the rolling shutter of the sCMOS camera and displaying multiple SLM frames per camera acquisition. The sCMOS runs in continuous rolling shutter mode and the SLM keeps the readout-line always inside a dark region presenting different SIM patterns before and after the readout/start-exposure line.Using this approach, we reached a raw frame rate of 714 frames per second (fps) resulting in a two-beam SIM acquisition rate of 79 fps with a region of interest (ROI) of 16.5 × 16.5 μm2.

Performance Analysis of Visible Light Communication Using CMOS Sensors.

This paper elucidates the fundamentals of visible light communication systems that use the rolling shutter mechanism of CMOS sensors. All related information involving different subjects, such as photometry, camera operation, photography and image processing, are studied in tandem to explain the system. Then, the system performance is analyzed with respect to signal quality and data rate. To this end, a measure of signal quality, the signal to interference plus noise ratio (SINR), is formulated. Finally, a simulation is conducted to verify the analysis.

Characterization of a three side abuttable CMOS pixel sensor with digital pixel and data compression for charged particle tracking

CMOS monolithic pixel sensor technology has been chosen to equip the new ALICE trackers for HL-LHC . PIXAM is the final prototype from an R&D program specific to the Muon Forward Tracker which intends to push significantly forward the performances of the mature rolling shutter architecture. By implementing a digital pixel allowing to readout of a group of rows in parallel, the PIXAM sensor increases the rolling shutter readout speed while keeping the same power consumption as that of analogue pixel sensors. This paper will describe shortly the ASIC architecture and will focus on the analogue and digital performances of the sensor, obtained from laboratory measurements.

Light Encryption Scheme Using Light-Emitting Diode and Camera Image Sensor

Visible light communication (VLC) is regarded as relatively secure communication when compared with other wireless communications using radio frequency (RF). However, due to the visual nature, the VLC signals are still subject to eavesdropping when they are emitted by the light source. Here, we propose a light encryption scheme using devices having light-emitting diode (LED) and a camera image sensor, such as a mobile phone. The original visible signal sending from the lamp can be first received by the proposed light encrypter. The information can be encrypted and then emitted. The light encrypter acts as an encryption gateway for signals in optical domain. The rolling shutter effect of the complementary metal--oxide semiconductor (CMOS) camera in the mobile phone can be used. By demodulating the rolling shutter pattern, the data information can be obtained. We also propose and demonstrate using the Otsu thresholding scheme to define the data logic in the rolling shutter pattern. We show that the Otsu scheme is effective for estimating the bit error rate (BER). The optimum number of intervals (segmentations) and the process time of the Otsu method are also studied.

Visible Light Communication Using Receivers of Camera Image Sensor and Solar Cell

We propose an electronic label and sensor system using visible light communication (VLC). The downlink signal is transmitted by a white-light light-emitting diode (LED) lamp that can provide lighting, VLC, and energy harvesting for mobile devices. The downlink is received by a solar cell. The uplink can be captured by a surveillance camera image sensor. However, using the camera image sensor as a VLC receiver (Rx) is challenging since the data rate is limited by the frame rate and due to uneven light exposure. The rolling shutter effect of the image sensor can be used to increase the data rate. In this paper, we propose and demonstrate how to demodulate the obtained rolling shutter pattern using a second-order polynomial (SOP) extinction ratio (ER) enhancement scheme, together with iterative and modified quick adaptive thresholding schemes. The ER enhancement scheme can significantly reduce the large ER fluctuation. Experimental results show that by using the proposed SOP ER enhancement scheme, the bit error rate (BER) improvement can be up to two orders of magnitude. We also believe that the proposed electronic label and sensor system may be applicable to Internet-of-Things sensing networks for connecting a number of mobile devices.

Decoding mobile-phone image sensor rolling shutter effect for visible light communications

Optical wireless communication (OWC) using visible lights, also known as visible light communication (VLC), has attracted significant attention recently. As the traditional OWC and VLC receivers (Rxs) are based on PIN photo-diode or avalanche photo-diode, deploying the complementary metal-oxide-semiconductor (CMOS) image sensor as the VLC Rx is attractive since nowadays nearly every person has a smart phone with embedded CMOS image sensor. However, deploying the CMOS image sensor as the VLC Rx is challenging. In this work, we propose and demonstrate two simple contrast ratio (CR) enhancement schemes to improve the contrast of the rolling shutter pattern. Then we describe their processing algorithms one by one. The experimental results show that both the proposed CR enhancement schemes can significantly mitigate the high-intensity fluctuations of the rolling shutter pattern and improve the bit-error-rate performance.

Comparison of thresholding schemes for visible light communication using mobile-phone image sensor.

Based on the rolling shutter effect of the complementary metal-oxide-semiconductor (CMOS) image sensor, bright and dark fringes can be observed in each received frame. By demodulating the bright and dark fringes, the visible light communication (VLC) data logic can be retrieved. However, demodulating the bright and dark fringes is challenging as there is a high data fluctuation and large extinction ratio (ER) variation in each frame due. Hence proper thresholding scheme is needed. In this work, we propose and compare experimentally three thresholding schemes; including third-order polynomial curve fitting, iterative scheme and quick adaptive scheme. The evaluation of these three thresholding schemes is performed.

Rolling Shutter Effect aberration compensation in Digital Holographic Microscopy

Due to the sequential-readout nature of most CMOS sensors, each row of the sensor array is exposed at a different time, resulting in the so-called rolling shutter effect that induces geometric distortion to the image if the video camera or the object moves during image acquisition. Particularly in digital holograms recording, while the sensor captures progressively each row of the hologram, interferometric fringes can oscillate due to external vibrations and/or noises even when the object under study remains motionless. The sensor records each hologram row in different instants of these disturbances. As a final effect, phase information is corrupted, distorting the reconstructed holograms quality. We present a fast and simple method for compensating this effect based on image processing tools. The method is exemplified by holograms of microscopic biological static objects. Results encourage incorporating CMOS sensors over CCD in Digital Holographic Microscopy due to a better resolution and less expensive benefits.