On the Mathematical Foundations of Computational Photography

Abstract

Flutter shutter (coded exposure) cameras allow to extend indefinitely the exposure time for uniform motion blurs. Recently, Tendero et al. (SIAM J Imaging Sci 6(2):813---847, 2013) proved that for a fixed known velocity v the gain of a flutter shutter in terms of root means square error (RMSE) cannot exceeds a 1.1717 factor compared to an optimal snapshot. The aforementioned bound is optimal in the sense that this 1.1717 factor can be attained. However, this disheartening bound is in direct contradiction with the recent results by Cossairt et al. (IEEE Trans Image Process 22(2), 447---458, 2013). Our first goal in this paper is to resolve mathematically this discrepancy. An interesting question was raised by the authors of reference (IEEE Trans Image Process 22(2), 447---458, 2013). They state that the "gain for computational imaging is significant only when the average signal level J is considerably smaller than the read noise variance $$\sigma _r^2$$?r2" (Cossairt et al., IEEE Trans Image Process 22(2), 447---458, 2013, p. 5). In other words, according to Cossairt et al. (IEEE Trans Image Process 22(2), 447---458, 2013) a flutter shutter would be more efficient when used in low light conditions e.g., indoor scenes or at night. Our second goal is to prove that this statement is based on an incomplete camera model and that a complete mathematical model disproves it. To do so we propose a general flutter shutter camera model that includes photonic, thermal (The amplifier noise may also be mentioned as another source of background noise, which can be included w.l.o.g. in the thermal noise) and additive [The additive (sensor readout) noise may contain other components such as reset noise and quantization noise. We include them w.l.o.g. in the readout.] (sensor readout, quantification) noises of finite variances. Our analysis provides exact formulae for the mean square error of the final deconvolved image. It also allows us to confirm that the gain in terms of RMSE of anyflutter shutter camera is bounded from above by 1.1776 when compared to an optimal snapshot. The bound is uniform with respect to the observation conditions and applies for any fixed known velocity. Incidentally, the proposed formalism and its consequences also apply to e.g., the Levin et al. motion-invariant photography (ACM Trans Graphics (TOG), 27(3):71:1---71:9, 2008; Method and apparatus for motion invariant imag- ing, October 1 2009. US Patent 20,090,244,300, 2009) and variant (Cho et al. Motion blur removal with orthogonal parabolic exposures, 2010). In short, we bring mathematical proofs to the effect of contradicting the claims of Cossairt et al. (IEEE Trans Image Process 22(2), 447---458, 2013). Lastly, this paper permits to point out the kind of optimization needed if one wants to turn the flutter shutter into a useful imaging tool.