Robust extension of light fields with probable 3D distribution based on iterative scene estimation from multi-focus images

Abstract

In this study, we propose an iterative approach to perform pseudo-scene reconstruction based on multi-focus images for estimating the 4D light fields of the scene robustly. An analogous scene-independent linear filtering approach has been proposed to efficiently achieve free viewpoint image reconstruction and scene refocusing with virtual bokeh by solely extracting 3D frequency components. However, it only succeeded in synthesizing limited regions of 4D light fields from multi-focus images, which can lead to awful degradation of reconstructed free viewpoint images outside the range due to the lack of the corresponding frequency components. In this paper, we directly utilize a linear relationship of 3D convolution combining multi-focus images and 3D scene by 3D blurring filters to estimate the 3D distribution itself of the scene instead of deriving the scene-independent linear filters from the relationship. In other words, we perform the scene reconstruction as a simplified convex optimization problem by modeling multi-focus images in multi-layered blurring composed of the scene and a bokeh filter. Algebraic reconstruction techniques of computed tomography, which uses the classical steepest descent method combined with projection onto convex sets (POCS) to ensure the convexity of the problem constraints, are introduced into such 3D scene reconstruction. The new method can be highly anticipated to assist us in extensively estimating robust 4D light fields. Experimental results illustrate that this approach enables good quality 3D image reconstruction with a wider range of 4D light fields. We also discuss the reduction of high computational costs and enormous memory requirements imposed by iterative algebraic reconstruction techniques. The quality of the obtained 4D light fields is comprehensively evaluated using both synthetic and real multi-focus images of practicable image resolution and depth range, additionally to an analysis of the faster processing times when comparing our iterative approach to the performance of the prior linear filters.