Shape From Shading Algorithms Research Articles

Surface inspection system for industrial components based on shape from shading minimization approach

An inspection system using estimated three-dimensional (3-D) surface characteristics information to detect and classify the faults to increase the quality control on the frequently used industrial components is proposed. Shape from shading (SFS) is one of the basic and classic 3-D shape recovery problems in computer vision. In our application, we developed a system using Frankot and Chellappa SFS method based on the minimization of the selected basis function. First, the specialized image acquisition system captured the images of the component. To eliminate noise, wavelet transform is applied to the taken images. Then, estimated gradients were used to obtain depth and surface profiles. Depth information was used to determine and classify the surface defects. Also, a comparison made with some linearization-based SFS algorithms was discussed. The developed system was applied to real products and the results indicated that using SFS approaches is useful and various types of defects can easily be detected in a short period of time.

Application of 3D-Representation Algorithms in Small-Bowel Capsule Endoscopy

In video endoscopy, there is on-going research for a three-dimensional (3-D) reconstruction of the gastrointestinal (GI) tract using stereoscopic cameras [1,2]. This could lead to improved diagnostic yield by enhancing textural features of the mucosa and/or abnormalities [3,4]. However, due to current limitations in capsule endosocpy (CE) technology i.e. size, power consumption, and resolution, accurate 3-D imaging of the intestinal lumen in CE is still unfeasible [5]. To address this shortcoming, various algorithms such as Shape-from-Shading (SfS) have been utilized in CE. Essentially, SfS algorithms reconstruct surface appearances through an inversion of the image irradiance equation thus recovering the shape of GI structures and/or lesions given a single twodimensional 2-D image. During a recent CE examination in our Department, the ampulla of Vater was identified (Figure 1). By applying 4 publicly available SfS algorithms [3, 6-9] on the 2-D image surface features become accentuated (Figure 2), although the outcome is variable and depends on the method used (unpublished authors' data).

Evaluation of 4 three-dimensional representation algorithms in capsule endoscopy images

To evaluate the three-dimensional (3-D) representation performance of 4 publicly available Shape-from-Shading (SfS) algorithms in small-bowel capsule endoscopy (SBCE). SfS techniques recover the shape of objects using the gradual variation of shading. There are 4 publicly available SfS algorithms. To the best of our knowledge, no comparative study with images obtained during clinical SBCE has been performed to date. Three experienced reviewers were asked to evaluate 54 two-dimensional (2-D) images (categories: protrusion/inflammation/vascular) transformed to 3-D by the aforementioned SfS 3-D algorithms. The best algorithm was selected and inter-rater agreement was calculated. Four publicly available SfS algorithms were compared. Tsai's SfS algorithm outperformed the rest (selected as best performing in 45/54 SBCE images), followed by Ciuti's algorithm (best performing in 7/54 images) and Torreão's (in 1/54 images). In 26/54 images; Tsai's algorithm was unanimously selected as the best performing 3-D representation SfS software. Tsai's 3-D algorithm superiority was independent of lesion category (protrusion/inflammatory/vascular; P = 0.678) and/or CE system used to obtain the 2-D images (MiroCam/PillCam; P = 0.558). Lastly, the inter-observer agreement was good (kappa = 0.55). 3-D representation software offers a plausible alternative for 3-D representation of conventional capsule endoscopy images (until optics technology matures enough to allow hardware enabled-"real" 3-D reconstruction of the gastrointestinal tract).

Linear-nonlinear neuronal model for shape from shading

The goal of shape from shading (SFS) is to recover a relative depth map from the variations of image intensity associated to changes in surface shape. There have been very few attempts at developing biologically plausible solutions to this problem, and a sound neurophysiological basis is still missing. Here we present a biologically inspired approach to SFS, formulated in terms of the well-known linear-nonlinear model of neuronal responses. Without resorting to the image irradiance equation, which is at the heart of the traditional SFS algorithms, we submit the input image to a linear filter followed by nonlinear transformations modelled on the tuning curves of the disparity-selective binocular neurons. This yields plausible shape estimates, without requiring information regarding surface reflectance or illumination.

A solution to illumination direction estimation of a shaded image: Genetic algorithm

In oblique shape from shading (SfS), the illumination direction is essential for recovering the 3D surface of a shaded image. On the other hand, fast marching methods (FMM) are SfS algorithms that use the mechanism of wave propagation to reconstruct the surface. In this paper, the estimation of illumination direction is addressed and we model it as an optimization problem. The idea is to minimize the inconsistency of wave propagation of FMM during the reconstruction. As the consistency of wave propagation is a multi-modal function of illumination direction, genetic algorithm (GA) is utilized. The proposed algorithm is examined on four synthetic models and a real world object. The experimental results show that the proposed algorithm is superior to benchmark methods.

A fractal-based relaxation algorithm for shape from terrain image

We consider the problem of extracting surface shape from a single terrain image. Although fractal models play an important role in simulating terrain models, the various Shape-from-Shading (SFS) techniques that have been applied to this kind of problem have not been coupled with a fractal prior. In this paper, we define the SFS problem of terrain imaging as a fractal-regularized problem, and solve it using Maximum-A-Posterior (MAP) estimation. In addition, we also propose a relaxation algorithm based on Landweber iteration in order to solve it. The optimum terrain surface corresponding to the observed image does not have to be the convergent result. The result can be picked up during the process of iteration with the number of iterations specified by an image-based estimation method proposed in this paper. Experimental results on both simulated data and real data show that our algorithm can efficiently extract terrain surfaces, and is more accurate than some well-known SFS algorithms, including the Horn, Zheng–Chellappa, Tsai–Shah, Pentland linear, and Lee–Rosenfeld methods.

Perception of illuminance flow in the case of anisotropic rough surfaces

Human observers estimate the illumination direction of rough surfaces rather precisely. When surfaces are rough, the illumination generates visible "texture" from differential shading at the level of the roughness, whereas differential illumination at the level of significant global surface curvature leads to the more familiar "shading". The shading is used in conventional shape-from-shading (SFS) algorithms, which ignore the illumination texture cue. Because of this simplification, SFS algorithms are typically formulated as global problems (partial differential equations, etc.). Human observers are likely to apply different methods than do these conventional SFS algorithms, however. When the roughness is not isotropic, one expects systematic errors in the visual detection of illumination direction, conceivably giving rise to erroneous shape estimates. Here we addressed this issue through systematic psychophysics on illumination direction detection as a function of the roughness anisotropy. Our expectations were fully borne out, in that the observers committed the predicted systematic errors. These results are precise enough to allow the inference that illumination direction detection is based on second-order statistics--that is, of edge detector (rather than line detector) activity.

Light intensity analysis of a passive visual sensing system in GTAW

In manual welding, an experienced welder acquires 80% of the welding process information by vision. Visual sensing has the advantage of supplying abundant information without contact with the workpiece or welding circuit. Visual sensors have been widely used in the field of welding. According to optical imaging principles, the object imaging process depends on the shape, size, position, and angle of the light source; the shape of the object; the reflectance characteristic of the object’s surface; and the position, angle, and inner parameters of the camera. In order better to utilize the image information of a visual sensor and to ascertain the weld shape and size, arc light intensity is calculated from the arc length and the welding current. The visual sensing system is analyzed from the point of the view of light intensity. The function of adjusting the system to the light intensity is formulated in this paper. A relationship between the gray image, arc length, and welding current can be obtained. Welding experiments were implemented. Thus, we can recover the shape and height of the weld pool by SFS (shape from shading) algorithms from the weld pool image.

Shape from Shading with a Generalized Reflectance Map Model

Most conventional SFS (shape from shading) algorithms have been developed under three basic assumptions on surface properties and imaging geometry to simplify the problem, namely, a Lambertian surface, a distant point light source and orthographic projection. In this research, we derive a physics-based generalized reflectance map model which includes diffuse and specular reflection effects, a nearby point light source and perspective projection, and then we develop a new direct shape recovery algorithm from shaded images. The basic idea of our solution method is to discretize the image irradiance equation with a finite triangular element surface model, to express the resulting nonlinear system of equations in terms of depth variables only and to recover the object shape by linearizing the nonlinear equations and minimizing a quadratic cost functional. We perform numerical experiments with one or multiple photometric stereo images to demonstrate the performance of the derived physics-based reflectance map model and the proposed SFS algorithm.

Shape from Shading and Photometric Stereo Using Surface Approximation by Legendre Polynomials

In this paper, a new iterative shape from shading (SFS) algorithm is proposed. In the proposed algorithm, the given 3D surface is approximated by Legendre polynomials and the relationships between the given surface and its derivatives are represented in matrix forms using a polynomial coefficient vector. Then the relative depth and its derivatives are iteratively computed by updating the coefficient vector. Also the proposed SFS algorithm is extended to a photometric stereo case. In the proposed photometric stereo algorithm, the reflectance map is linearized and the cost function expressed in quadratic matrix form is minimized. The relative depth and its derivatives are also obtained by updating them iteratively. Performance of the proposed SFS and photometric stereo algorithms is evaluated in terms of three different error measures: the brightness error, orientation error, and height error. In addition, a performance comparison of the proposed and conventional SFS algorithms is shown.

Shape from Shading with Perspective Projection

Most conventional SFS (shape from shading) algorithms have been developed under the assumption of orthographic projection. However, the assumption is not valid when an object is not far away from the camera and, therefore, it causes severe reconstruction error in many real applications. In this research, we develop a new iterative algorithm for recovering surface heights from shaded images obtained with perspective projection. By dividing an image into a set of nonoverlapping triangular domains and approximating a smooth surface by the union of triangular surface patches, we can relate image brightness in the image plane directly to surface nodal heights in the world space via a linearized reflectance map based on the perspective projection model. To determine the surface height, we consider the minimization of a cost functional defined to be the sum of squares of the brightness error by solving a system of equations parameterized by nodal heights. Furthermore, we apply a successive linearization scheme in which the linearization of the reflectance map is performed with respect to surface nodal heights obtained from the previous iteration so that the approximation error of the reflectance map is reduced and accuracy of the reconstructed surface is improved iteratively. The proposed method reconstructs surface heights directly and does not require any additional integrability constraint. Simulation results for synthetic and real images are demonstrated to show the performance and efficiency of our new method.

Surface reconstruction from photometric stereo images

In previous research (Tech. Rep. 172, University of Southern California, Los Angeles, Calif., 1991) we developed an iterative shape-from-shading (SFS) algorithm that uses a single image, by combining a triangular-element surface model with a linearized reflectance map. In the current research we generalize the single-image SFS algorithm to the photometric stereo SFS algorithm, which uses multiple images taken under different lighting condition for more-accurate surface reconstruction. An explicit surface model based on nodal basis-function representation is used so that the integrability problem that arises in conventional photometric stereo SFS algorithms can be solved easily. It is also shown that single-image SFS algorithms have an inherent problem; i.e., the accuracy of the reconstructed surface height is related to the slope of the reflectance-map function defined on the gradient space. The accuracy can be greatly improved by combining two photometric images properly, and the optimal illumination condition that leads to the best shape reconstruction is examined. Simulation results for several test images are given to demonstrate the performance of our new algorithm.

Reconstructing a 3-D depth map from one or more images

Several algorithms are suggested for recovering depth and orientation maps of a surface from its image intensities. They combine the advantages of stereo vision and shape-from-shading (SFS) methods. These algorithms generate accurate, unambiguous and dense surface depth and orientation maps. Most of the existing SFS algorithms cannot be directly extended to combine stereo images because the recovery of surface depth and that of orientation are separated in these formulations. We first present an SFS algorithm that couples the generation of depth and orientation maps. This formulation also ensures that the reconstructed surface depth and its orientation are consistent. The SFS algorithm for a single image is then extended to utilize stereo images. The correspondence over stereo images is established simultaneously with the generation of surface depth and orientation. An alternative approach is also suggested for combining stereo and SFS techniques. This approach can be used to combine needle maps which are directly available from other sources such as photometric stereo. Finally we present an algorithm to combine sparse depth measurements with an orientation map to reconstruct a surface. The same algorithm can be combined with the above algorithms for solving the SFS problem with sparse depth measurements. Thus various information sources can be used to accurately reconstruct a surface.