Uncalibrated and Unsupervised Photometric Stereo with Piecewise Regularizer

For planetary exploration, path planning for a rover is one of the important missions. This paper deals with Shape from Shading (SfS) scheme for estimation of planet terrain. As a reliable reflectance model of a surface, the Hapke model is formed in the world of remote sensing. This paper proposes to utilize the Hapke model in the SfS algorithm for multiple camera images. Since the Hapke model has singularity when the gradient vector of a surface element is coincident with a certain direction, the model is modified not to show the singularity. As a result, the SfS algorithm is applicable to multiple camera images and valid regardless of the degree of albedo of the surface. Applying the proposed SfS algorithm for multiple images, an autonomous path planning based on Dynamic Programming is shown. The effectiveness of the proposed scheme is investigated in numerical simulations, and some discussion about the results is presented. INTRODUCTION For planetary exploration, considering the communication time delay between the planet and the earth, the mission is desired to be completed autonomously even in unknown environments. Estimated 3-D elevation map of planetary terrains is very efficient for path planning of rovers. Estimation from camera images is desirable from a point that the images contain terrain information over the region. ’Shape from Shading (SfS)’[1];[2] is one of the most useful techniques because of the simple equipment: a camera and signal processors. However, it has a problem to be solved, the improper convergence of the algorithm caused by Research Associate, member of AIAA y Associate Professor, member of AIAA local minimum, self-shadow, occlusion and noises on camera image. Essentially, estimation of 3-D elevation from only one 2-D camera image is an ’ill-posed’ problem and it has no unique solution. To avoid the problem, it is assumed typically for the standard SfS algorithm that the terrain surface is smooth, and the derivatives of the gradients are minimized. However, the assumption neither be always satisfiable nor improves the results drastically. To utilize multiple camera images for SfS algorithm, the Lommel-Seeliger model[3];[4] has been proposed for the reflection of a surface. However, some experimental results using real camera images are much worse than expected, while the approach can improve the results at least theoretically. One of the most influential factors for the results is improperness of the reflectance model for the albedo of the surface. On the other hand, in the world of remote sensing, the Hapke model[5] [7] is considered as the most reliable reflectance model of a surface. However, the Hapke model shows singularity when the gradient vector of a surface element is coincident with such certain direction that the reflected light angle is close to the incident light angle. Considering that the accuracy for estimated gradient of a surface element is quite low around the singularity, and that 3-D elevation map is produced to put up the surface elements one by one, the inaccuracy caused by the singularity results in unacceptable error. Therefore, this paper proposes to modify the Hapke model and apply the model to the SfS scheme for multiple camera images. First, this paper summarizes the standard SfS algorithm. Then, the explanations for the SfS utilizing multiple images and the reflectance model are followed. Making a comparison with the different models, the modified Hapke model is proposed, which is valid for 1 American Institute of Aeronautics and Astronautics AIAA/AAS Astrodynamics Specialist Conference and Exhibit 5-8 August 2002, Monterey, California AIAA 2002-4825 Copyright © 2002 by the American Institute of Aeronautics and Astronautics, Inc. All rights reserved. the different degree of albedo. Based on the proposed reflectance model, a new SfS algorithm and the scheme for an autonomous path planning by Dynamic Programming are introduced. Finally, the effectiveness of the proposed scheme is investigated in numerical simulations, and some discussions about the results are followed. SHAPE FROM SHADING ALGORITHM

Photometric stereo (PS) is an established technique for high-detail reconstruction of 3D geometry and appearance. To correct for surface integration errors, PS is often combined with multiview stereo (MVS). With dynamic objects, PS reconstruction also faces the problem of computing optical flow (OF) for image alignment under rapid changes in illumination. Current PS methods typically compute optical flow and MVS as independent stages, each one with its own limitations and errors introduced by early regularization. In contrast, scene flow methods estimate geometry and motion, but lack the fine detail from PS. This paper proposes photogeometric scene flow (PGSF) for high-quality dynamic 3D reconstruction. PGSF performs PS, OF, and MVS simultaneously. It is based on two key observations: (i) while image alignment improves PS, PS allows for surfaces to be relit to improve alignment, (ii) PS provides surface gradients that render the smoothness term in MVS unnecessary, leading to truly data-driven, continuous depth estimates. This synergy is demonstrated in the quality of the resulting RGB appearance, 3D geometry, and 3D motion.

Improved SFS 3D measurement based on BP neural network

Background:Polyp shapes play an important role in colorectal diagnosis. However, endoscopy images are usually composed of nonrigid objects such as a polyp. Hence, it is challenging for polyp shape recovery. It is demanded to establish a support system of the colorectal diagnosis system based on polyp shape.Introduction:Shape from Shading (SFS) is one valuable approach based on photoclinometry for polyp shape recovery. SFS and endoscope image are compatible on the first sight, but there are constraints for applying SFS to endoscope image. Those approaches need some parameters like a depth from the endoscope lens to the surface, and surface reflectance factor . Furthermore, those approaches assume the whole surface which has the same value of for the Lambertian surface.Methods:This paper contributes to mitigating constraint for applying SFS to the endoscope image based on a cue from the medical structure. An extracted medical suture is used to estimate parameters, and a method of polyp shape recovery method is proposed using both geometric and photometric constraint equations. Notably, the proposed method realizes polyp shape recovery from a single endoscope image.Results:From experiments it was confirmed that the approximate polyp model shape was recovered and the proposed method recovered absolute size and shape of polyp using medical suture information and obtained parameters from a single endoscope image.Conclusion:This paper proposed a polyp shape recovery method which mitigated the constraint for applying SFS to the endoscope image using the medical suture. Notably, the proposed method realized polyp shape recovery from a single endoscope image without generating uniform Lambertian reflectance.

Constant albedo of surface is an important premise widely used for Shape from Shading(SFS). Since it is not satisfied when the surface is in multicolor, the application of SFS methods are very limited. In this paper we introduce a novel method for gray-scale transformation as a preprocess of SFS methods, which removes intensity differences between colors. First we cluster the pixels based on color using an improved Hough Transform approach, and generate Color Lines to describe color information. Then the color is adjusted using Color Lines, and the multicolor image is converted into gray-scale without sudden change of intensity between regions in different color. Experimental results show that our method outperforms existing methods such as linear luminance when 3D models are reconstructed by SFS.

We propose a novel method for 3D head reconstruction and view-invariant recognition from single 2D images. We employ a deterministic Shape From Shading (SFS) method with initial conditions estimated by Hybrid Principal Component Analysis (HPCA) and multi-level global optimization with error-dependent smoothness and integrability constraints. Our HPCA algorithm provides initial estimates of 3D range mapping for the SFS optimization, which is quite accurate and yields much improved 3D head reconstruction. The paper also includes significant contributions in novel approaches to global optimization and in SFS handling of variable and unknown surface albedo, a problem with unsatisfactory solutions by prevalent SFS methods. In the experiments, we reconstruct 3D head range images from 2D single images in different views. The 3D reconstructions are then used to recognize stored model persons. Empirical results show that our HPCA based SFS method provides 3D head reconstructions that notably improve the accuracy compared to other approaches. 3D reconstructions derived from side view (profile) images of 40 persons are tested against 80 3D head models and a recognition rate of over 90% is achieved. Such a capability was not demonstrated by any other method we are aware of.

Many shape recovery algorithms—in particular, shape from shading (SFS)—are based on a point source at infinity or a uniform hemispheric source. It will be convenient and useful if we can perform SFS under indoor lights that may be spherical, cylindrical, or flat (ceiling lights) in shape in an uncontrolled environment. As a first step toward this goal we propose a light source model for each of the above shapes for performing SFS. In this study we give the derivation of the rectangular, spherical, and cylindrical light source models. In indoor environments the positions (usually on the ceiling or walls) of the light sources are known. Assuming that the target object is small relative to the distances from the sources, we have derived a reflectance map for the Lambertian surface of an object under a mixture of light sources of the above shapes. Hence the shape recovery can be performed by using the SFS technique. This is a significant step toward the application of SFS in uncontrolled practical environments under household or office lighting. This technique of shape recovery is verified by many examples of simulations and real experiments, and the results are good.

We present photometric stereo algorithms robust to non-Lambertian reflection, which are based on a convolutional neural network in which surface normals of objects with complex geometry and surface reflectance are estimated from a given set of an arbitrary number of images. These images are taken from the same viewpoint under different directional illumination conditions. The proposed method focuses on surface normal estimation, where multi-scale feature aggregation is proposed to obtain a more accurate surface normal, and max pooling is adopted to obtain an intermediate order-agnostic representation in the photometric stereo scenario. The proposed multi-scale feature aggregation scheme using feature concatenation is easily incorporated into existing photometric stereo network architectures. Our experiments were performed with a DiLiGent photometric stereo benchmark dataset consisting of ten real objects, and they demonstrated that the accuracies of our calibrated and uncalibrated photometric stereo approaches were improved over those of baseline methods. In particular, our experiments also demonstrated that our uncalibrated photometric stereo outperformed the state-of-the-art method. Our work is the first to consider the multi-scale feature aggregation in photometric stereo, and we showed that our proposed multi-scale fusion scheme estimated the surface normal accurately and was beneficial to improving performance.

Abstract. Shape and Albedo from Shading (SAfS) techniques recover pixel-wise surface details based on the relationship between terrain slopes, illumination and imaging geometry, and the energy response (i.e., image intensity) captured by the sensing system. Multiple images with different illumination geometries (i.e., photometric stereo) can provide better SAfS surface reconstruction due to the increase in observations. Photometric stereo SAfS is suitable for detailed surface reconstruction of the Moon and other extra-terrestrial bodies due to the availability of photometric stereo and the less complex surface reflecting properties (i.e., albedo) of the target bodies as compared to the Earth. Considering only one photometric stereo pair (i.e., two images), pixel-variant albedo is still a major obstacle to satisfactory reconstruction and it needs to be regulated by the SAfS algorithm. The illumination directional difference between the two images also becomes an important factor affecting the reconstruction quality. This paper presents a photometric stereo SAfS algorithm for pixel-level resolution lunar surface reconstruction. The algorithm includes a hierarchical optimization architecture for handling pixel-variant albedo and improving performance. With the use of Lunar Reconnaissance Orbiter Camera - Narrow Angle Camera (LROC NAC) photometric stereo images, the reconstructed topography (i.e., the DEM) is compared with the DEM produced independently by photogrammetric methods. This paper also addresses the effect of illumination directional difference in between one photometric stereo pair on the reconstruction quality of the proposed algorithm by both mathematical and experimental analysis. In this case, LROC NAC images under multiple illumination directions are utilized by the proposed algorithm for experimental comparison. The mathematical derivation suggests an illumination azimuthal difference of 90 degrees between two images is recommended to achieve minimal error in SAfS reconstruction while results using real data presents similar pattern. Although the algorithm is designed for lunar surface reconstruction, it is likely to be applicable on other extra-terrestrial bodies such as Mars. The results and findings from this research is of significance for the practical use of photometric stereo and SAfS in the domain of planetary remote sensing and mapping.

A photometric stereo needs three images taken under three different light directions lit one by one, while a color photometric stereo needs only one image taken under three different lights lit at the same time with different light directions and different colors. As a result, a color photometric stereo can obtain the surface normal of a dynamically moving object from a single image. However, the conventional color photometric stereo cannot estimate a multicolored object due to the colored illumination. This paper uses an example-based photometric stereo to solve the problem of the color photometric stereo. The example-based photometric stereo searches the surface normal from the database of the images of known shapes. Color photometric stereos suffer from mathematical difficulty, and they add many assumptions and constraints; however, the example-based photometric stereo is free from such mathematical problems. The process of our method is pixelwise; thus, the estimated surface normal is not oversmoothed, unlike existing methods that use smoothness constraints. To demonstrate the effectiveness of this study, a measurement device that can realize the multispectral photometric stereo method with sixteen colors is employed instead of the classic color photometric stereo method with three colors.

This paper describes a novel method for 3D head reconstruction by shape from shading (SFS) with initial conditions estimated using hybrid principal component analysis (HPCA). Prevalent Shape from Shading methods do not achieve good results on realistic images if a close initial estimation of the true surface is unavailable. This happens frequently in practical situations. To overcome this problem, we propose a novel shape from shading method which is based on multiple-level optimization with initial conditions derived by HPCA algorithm. We find that the HPCA algorithm provides reliable surface estimation that serves as initial conditions for the multiple-level optimization. The proposed method is also insensitive to albedo variations that frequently occur on head surfaces. Experimental results show that the method achieves accurate reconstruction of human heads.

Because all the Shape from shading (SFS) algorithm depend on assumptions about the continuity of surface height and its partial derivatives, we add the smooth constraint to the objective function to enforce the integrability, and propose a new shape from shading algorithm based on maximum entropy. We present results of our new shape from shading method on a synthetic image. Results and comparisons showed that the new algorithm gives consistently better feature preserving reconstructions.

International audience

The aim of this thesis is to explore computational methods for the shape from shading problem as formulated through the image irradiance equation. We seek to develop robust, efficient methods and test our algorithms on synthetic images ranging from simple smooth surfaces to complex digital terrain model data. We consider three different approaches. The first approach revisits the method of characteristic strips with a view to using more stable integration schemes than had been used in earlier works. Stable schemes coupled with projections onto the image irradiance equation are used. Although the effects of noise are thereby reduced, the solution is still deemed unsatisfactory even for very simple surfaces. The second approach considers Horn's variational technique as a basis for producing a fast solver. We devise a discretization scheme coupled with a special continuation multigrid method for this formulation. We also allow for multiple image data and explicit knowledge of the location of discontinuities in surface height and orientation. Given multiple image data, we obtain excellent results even in the presence of discontinuities. The third approach examines a class of solution techniques based on fluid flow which are new to the shape from shading literature. This formulation is ill posed in general, so we propose a regularization of the problem. We observe that the algorithm is prone to producing spurious results. Analysis shows that these are due to the non-random accumulation of errors in the computed solution. Of the three approaches considered, the variational method yields the most promising results. Efficient, good quality reconstructions are obtained, especially when data from more than one image are available.

Photometric stereo (PST) is a widely used technique of estimating surface normals from an image set. However, it often produces inaccurate results for non-Lambertian surface reflectance. In this study, PST is reformulated as a sparse recovery problem where non-Lambertian errors are explicitly identified and corrected. We show that such a problem can be accurately solved via a greedy algorithm called orthogonal matching pursuit (OMP). The performance of OMP is evaluated on synthesized and real-world datasets: we found that the greedy algorithm is overall more robust to non-Lambertian errors than other state-of-the-art sparse approaches with little loss of efficiency. Along with providing an overview of current methods, novel contributions in this paper are as follows: we propose an alternative sparse formulation for PST; in previous PST studies (Wu et al., Robust photometric stereo via low-rank matrix completion and recovery, 2010), (S. Ikehata et al., Robust photometric stereo using sparse regression, 2012), the surface normal vector and the error vector are treated as two entities and are solved independently. In this study, we convert their formulation into a new canonical form of the sparse recovery problem by combining the two vectors into one large vector in a new “stacked” formulation in this domain. This allows for a large repertoire of existing sparse recovery algorithms to be more straightforwardly applied to the PST problem. In our application of the OMP greedy algorithm, we show that greedy solvers can indeed be applied, with this study supplying the first of such attempt at employing greedy approaches to estimate surface normals within the framework of PST. We numerically compare the performance of several normal vector recovery methods. Most notably, this is the first detailed test on complex images of the normal estimation accuracy of our previously proposed method, least median of squares (LMS).