Matthew Kerner's Photometric Stereo Project

Summary

In this project, we completed a skeleton program that approximates a plenoptic light field in a scene by analyzing multiple images taken under a sequence of varied lighting conditions. The camera response function was already normalized for us, and we were given a sequence of images of a chrome sphere as well as sets of images for various diffuse objects.

We processed the images as follows:

First, we used the chrome sphere images to calculate lighting directions. We found the surface normal of the sphere at the brightest point in the image, and used specular reflection equations to calculate the source of the light (i.e. we reflected the light across the normal and calculated its source's unit vector).
Next, we used the lighting direction information and pixel intensities in the images to calculate the surface normals of the objects by solving the least-squares approximation to the images with diffuse reflection equations.
Next, we calculated surface albedos similarly. In both this step and the previous step we accounted for shadows by weighting the least-squares approximation by the pixel intensity.
Finally, we calculated surface depth by solving for the least-squared error surface that satisifed the normals discovered in the previous steps.

We ended up with a representation of the object that could be manipulated and viewed under different lighting conditions with or without the albedo map.

What Went Well/What Didn't/Ideas for Improvement

The one trick that was non-obvious in this assignment was to account for singularities in the matrix used to calculate the depth field. When this matrix contained numbers of extremely small nonzero magnitudes, or near-infinite magnitudes, it results in a system that cannot be solved. As a result, this matrix must be altered to throw out singularities before being processed. The functions and constraints in float.h are useful for this purpose.

Generally speaking, the reconstructions are high quality, and the resulting model of the object is accurate. As you can see from the images below, the objects are recognizable and the approach is accurate.

However, there were some shortcomings of this method. Consider the second-to-last image on this page. There is a small white line coming off the leg of the cat which shows that the object's representation is inaccurate at this point (the detail is hidden when the albedo is applied because that part of the texture is so dark). I suspect that this artifact comes from the error in the surface representation at the edge of the mask, where some pixels are poorly lit in some of the images, and where pixels are censored from the data. As a result, the normals and surface are less accurate than elsewhere in the image.

One way of solving this problem would be to add smoothing to the surface to eliminate sudden changes in surface direction. Another option would be to generate "pseudo-pixels" sharing the same details as the real pixels which extend beyond the edge of the image, and present some surface which can be included in computations at that area.

Another shortcoming of this approach is that the camera and object must be fixed in the same location and orientation during the image acquisition phase. This means that only one view of the object can be captured and recreated. One option would be to take multiple sets of photos - one from each side of the object, and each set could be computed. The edges of the resulting surfaces could be stitched together (with a blending algorithm to make the edges converge). Then the object could be modelled in full 3d. Even if the object models weren't stitched together, a visual representation of the object could be obtained by switching to the model of the object that best approximates the current viewing angle on the object (i.e. the viewer could dynamically switch between the possible views depending on the perspective).

Photometric Stereo Examples

Fun Examples

Here is a self-reflexive albedo map (an image of the rock in the tool) applied to the 3-d image of the rock, in the tool: