Assigned: Thursday, Feb. 12, 2004 |
|
In this project, you will implement a system to construct a height field from a series of images of a diffuse object under different point light sources. Your software will be able to calibrate the lighting directions, find the best fit normal and albedo at each pixel, then find a surface which best matches the solved normals.
To start your project, you will be supplied with some test images and skeleton code you can use as the basis of your project, and a sample solution executable you can use to compare with your program. If you are working in the lab, please don't download all the test images: we've placed a copy in your project area:
\\gfilesrv1\courses\cse455-04wi\psmImages
TODO a) Complete the function "normalOnSphere" in LightDirs.cpp. Given an image position and the center and radius of the chrome sphere, compute the normal at that position.
TODO b) Complete the function "computeOneLight" in LightDirs.cpp. The goal is to find the center of the highlight, from which you can compute the lighting direction. First find the "center of brightness" of the pixels using a weighted average of pixel intesities, where each pixel location is weighted by its brightness. This gives you an "average" of the brightest pixel locations. Then:
TODO c) Complete the function "reflect" in Vec.h. This will allow you to reflect the viewing vector (0,0,1) about the normal to obtain the lighting direction.
TODO d) Complete the function "computeOneNormal" in Normals.cpp. Construct the matrix M and vector b given the description above, and the skeleton code will solve the 3x3 system M^T M g = M^T b. Then normalize and return the result. (Some pixels may not be solvable; for instance if a pixel is black in every image, or is non-black in only one or two images. In this case you may get "NaN" values from solveLinearSystem. In this case, just return the vector (0,0,1).)
TODO e) Complete the function "computeOneAlbedo" in Albedos.cpp. Using the description above, compute J for each lighting direction, then sum up the numerator and denominator and divide.
TODO f) Complete the functions "makeAmatrix" and "makeBvector" in Depths.cpp. Using the description above, fill in the entries of the M matrix and v vector for the least squares approximation to the surface.
Note that the matrix M has one row for every constraint, and one column for every pixel in the normal image. To translate from image locations to indices in this matrix, read pixels from the image "zind", which contains an integer index at each pixel. Pixels in zind with the value -1 are outside the image mask, and you should not create constraints for them.
Also note that the object "normals" is of type NormalImage &, which is defined in ImageTypes.h: its elements are of type Vec3f, and it has only one channel. So to access the z component of the normal at pixel i,j, you would write:
normals.Pixel(i,j).z
In the sample images directory, we have provided a few files computed by the sample solution, for comparison with your own solution and also so that you can work on these tasks independently. The files are:
We have provided you with matrix and vector classes which handle most of the heavy lifting for solving linear systems. You'll just have to fill in the entries appropriately. You can get and set matrix and vector elements using function notation: mat(i,j), where i is the row and j is the column, or vec(i). There is also a somewhat more compact Vec3 template for triples like normals and light directions: you can access its members as v.x, v.y, v.z.
There are actually two different arbitrary-size matrix classes.
The first one, CMatrix, is used in solving for normals, and it's very
straightforward. The second, CSparseMatrix, is used in solving for
depths, and it's designed particularly for problems like this one in
which the matrices are very large but most of the entries are zero.
Only nonzero entries are stored, in an efficient data structure. Any
entry which is not specifically set is assumed to have a value of 0.
We mention this explicitly because your code should NOT loop
through the rows and columns of this matrix setting all the values to
0! If you do it will actually be considerably less efficient than the
vanilla CMatrix! So, the moral of the story is: Only set elements
that have nonzero entries.
Turn in the executable (psm.exe). Turn in the code that you wrote (just the .cpp files you
modified and any new files you needed). In the artifact directory, turn in a web page containing
snapshots of the following, for at least three different example
datasets:
RGB-encoded normals
needle map
albedo map
two or more views of the reconstructed surface without albedos
two or more views of the reconstructed surface with albedos
Here are a few suggestions for extra credit.
Capture your own source images and reconstruct them. (See
a TA or Prof. Seitz early if you want to do this, there are a number
of "gotchas" but it can be fun!)
Implement example-based photometric stereo, as described
in class. We can provide some sample datasets if you're interested
in this.
Modify the objective function for surface reconstruction
and rewrite the matrix accordingly. One idea is to include a
"regularization" (smoothing) term. This can sometimes help reduce
noise in your solutions. Another idea is to weight each edge
constraint by some measure of confidence. For instance, you might
want the constraints for dark pixels where the normal estimate is bad
to be weighted less heavily than bright pixels where the estimate is
good.
In our formulation of solving for normals, we weighted
each term by the image intensity, to reduce the influence of shadowed
regions. This approach has the drawback of overweighting
specular highlights, where they appear? Can you come up with an
alternate weighting scheme that avoids this? Can you apply it to
solving for albedos also? Depths?
Be creative! Do something we didn't think of...
Turnin
Also in your web page, include a discussion of your results. What
worked and what didn't? Which reconstructions were successful and
which were not? What are some problems with this method and how
might you improve it?
Bells and Whistles