CSE 455: Homework Set 4
Panoramic Mosaic Stitching

Materials

Contents

In this project, you will implement a system to combine a series of photographs into a 360° panorama.  Your software will automatically align the photographs (determine their overlap and relative positions) and then blend the resulting photos into a single seamless panorama.

What to Do

Note: The skeleton code includes an image library, ImageLib, that is fairly general and complex.  It is NOT necessary for you to peek extensively into this library!  We have created some notes for you here.

Writing the Code

Here are the algorithmic steps you'll take to convert a set of images into a panorama:

1. Warp each image into cylindrical coordinates. (file: WarpCylindrical.cpp, routine: warpCylindricalField)

   The code for this has been provided for you.

2. Compute the alignment of the images in pairs. (file: LucasKanade.cpp, routine: PyramidalLucasKanade, LucasKanadeStep)

    To do this, you will have to implement a hierarchical (coarse-to-fine) Lucas-Kanade style translational motion estimation.  The skeleton for this code is provided in LucasKanade.cpp.

   PyramidalLucasKanade constructs an image pyramid of specified number of levels for two images img1 and img2, and walks down the hierarchy (from coarse to fine), at each level updating the displacement of img2 from img1 by iteratively calling LucasKanadeStep.

   LucasKanadeStep takes two images img1, img2, and the initial translation vector (u,v) as input, and computes an updated translation (u',v') = (u+du,v+dv) which minimizes |img2(x+u',y+v')-img1(x,y)| over all x, y. Note that, instead of evaluating the image derivatives (Ix, Iy and It) between img2(x+u,y+v) and img1(x,y), it first creates img2t, a warp of img2 using the translation (u,v), then computes the image derivatives between img2t(x,y) and img1(x,y).

   [TODO] Fill in the missing code in LucasKanadeStep to:
   

   1. compute the per-pixel error and intensity gradients
   2. accumulate the 2x2 matrix and 2x1 vector
   3. solve the 2x2 system and update the translation estimate (Optical Flow slide 13)

3. Stitch and crop the resulting aligned images. (file: BlendImages.cpp, routines: BlendImages, AccumulateBlend, NormalizeBlend)
   

   [TODO] Resample each image to its final location and blend it with its neighbors (AccumulateBlend, NormalizeBlend). Try a simple horizontal “hat” function as your weighting function, where weight decreases linearly with horizontal distance from the center of the image (this is a simple 1-D version of the distance map described in [Szeliski & Shum]).  For extra credit, you can try other blending functions or figure out some way to compensate for exposure differences. In NormalizeBlend, remember to set the alpha channel of the resultant panorama to opaque!

Running the Program

Here's how you use your program to run the above steps:

1. Use the above program you wrote to warp/align/stitch images into the resulting panorama.
   
   • To remove the radial distortion and warp the image input1.tga into cylindrical coordinate with focal length = 600, radial distortion coefficients k1=-0.21 and k2=0.25 (v4gP1 is the name of the program):
     
         v4gP1 cylWarp input1.tga warp1.tga 600 -0.21 0.25
     
     
   • To align two images warp1.tga and warp2.tga which have a displacement of ~200 pixels horizontal and ~0 vertical, with 4 levels of image pyramid and 3 steps of Lucas-Kanade search at each level:
     
         v4gP1 LucasKanade warp1.tga warp2.tga 200 0 4 3
     
     
   • Run the previous step for all adjacent pairs of images and save the output into a separate file pairlist.txt which may look like this:
     
         warp1.tga warp2.tga 213.49 -5.12
         warp2.tga warp3.tga 208.19 2.82
         ......
         warp9.tga warp1.tga 194.76 -3.88
     
     
   • Then stitch the images into the final panorama pano.tga:
     
         v4gP1 blendPairs pairlist.txt pano.tga
     

   You may also refer to the file stitch2.txt provided along with the skeleton code for the appropriate command line syntax. This command-line interface allows you to debug each stage of the program independently.
   
   

2. Convert your resulting image to a JPEG. If you want, try to get it working with the interactive viewer.  Click here for instructions on how to do this.

For the test images, use the following parameters:
focal length = 595 pixels
k1 = -0.15
k2 = 0

For the full set of provided images, use the following parameters:
focal length = 674.79106 pixels
k1 = -0.21483
k2 = 0.32286

For extra credit, you can create your own panorama. You can find some information on how to compute the focal length for your camera here. If you make your own panorama, make sure you save your images in Targa format (.tga), make sure the aspect ratio of your images is either 4:3 or 3:4, and make sure the image is saved in 24-bit Targa, not 32-bit.

Turnin

1. Turn in the executable (v4gP1.exe).

2. Turn in the code that you wrote (just the .cpp files you modified and any new files you needed).

3. Turn in the test panorama and the provided full panorama (and any additional ones you made, for extra credit), along with a brief description of what you did.

Send the above to Ian (iansimon@cs).

This assignment is due on Thursday, March 10th, by 11:59 PM.