CSE 590SS Project 1: Panoramic Mosaic Stitching

version 4.0, January 20, 2001

OUT:  Jan 10, 2001

DUE:  Jan 24, 2001

In this project, you will implement a system to combine a series of photographs into a 360° panorama.  Your software will automatically align the photographs (determine their overlap and relative positions) and then blend the resulting photos into a single seamless panorama.  You will then be able to view the resulting panorama inside an interactive Web viewer.  To start your project, you will be supplied with some test images, skeleton code you can use as the basis of your project, and instructions on how to use the viewer.

Here are the suggested steps you should follow (skip steps 1 - 3 for test data):

1. Take a series of photos with a digital camera mounted on a tripod.  You should sign up to borrow the Kaidan head that lets you make precise rotations and the Nikon 990 Camera for this purpose. For best results, overlap each image by 50% with the previous one, and keep the camera level using the levelers on the Kaidan head.
2. Also take a series of images with a handheld camera.  You can use your own or sign up to borrow the Canon G1 camera from Cris Mesling (mesling@cs.washington.edu). If you are using the Canon camera, it has a “stitch assist” mode you can use to overlap your images correctly (this works in regular landscape mode).
3. Make sure the images are right side up (rotate the images by 90° if you took them in landscape mode), and downsample them to a more workable size (around 400x500--25% size for the course cameras). You can use external software such as PhotoShop or the Microsoft Photo Editor to do this.  If you are using the skeleton software, save your images in (TrueVision) Targa format (.tga), since this is the only format the skeleton software can currently read.
4. Warp each image into cylindrical coordinates.  The easiest way to do this is to flesh out the skeleton code in WarpCylindrical.cpp by adding the necessary coordinate transformation equations (from the lecture notes).  Warp each image using the warpCylindrical routine, using the following focal length f estimates for the quarter-resolution images (you have to scale f with image size):

   ·        Nikon Coolpix 990,        384x512,      f  = 595 pixels,      f.o.v. = 46.5°,    k1 = 0.15,  k2 = 0.0

   ·        Canon Powershot G1,     384x512,      f  = 523 pixels,      f.o.v. = 52.1°,    k1 = 0.13,  k2 = 0.0

5. Compute the alignment of the images in pairs.  To do this, you will have to implement a hierarchical (coarse-to-fine) Lucas-Kanade style translational motion estimation.  The skeleton for this code is provided in LucasKanade.cpp.  You will have to fill in the missing code in LucasKanade to:
   1. compute the per-pixel error and intensity gradients
   2. accumulate the 2x2 matrix and 2x1 vector
   3. solve the 2x2 system and update the translation estimate
   The control routine that uses the image pyramid to search for the best displacement at a coarse level, and then refines these estimates using Lucas-Kanade has already been written for you.  Once you have computed each pair-wise translation (including the translation between the last image and the first one), write these numbers out to a file.
6. Stitch the resulting aligned images.  Read in a descriptor file containing the warped image names and their relative displacements.  Figure out how large the final stitched image will be.  Then, resample each image to its final location and blend it with its neighbors.  Try a simple horizontal “hat” function as your weighting function (this is a simple 1-D version of the distance map described in [Szeliski & Shum]).  For extra credit, you can try other blending functions or figure out some way to compensate for exposure differences.
7. Crop the resulting image to make the left and right edges seam perfectly.  The horizontal extent can be computed in the previous blending routine since the first image occurs at both the left and right end of the stitched sequence (draw the “cut” line halfway through this image).  Use an affine warp to remove any vertical “drift” between the first and last image. Optionally, remove any black (a = 0) pixels at the top and bottom edges.
8. Convert your resulting image to a JPEG and paste it on a Web page along with code to run the interactive viewer.  The instruction on how to do this can be found at http://www.cs.washington.edu/education/courses/cse590ss/01wi/projects/project1/panoviewing.htm .

   Turn in the code that you wrote (just the .cpp files you modified and any new files you needed), and a web page containing the following:

   • At least three panoramas:  (1) the test sequence, (2), one from the Kaidan head, and (3) one from a hand-held sequence.  Each panorama should be shown as (1) a low-res inlined image on the web page, (2) a link that you can click on to show the full-resolution .jpg file, AND (3) embedded in a viewer as described above.

   • a short description of what worked well and what didn’t.  If you tried several variants or did something non-standard, please describe this as well.  

   Turnin instructions:

   • Create a "turnin" directory in your class data folder:  \\gfilesrv1\projects\courses\cse590ss\<yourname>
     If for some reason you don't have a folder here, send email to seitz@cs.washington.edu immediately.

   • Create a "project1" directory inside "turnin" and put your web page (as "index.html") and other files, including panoramas, code source, and binaries in the project1 directory.

For instructions on the skeleton code, click here.

Extra credit:  You can try some of the following for extra credit:

1. Try shooting a sequence with some people moving in it.  What did you do to remove “ghosted” versions of the people?
2. Try stitching something other than a cylindrical panorama, e.g., seam together two scans from a flatbed scanner.  Warning:  this is a fair amount of work, since your alignment code now needs to estimate rotations or even more.
3. Detect and remove moving objects from your panorama, filling in the background behind them.