The part I spend most time on was implementing Lucas-Kanade algorithm. I tried to create a mosaic using the sample executable and the result was terrible and most images were not aligned correctly. After adjusting the x and y offset input, I got a fairly good image. Lucas-Kanade seems to be pretty sensitive to the initial offset between two images. So I used the set of adjusted x and y offset for my debugging. I tried two different equations to compute the x and y gradients. Sobel operator gave slightly improved results in assigning images.
Some other issues were
Tripod sequence: Some images had very different exposure but they align well after I manually adjust the exposure and color balance.
Handheld sequence: I didn't pay much attention in keeping the center of rotation and the horizontal level of images. As a result, my mosaic has some black area near the bottom of the image after fixing vertical drift.