Cullen Su

Computer Vision (CSE 455), Winter 2012

Project 4: Eigenfaces

In this project, we tried to find faces using eigenfaces. How this worked is by getting a set of training faces and calculating a set of eigenvectors that we use to define a new coordinate system. We used PCA to reduce our problem to something more manageable. This works because the set of face vectors defines a face plane and we can label each potential face with respect to the average face and a coefficient times an eigenface.

The coding was especially difficult, i ran into difficulty at first because the code was terribly documented. I felt like I knew exactly what I wanted the code to do, but had a lot of trouble finding the right way to get the code to act as I wanted with the skeleton code. Also, reducing the false positives was very difficult, I tried the method in the writeup where you multiply by the distance of the face to the average, and then divide by the variance, but the false positive rate is still high.

I learned that we can project a face with respect to eigenfaces. This works because each eigenvector of A essentially defines a direction in the space of images and we can get to any image with a combination of the eigenvectors times some coefficient. Just like how in 2d you can define any point with respect to the average orange point and v1/v2 in the slides, I learned that the same methodology works for any number of dimentions.

This goal of this project is to recognize and find faces in images. This works by first getting some training data to generate a set of eigenfaces, then comparing various images (faces) to our average training face and a linear combination of eigenfaces to see if we can classify an image as a face We basically redefine the image in terms of eigenfaces by getting the coefficients of the eigenfaces and then see how close our coefficients bring us to the image. If the image is a face then the reconstructed image should be pretty close to our input.

The project is structured as follows:

• Faces::eigenFaces - Gets a list of files for training faces and calculates the average face and the eigenfaces. It treats each image as a column vector and then multiplies (x-xbar) with (x-xbar)T to get a square matrix A that is size (width*height)x(width*height). Then it uses the jacobi function to get the eigenvectors of A and lastly picks n eigenvectors with the largest eigenvalue and stores them as eigenfaces.
• EigFaces::projectFace - Calculates the coefficients a1...ak by taking the dot product of (x-xbar) and each eigenface. This is essentially computing the projection of the face-averageface onto each eigenface.
• EigFaces::constructFace - This takes a list of coefficients and uses them to construct a face images. It takes those coefficients and applies them to the eigenfaces and then adds the average face to generate the face.
• EigFaces::isFace - This takes a face and a max_tolerance_mse to see if the mse of the face is less than the tolerance. It works by projecting the face into eigenface coordinates, then constructing a face with those coordinates and taking the MSE between the input face and the constructed face.
• EigFaces::verifyFace - This takes a list of coefficients for a user, and a face image. It projects the face image to get the coefficients of the face, then compares it to the input coefficients to determine if it is a match or not. It takes the mse of the two coefficients to see how alike they are.
• EigFaces::recognizeFace - This takes a list of users and a face image to see if it can find a matching user to the face. It searches the list of users and compares the coefficients with the projected face's coefficients. Then it sorts the users by mse and returns.
• EigFaces::findFace - This takes a image, scales to search through, number of faces to find, and returns either a cropped image to the best face or an image with boxes drawn around the best n faces. This works by searching through different scales of the image for a match with the eigenfaces and takes the best n face matches to draw boxes on. It also has to determine if a box overlaps with a different box and it will only take the best box if it overlaps with another box.

We computed 10 eigenfaces (25x25) from the set of neutral faces. Then we used that to create a userbase and here is the resulting eigenfaces/average face. The average face is on the left, and the 10 eigenfaces are on the right.

I also used a varying amount of eigenfaces computed to try and match smiling pictures to neutral pictures. Here is a plot of number of matches vs number of eigenfaces used.

Direct response to Question 1.

The plot suggests that after a certain number of eigenfaces it doesn't help to use any more eigenfaces. After 5 eigenfaces it seems to flatten out and suggest that the differences between using 5, 10, or 30 eigenfaces isn't very much when we are calculating matches in the userbase. It seems like the best number of eigenfaces to use would be somewhere around 7-10 because then you don't end up computing too many eigenfaces which can slow down computations by requiring more values being computed through projections.

Direct response to Question 2.

The images that didn't match up were fairly different between smiling and neutral. When there was an error, the error was usually reasonable and the correct andwer was usually in the top few matches. Here is a pair that was not matched. I think this is a reasonable mistake, because I would probably have made the same mistake.

Here is another example of mismatched pairs.

The results probably didn't match up very well in those cases because the differences between smiling and neutral were too much. Also, sometimes the program matched two people who looked alike together, which is reasonable. The fact that using more than 7 eigenfaces didn't seem to change anything was interesting, it sent against what I thought would have happened. I guess it means that the majority of identifying information is already captured in the first few eigenfaces and the rest are just unnecessary for identifying purposes.

We used the 10 eigenfaces computed previous to try and find faces in images.

I also tried to use the program to find faces in group pictures. The false positive rate was high, most groups of 3 had only 2 real faces in the top 3 faces. I tried the method in the writeup, but it did not seem to work and I spent far too long trying to fix this issue. In fact, this entire project took far too long, the writeup and experiments by themselves took longer then the average 400-level cs projects I've done...Here is an example of what I have. It CAN find faces, but it also thinks some other things are faces...

Direct response to Question 1.

The easiest way I did this was by getting the picture's dimensions, then estimating how many pixels wide the face for the people were. Then I used that to decide how to scale and I generally scaled in 0.01 or 0.02 with an interval of 0.04 in each direction from what I thought the scale was. This saved time because searching a large number of scales took a long time.

Direct response to Question 2.

I can only assume that I somehow screwed up modifying the mse because it just didn't work very well in general on the group pictures.

I don't know why the results were off, I tried to figure it out but nothing I tried worked. The code did find the faces, but it would sometimes rate a non-face higher than a real face. .

We used 6 eigenfaces in this part to see if we could verify people from their neutral faces to smiling faces.

Direct response to Question 1.

To find a good threshold, you can just use the output from the previous "best match" mse as a guideline and kind of binary search/guess your way to a good threshold. I really don't have time to sit here and do this over and over...The MSE for the best matches were generally around the 30k-80k range with 6 eigenfaces.

Direct response to Question 2.

As the MSE threshold goes up, you will get more matches in general, and as a result you will get more "yes" answers when you are testing 2 images of the same person, but you will also get more "yes" answers when testing 2 images of different people.