In this project, we want to create a face recognition program that recognize faces in an image provided with a set of face training images. From the set of training images as vectors, the program uses Principal Component Analysis (PCA) to find the space of faces. Since the space is spaned by a few vectors, each face can be represented by a combination of these vectors with some coefficients weighting each vector. We use the mean square error between an image area in the image and its projected face (projecting onto the face space we created) to determine whether that image area is a face or not.

The most challenging part of this project is to actually finding the faces given various images. We find that the program frequently recognize background wall (soft-lighting) to be faces

After series of experiment with pictures of different light settings, different number of people and facial expressions, we learned that the program works well with some settings and poorly with others. For example, a picture with more than 10 people tends to be really hard for the program to work well. When the faces are tilted at an angle or have more extreme facial expressions, it doesn't work very well either.

Some important classes in this project are:

• Vector - representation of a vector
• Image - derived from Vector, representation of an image
• Face - derived from Image, has additional functionalities like saving as a .tga file and creating a face from a subimage of an image
• Faces - an array of Face
• Eigfaces - derived from Faces, stores all the eigenfaces, contains the important functions in this project

There are also classes that handle file IO and users/userbase.

The project is structured as follows:

• Faces::eigenFaces - Given a set of training face images, creates top n eigenFaces using the PCA
• EigFaces::projectFace - Projects a face onto the face space to generate k coefficients, one for each eigenfaces.
• EigFaces::constructFace - Given k coefficients, reconstruct the face using the average face and eigenfaces.
• EigFaces::isFace - Determines whether a given image is a face by computing the Mean Square Error(MSE) of the the projected face (projected onto the face space) and the original
• EigFaces::verifyFace - Determines if the given face is of the specified user by computing the difference of the MSE of the two faces projected onto face space.
• EigFaces::recognizeFace - Matches the given face with the closest face in the user base by computing the difference of MSE of the two projected faces.
• EigFaces::findFace - Finds n faces in the given image by scaling the image and use the isFace function to search for possible faces.

Here's the average face and the top 10 eigenfaces:

Average face

Eigenfaces

Using the provided script, I experimented with the number of eigenfaces created (1 to 33) from the nonsmiling face images. From each case, I used the recognizeFace function to recognize faces from the smiling face images. Then I recorded number of correctly recognized faces along with the number of eigenfaces in Excel and plot a graph.

In the plot, it can be clearly seen that there's a rapid increase in the number of correctly recognized faces as the number of eigenfaces increases. The increase however becomes very gradual around 8 or 9 eigenfaces. After this point, the number of correctly recognized faces incresed only 1 to 2 even if the number of eigenfaces is increased to 33.

There are some recognition in this experiment. However, examining the faces pictures that are falsely recognized by the program, we find that they are very similar to each other, by facial expression, skin color, or glasses. Below are some example pairs:

The above result pairs are some of the pairs that the the program consistantly recognizes correctly. We can see the similarity of facial expression in them.

The above result pairs are also what the program consistantly recognizes correctly.

Using the findface function, I vary the scaling factors as well as the number of faces we intend to find in order to produce the best face detection result.

I also implemented 2 algorithms to improve the accuracy. One is multiplying the MSE by the distance between the face and the average face and then dividing by the variance of the face. The other is to save original training face images and count how many pixels this face image has that are very close in color with the pixels in the training face images. Both algorithms, however, don't make noticeable improvements. More over, the second algorithm slows down the program significantly, and thus is not included in the code I turned in.

The min_scale, max_scale, and scale step I used are listed below each image.

There are false positives and flase negatives. For most of the false positives, they generalize into 2 categories. One is with background-walls that are light-orange-brown color because of the lighting. Another is when edges create some shapes that are similar to the shape of a face; for example, I frequently get false positives that are at a bent elbow. For the false positives, some of them are caused by tilted or object blocked faces.

I was able to find the baby's face when I specified the number of faces to be 3.

main --findface elf.tga nonsmile_eig10.face 0.45 0.55 0.01 mark 3 findResultElf.tga

These are some group shots from our class. I can find at least 1/2 of the faces in the pictures. In some cases though, I had to specify more faces than there actually are.

main --findface faceImages\group\group_neutral_2.tga nonsmile_eig10.face 0.60 0.95 0.05 mark 3 findResult2.tga
main --findface faceImages\group\group_neutral(11).tga nonsmile_eig10.face 0.55 0.95 0.05 mark 4 findResult11.tga
main --findface faceImages\group\group_neutral(1).tga nonsmile_eig10.face 0.60 0.90 0.05 mark 4 findResult1.tga

This is a panorama picture of all the students int our class. I was able to find 7 faces in this group picture. As mentioned above, the program doesn't perform spectacularly with group photos that contain a lot of people.

main --findface faceImages\group\class_pano_handheld.tga nonsmile_eig10.face 0.20 0.30 0.05 mark 45 findResultClass.tga

This is a photo of my friend and I when we went to BBC TV Station to visit. I found both of our faces by specifying the number of faces to be 4.

main --findface me1.tga nonsmile_eig10.face 0.10 0.40 0.05 mark 4 findResultMe1.tga

This is a picture of 3 of my friends and I in Spain. I was able to find (almost) 2 out of 4 faces.

main --findface me8.tga nonsmile_eig10.face 0.45 0.65 0.01 mark 4 findResultMe8.tga

Using a user base made from cropped non-smiling faces and 6 eigenfaces, call verifyFace function on a nonsmiling face and smiling face images of the same person. Also call verifyFace function on a nonsmiling face and smiling face images of different people. Compare the MSE's from all the user results and obtain the best MSE threshold.

I wrote a script to verify each cropped smiling face with the cropped non-smiling face of the same person in the user base. The partial output is below. Then I tried verifying some smiling faces with non-smiling faces of different people in the user base. I came up with an MSE threshold of 2200.0 which I think works well because it distinguishes whether a face is of a particular person correctly for most cases.

Using the MSE threshold mentioned above, we get about 15% of false negative. It's worth noting that for most of the cases where the program can't verify correctly the smiling faces are from the same person as the non-smiling faces, the facial expressions in the smiling faces are extreme, with face tilted, turning sideways, or with mouth opened. Below are some examples: