CS180 Project3: Face Morphing

Author: Ziqian Luo

Overview

This project involved creating an animation that morphs an individual's face into another individual's face, calculating the mean face of a population, and generating a caricature of myself by exaggerating deviations from the average. The morphing process was achieved by warping the image shapes and smoothly blending colors.

Key Objectives

• Generate an animation morphing a person's face into another person's face.
• Calculate the mean face of a population.
• Create a caricature of myself based on deviations from the population mean.

Techniques Used

• Facial keypoint selection and triangulation
• Affine transformations for image warping
• Cross-dissolve for color blending

Part 1: Defining Correspondences

To begin, I manually selected 76 keypoints (excluding boundary points) on both the source face and the target face to establish correspondences for key features like the eyes, nose, and mouth. These correspondences ensured that the morphing process would be smooth. I used the ginput function to mark these points. Below is an example of the resulting keypoints and triangulation:

Part 2: Computing the Mid-Way Face

To compute the mid-way face between my image and the target image, I computed the average position for each corresponding keypoint. Mathematically, given keypoints p_i from my face and q_i from the target face, the average keypoint location m_i is calculated as:

m_i = (p_i + q_i)/2

Using these average positions, I warped both faces into this new shape. The warping process was the most challenging part, as it required morphing both images to match the average geometry through inverse affine transformations, followed by filling in the pixel colors using bilinear interpolation to ensure smooth transitions. Finally, I blended the colors of the warped images to produce a smooth mid-way face, resulting in an average representation that visually combines features from both original faces. Below are the original images and the resulting mid-way face:

Part 3: The Morph Sequence

I implemented a function called morph(im1, im2, im1_pts, im2_pts, tri, warp_frac, dissolve_frac) to generate 45 frames of the morph sequence from my face to the target face. Each frame involved adjusting the shape and blending the colors. When warp_frac and dissolve_frac are set to 0, the resulting frame is exactly the first image (im1), meaning there is no warping or blending applied. Conversely, when both warp_frac and dissolve_frac are set to 1, the resulting frame is fully transformed into the target image (im2), with complete warping and blending. Values between 0 and 1 create intermediate frames that gradually transition from the source to the target.

Part 4: The Mean Face of a Population

I used the Danes dataset, which contains labeled facial images, to calculate the mean face of the neutral male population. I warped each individual face in the dataset to the average shape and then blended them together.

Additionally, I warped my face into the average shape and also warped the average face into my facial geometry:

Part 5: Caricature

To create a caricature of myself, I exaggerated the differences between my face and the average face using the equation (mean_shape + (my_shape - mean_shape) * alpha). The negative alpha looked particularly weird.

Bells and Whistles

PCA Analysis

I applied Principal Component Analysis (PCA) on the Dane dataset (the vectors are the flattened images) to analyze the major components of facial variation and understand how different facial features contribute to overall facial structure.

I created a random face using the eigenfaces by randomly assigning coefficients to the basis.

Also, I created various caricatures with different numbers of principal components exaggerated by a factor of 3. I think principal components have much greater impact on the overall image, and it would be better to tweak those principal components to change the major features.

Gender Transformation

I transformed my face from male to female using the average face of Han Chinese woman with three different methods: shape only (morphing my facial geometry to the mean face), appearance only (using my shape but cross-dissolving with the mean face), and both shape and appearance (warping and cross-dissolving).

Morphing Music Video

I created a morphing music video featuring all U.S. presidents (because they are all in public domain and will soon get updated). I utilized shape_predictor_68_face_landmarks.dat to automatically mark keypoints and crop the faces and then mark the keypoints again on those cropped images.

Here is the YouTube link of the music version featuring Star-Spangled Banner in Minor Key: Watch on YouTube.

Conclusion

This project provided hands-on experience with various image processing techniques, including affine transformations, triangulation, and color blending, to achieve face morphing, mean face calculations, and caricature generation. It deepened my understanding of facial structure representation and morphing methods.