Photo-realistic Face Manipulation in Scalismo

mona_lisa_collage

Collage of two manipulated versions of the Mona Lisa. The face was modified to appear more likable (left) and less likable (right) than the original. (original photo by C2RMF, Public Domain via Wikimedia Commons)

Perception and interpretation of faces play an important role in the communication between humans. In situations where we lack information about a stranger, we tend to form a first impression, solely based on the external appearance of that person. The face greatly influences our judgment of our vis-a-vis's personality traits. Moreover, we are able to capture subtle changes in a person's facial expression and infer information such as mood or intentions.

Our intrinsic knowledge about the natural appearance of faces as well as our ability to evaluate facial expressions enables us to spot artificial manipulations in face images rather easily.

In this article, we present an approach to photo-realistic manipulation of 2D face images using the open-source software framework Scalismo [1, 2]. Scalismo is a library for statistical shape modeling and model-based image analysis. It has its origin in the research done at the Graphics and Vision Research Group (GraVis) at the University of Basel.

We will first introduce the basics behind our model-based image analysis approach. In the second part of the article, we show how it is used as a tool in the field of psychology. With an accompanying demo application, convincing manipulations of perceived personality traits can be applied to user provided photographs.

The demo application can be found on: https://face-morpher.scalismo.org

A model for face analysis

In order to be able to manipulate a face in an image, a quantitative representation of that face is needed. For this purpose, we make use of a statistical model of the face, namely the Basel Face Model (BFM) [3]. The BFM was constructed from 200 3D face scans from 100 female and 100 male persons, most of them Europeans.

Using principal component analysis, the main modes of variation (eigenmodes) for the shape and the texture in those scans were extracted. Figure 1 visualizes the first 5 shape eigenmodes of the BFM. A face, in the space spanned by these eigenmodes, can therefore be represented by a finite set of parameters.

Figure 1: Representation of the variation (-2 to +2 std. deviations) along the first 5 principal component axes for shape (eigenmodes) of the the BFM.

By sampling these parameters, we can generate new synthetic faces (see Figure 2). Due to the statistical nature of the model, the generated faces respect the natural variations and proportions of the human face.

Figure 2: Set of samples from the BFM. Here, the model parameters have been randomly drawn from the range [-2; 2].

Finding faces in images

Using a registration process, we establish correspondence between the 2D representation of a face in an image and the 3D face model. This is achieved by using an analysis-by-synthesis approach. In this fitting process, the rendering parameters (e.g. camera position, illumination) and model parameters for shape and texture are iteratively adjusted, until the 2D rendering of the synthetic 3D face is close enough to the target image [4]. Figure 3 illustrates the model fitting process. As a result, we get a 3D reconstruction of the face in the 2D target image.

Figure 3: Visualization of the fitting process using portraits of the actors Steve Carell (left) and Uma Thurman (right). Starting from the mean face of our model, the parameters are iteratively adjusted until a good similarity with the target image is reached. (original photos by Gorup de Besanez CC-BY-SA via Wikimedia Commons and Sgt. Michael Connors Public Domain via Wikimedia Commons)

Photo-realistic face manipulation

Once we have obtained the representation of a face in terms of our model, we can change its appearance by manipulating its parameter vector. Having a direct correspondence between points on the 3D model and the target image, allows us to project manipulations, performed in the 3D model space, back into the 2D image.

Since the parameters of a model fit are a quantitative measure for the characteristics of a face, caricatures can be created by a simple scaling operation. As with real caricatures, artifacts appear as the effects gets stronger. This is visualized in Figure 4 using the face fits of Steve Carell and Uma Thurman from the previous section.

Figure 4: Caricatures of Steve Carell (left) and Uma Thurman (right) by scaling the shape parameters of the face fit.

Instead of emphasizing the features of one persons face, we could also morph one face into another by creating a weighted sum of the respective parameter vectors. In Figure 5, the face of Steve Carell is gradually morphed into the one of Uma Thurman.

Figure 5: Mixing of two face fits. The animated GIFs show the transition from Steve Carell to Uma Thurman. Left: Warp of the shape reconstruction. Right: Combined warp of shape and texture.

In general, we can express arbitrary manipulations as a vector in the BFM model space. The manipulations can be applied to any face by adding/subtracting the corresponding vector to/from the parameter vector of the target face. The magnitude of the effect is regulated by scaling the vector. After applying the manipulation, the modified 3D representation is projected back into the 2D input image. As an example, the effect of a manipulation for the appearance of the nose is shown in Figure 6.

Figure 6: Effect of a manipulation vector for the size of the nose using a portrait of Leonhard Euler. Left: Manipulation applied to 3D shape reconstruction. Right: Manipulation projected back in to 2D image. (original photo by Kunstmuseum Basel Public Domain via Wikimedia Commons)

Application: Modeling perceived personality traits

As mentioned above, subtle changes in a person's face can have a strong influence on how this person is perceived by others. This phenomenon was studied by Walker [5, 6].

In several studies, participants were presented face scans and asked to answer various questions about their perception of the persons in the image. Using the collected data, vectors in the BFM model space for several selected personality dimensions were reverse engineered.

The so obtained personality vectors can be used to manipulate the perceived personality traits of a person in a real photograph. See Figure 7 for a visualization of a manipulation along the perceived personality dimension of aggressiveness.

Figure 7: Variations of Leonardo Di Caprio's face along the perceived personality dimension of aggressiveness. Left: Manipulation of 3D shape reconstruction. Right: Projection into 2D target image. (original photo by Colin Chou CC-BY-SA via Wikimedia Commons.jpg))

Online Demo: ScalismoFaceMorpher

We present an interactive demo application to illustrate the method introduced above. Faces are automatically detected in user provided images and correspondence to our 3D face model is established. The user has then the possibility to apply a set of manipulations from the above mentioned study.

https://face-morpher.scalismo.org

Images generated with this service may only be published together with a proper mention of the following publications: - Gerig, T. et al., Morphable Face Models - An Open Framework. 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi'an, 2018, pp. 75-82. - Walker, M. & Vetter, T. (2016). Changing the personality of a face: Perceived Big Two and Big Five personality factors modeled in real photographs. Journal of Personality and Social Psychology, 110(4), 609-624.

Further reading

The website of Mirella Walker (Division of Social Psychology, University of Basel) contains further information about the use of face modeling in psychology.

References

  1. Scalismo - Scalable Image Analysis and Shape Modelling. Graphics and Vision Research Group, University of Basel. https://github.com/unibas-gravis/scalismo
  2. Scalismo-faces - Scalismo library for image processing, rendering and handling of the Morphable Model. Graphics and Vision Research Group, University of Basel. https://github.com/unibas-gravis/scalismo-faces
  3. Gerig, T. et al., Morphable Face Models - An Open Framework. 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi'an, 2018, pp. 75-82.
  4. Schönborn, S. et al., Markov Chain Monte Carlo for Automated Face Image Analysis. Int J Comput Vis (2017) 123: 160.
  5. Walker, M., & Vetter, T. (2016). Changing the personality of a face: Perceived Big Two and Big Five personality factors modeled in real photographs. Journal of Personality and Social Psychology, 110(4), 609-624.
  6. Walker, M., Vetter, T. (2009). Portraits made to measure: Manipulating social judgments about individuals with a statistical face model. Journal of Vision 2009;9(11):12.