Reconstruction and Rendering of the Human Head

We are developing algorithms for reconstructing the human head in high detail from calibrated images. Our method is passive, i.e. we do not use projected patterns (as, for example, structured light approaches do) and we don’t use photometric normals. We are interested in recovering not only fine details in the face but also the appearance and some geometric detail of the subject’s hair, thereby capturing the complete human head. This is challenging due to the intricate geometry of hair and its complex interaction with light.

Our approach is based on a generic nonlinear, image-based warp estimation scheme which we have used for 2D applications such as tracking or camera deshaking. Roughly, it can be seen as patch-based stereo with connected patches (i.e. “mesh-based stereo”) and simultaneous optimization of all patches with smoothness constraints.

We developed a piecewise hierarchical approach to warp estimation which allows us to process images of very high resolution. We found that for images with moderate baselines correspondences of very high quality can be recovered with a completely generic warp which is agnostic of camera geometry. This requires a coarse-to-fine strategy not only for the image size but also for the meshes. Also, lighting inconsistencies need to be compensated by a photometric correction.

We also investigated a low-detail patch-based approach to head reconstruction and rendering. We used our warp estimation framework to recover the position and orientation of a planar patch in space. In contrast to most patch-based methods, we didn’t aim for a highly detailed geometry. Instead we used a small amount of textured patches to render the head from different viewpoints. For small viewpoint changes as required, for example, for view interpolation, this works amazingly well.