Non-Rigid Structure-from-Motion

Reconstructing the 3D shape of a deformable object from a monocular image sequence is a challenging problem, because multiple shape configurations can produce the same image projection. We address the problem of reconstructing volumetric non-rigid 3D geometries under full perspective projection by employing a 3D template model of the object in rest pose. Volumetric non-rigid reconstruction is even more challenging than the reconstruction of planar-like surfaces, because only the front part of the object surface is visible in the image, while the back part and the interior have to be inferred without direct image information. While the object starts to deform, captured by only a single camera, the non-rigid shape is reconstructed sequentially by estimating the camera parameters and the deformation with respect to the template model in an optimization framework.

Non-rigid structure from motion plays an important role in computer vision application such as human-computer interaction, motion capture, tracking, etc. Without the knowledge of a shape deformation model, this task is severely under-constrained, because multiple shape configurations can project to the same image location. Our approach builds on volumetric template-based approaches and can be divided in two components:

1. Template computation of rest pose
2. Estimation of camera and shape deformation

First, we compute a 3D template model of the rest pose from a multi-view image set with traditional rigid reconstruction techniques. The template serves as geometric and topological prior for the NRSfM task, where the template model is modified in order to satisfy the constraints imposed by the new input image depicting the object in a deformed state. The energy function minimized comprises two main terms, one accounts for the data fitting, the other controls the smoothness of the deformation. The data fitting term enforces that specific 3D surface points project to the correct image location and penalizes volume configurations that project outside the object silhouette. The deformation is regularized three-fold by taking into account temporal and surface smoothness as well as volume preservation.

These videos shows an evaluation of our NrSfM approach on the galloping horse data set of Robert Sumner.