Tracking for Projector-Camera Systems

In dynamic scenes, object tracking maintains the illusion of an enhanced target object if the physical and virtual objects are well aligned. However, traditional trackers based on texture or contour information are often distracted by the projected content and tend to fail. We have developed a model-based tracking strategy, which directly takes advantage from the projected content for pose estimation in a projector-camera system.

An iterative pose estimation algorithm captures and exploits visible distortions caused by object movements. In a closed-loop, the corrected pose allows the update of the projection for the subsequent frame. 

Synthetic frames simulating the projection on the model are rendered and an optical flow-based method minimizes the difference between edges of the rendered and the camera image. Since the thresholds automatically adapt to the synthetic image, a complicated radiometric calibration can be avoided.

The pixel-wise linear optimization is designed to be easily implemented on the GPU. Our approach can be combined with a regular contour-based tracker and is transferable to other problems, like the estimation of the extrinsic pose between projector and camera.

The following video shows how the 3d pose of the moving object is estimated from the distortion of the projection, which is then corrected to match the object surface again.

The recent trend towards miniaturization of mobile projectors is allowing new forms of information presentation and interaction. Projectors can easily be moved freely in space either by humans or by mobile robots. Our methods are also able to dynamically track the orientation and position of the projection plane only by analysing the distortion of the projection by itself, independent of the presented content. This enables distortion-free projection with a fixed metric size for moving projector-camera systems.

The algorithms have been integrated in an augmented reality (AR) software implementing the closed-loop projection-based object tracking. The software allows to create and configure AR scenes, performs pose estimation in real-time and provides a websocket interface to communicate with other sensors in a multimodal system.