ofxReprojection

Calibration test on 2013-09-06:

Draft

ofxReprojection is still at the draft stage. Do not expect it to be working.

Description

ofxReprojection is a toolkit for dynamic projection mapping using a depth cam and projector.

The depth cam (e.g. the Kinect camera for XBox) will give you a 3D image of the world. The projector outputs a 2D image onto the 3D world. By finding a tranformation from the 3D point from the camera to the 2D points on the projector, we can analyze the world and project color back onto it. This is done by a camera/projector calibration where a chessboard (or other automatically recognizable shape) is projected onto the world, and then filmed by the depth camera.

By using computer vision methods (provided by the OpenCV 2 library), we can automatically identify the chessboard pattern on the camera image.

As the depth cam gives a third corrdinate for each (x,y) point, we now have a set of points in the camera's 3D space and a set of corresponding points in the projector's 2D space (the coordinates of the chessboard we drew using the projector). This can be used to estimate a transformation from the higher dimensional space to the lower (3D to 2D). We assume that the transformation is affine, i.e. that it is linear in homogenous coordinates.

The approximate transformation is found by using a least squares fit to a 4x3 matrix. The Levenberg–Marquardt algorithm (as implemented in the lmfit C library) is used to solve the regression .

Requires

• openFrameworks v0.8.
• Depth camera (e.g. Kinect) through an openFrameworks addon offering the ofxBase3DCam interface. Known alternatives:
  • ofxONI2, which requires OpenNI 2 and optionally NiTE 2. These are experimental releases, and some have problems with them.
  • ofxONI1_5, which requires OpenNI 1.5 and optionally NiTE 1.5. These are more stable for some testers.
  • ofxKinect, which is included in openFrameworks. Probably the simplest and most stable solution, but does not support NiTE which has the cool user tracker features.
• Projector configured as an extended screen/desktop.

Quick start

• Install and test a depth cam addon.

• Clone the ofxReprojection addon repository into your openFramworks addons folder:

    cd ~/openframeworks/addons/
    git clone https://github.com/luteberget/ofxReprojection.git
  

• Build the simple example for your depth cam addon (e.g. example-ofxKinect).

• Run the example. It consists of two stages:

  • Calibration stage. Here you will see the color and depth images from the depth cam on you screen, and your projector should be showing a chessboard. Arrange an object (such as a piece of cardboard) in various positions so that the chessboard is detected by the program.
  • When you have enough chessboard measurements (5-10 should suffice), finalize the calibration.
  • Rendering stage. Here you will see a hue-valued depth image projected onto the scene using the projector. The image from the projector should approximately match the actual space your projector and depth cam are pointed at.

• Install NiTE and test one of the user tracker examples.

Documentation

Coming.

Authors

ofxReprojection is made by Bjørnar Steinnes Luteberget and Tom Andreas Nærland.

Licence

ofxReprojection is free to use for non-commercial purposes.