Combining online and offline information

The great problem right now is getting rid of error accumulation, which seems to be possible using the keyframe approach.

Well, reading the paper "Fusing Online and Offline Information for Stable 3D Tracking in Real-Time", Vacchetti, Lepetit, Fua, the following procedure is used to combine information from keyframe and previous tracked frame:

"In our approach we therefore attempt to combine the strengths of both the online and offline information as follows: first, we match the current frame with the chosen keyframe and apply RANSAC to the set of points we found, discarding the outliers and retaining a set of points Mk free from error accumulation. Then, we perform a modified RANSAC estimation over the matches between the current frame with the previous one: if an R,T sample tested by the RANSAC estimator rejects some points in Mk , this sample is not considered by this second stage. This way, this stage estimates the values R,T using all the points in Mk , which provide reliable but partial information, and the matches between the previous and the current frames that provide additional information."

Reading their following paper, the combination of information is more mathematically elaborated, yielding to the bundle adjustment problem, which seems to use the Levenberg-Marquardt algorithm, which is basically a least-squares for non-linear functions (as is the case of the projection error). The implementation of Levenberg-Marquardt seems to have been already made in Gnu Scientific Library. Checking "Multiple-View Geometry" bundle adjustment chapter and the paper "Model-Based Bundle Adjustment with Application to Face Modeling" are also good ideas to see if this is the correct way to go.

For now, the simpler approach of "Fusing Online and Offline Information for Stable 3D Tracking in Real-Time" will be used. If it does not yield good results, this other more formal approach will be implemented.

ps: Should take a closer look at http://www.thepixelfarm.co.uk/ PFTrack

Comment by mierle, Feb 19, 2009

You may want to check out the paper Bundle Adjustment Rules. They do incremental bundle adjustment of the last N frames to great effect for real time 3D tracking. Also, if you are going to do bundle adjustment I suggest checking out the sparse bundle adjustment package SBA. Also, you may want to check out libmv (shameless plug, I started the project) .

Comment by danielbaggio, Feb 19, 2009

Thank you a lot, Mierle. I'll check both the papers and softwares. I'll probably use the keyframe approach firstly and then the bundle adjustment. By the way, do you know if they are appropriate for real time processing? Thank you a lot for your comment.

Comment by wangxin.cpp, Feb 19, 2009

Hi,danielbaggio. The EHCI project is very helpful for me. I have learned lots of things from it. I'm also trying to implment the paper "Fusing Online and Offline Information for Stable 3D Tracking in Real-Time" based on your current EHCI open-source project.

Do you have any idea about how to mapped the 2D interestd points back to the 3D head model surface? I'm not familar with the OPENGL SDK, is it a opengl programming trick?

Comment by danielbaggio, Feb 20, 2009

Hi, wangxin :)

It's really nice to know that I'm not alone in this task. Well, I'm using a sinusoidal model, so, it's quite straightforward to backproject the points (they are normalized by head width and height and it's just a sin function in the Z axis), but I had tried to use a head model before that.

The idea is to draw the 3d head using OpenGL and colour each of the cells with a different colour. I think that indexed colours are the way to go. If I'm not mistaken you use a function that retrieves the colour of a pixel and then you map back to which id you used to draw that colour. The OpenGL function is probably glReadPixels http://www.opengl.org/documentation/specs/man_pages/hardcopy/GL/html/gl/readpixels.html

I'm also in favour of benchmarking the time to see the plane-line segment intersection algorithm. I know that callbacks to GPU have some delays, and the algorithm is quite linear to the number of surfaces. I'm still in doubt if using the GPU is really the fastest procedure. So, it's worth an attempt.

Thank you all for your help :)

Comment by wangxin.cpp, Feb 26, 2009

Thank you very much, danielbaggio.

You said you had used a head model before. So, Did the really 3d head model gained better tracking results than the sinusoidal model?

Comment by danielbaggio, Feb 26, 2009

Hi,

when I tried to use the head model I used the colour/face id but I was getting float rounding errors that could give me the wrong face id back (through the readpixel idea), so I didn't go that much further with the implementation. But I believe I'll see some enhancements, mainly in the point head above and point down movements (which might confuse the sinusoidal model). Well, let's give it a try ;)