My favorites | Sign in
Project Home Downloads Wiki Issues Source
READ-ONLY: This project has been archived. For more information see this post.
Search
for
ModuleIntegration  
Updated Dec 10, 2012 by an...@norture.com

In this section we propose two way to fuse commands coming from speech and gesture. The two fusion strategies are as follows:

  • Primary Integrator : This integrator makes command recognition more robust. It cross verify the commands from both modules.
  • Complementary Integrator : Here one module acts as a complementary action to the other.

Pairing between speech and gesture commands:

Speech Gesture
Forward Forward
Backward Backward
Left Left
Right Right
Slower Slower
Faster Faster
Stop Stop

Primary Integrator

Purpose of this integrator is to fuse information coming from two modules. Gesture module gives distribution: confidences for all commands present in database. However, speech module gives only "Winner" (command with highest confidence) and it's confidence. So we take the winning command from both modules and checks them.

In case when the second command doesn't arrive in three seconds, we execute the available command if it confidence is greater than threshold (0.9). Logic of integrator is as follows:

if (gesture.command == speech.command)
    if (gesture.confidence > gesture.lowThreshold ||  speech.confidence>speech.lowThreshold)
        execute command   // it is enough, since they are same
    else
        say "Again Please"    // robot is not certain about either modality
else   // commands are different

     if (gesture.confidence > gesture.highThreshold && speech.confidence < speech.lowThreshold)
        execute command   // gesture is very certain, speech is not certain
     else if (speech.confidence > speech.highThreshold && gesture.confidence < gesture.lowThreshold)
        execute command   // speech is very certain, gesture is not certain
     else if (speech.confidence>speech.lowThreshold && gesture.confidence > gesture.lowThreshold)
        say "Again Please" //both confidences are high. Cannot decide.

Complementary Integrator

It is meant to integrate speech and gesture commands in a complementary fashion. Example of this kind of command is "Go there": saying gothere and pointing to certain direction. Possible improvements would be to include some other similar commands.

Note: We assume here that speech command is given before gesture command. This prerequisite was necessary to avoid synchronization conflict between speech and gesture commands.

Logic of complementary integrator is as follows:

  1. Speech command "Go" is received.
  2. Speech thread wait maximum for 4 seconds for corresponding gesture command to arrive.
  3. While the speech thread is waiting, if the gesture command arrives, then the complementary integrator sends two commands:
    • Rotate in the direction of gesture.
    • Move forward in that direction.

Note: The complementary integrator is called only when "Go" command is recognized by speech module otherwise primary integrator is called.

Powered by Google Project Hosting