In this section we propose two way to fuse commands coming from speech and gesture. The two fusion strategies are as follows:
- Primary Integrator : This integrator makes command recognition more robust. It cross verify the commands from both modules.
- Complementary Integrator : Here one module acts as a complementary action to the other.
Pairing between speech and gesture commands:
| Speech | Gesture |
| Forward | Forward |
| Backward | Backward |
| Left | Left |
| Right | Right |
| Slower | Slower |
| Faster | Faster |
| Stop | Stop |
Primary Integrator
Purpose of this integrator is to fuse information coming from two modules. Gesture module gives distribution: confidences for all commands present in database. However, speech module gives only "Winner" (command with highest confidence) and it's confidence. So we take the winning command from both modules and checks them.
In case when the second command doesn't arrive in three seconds, we execute the available command if it confidence is greater than threshold (0.9). Logic of integrator is as follows:
if (gesture.command == speech.command)
if (gesture.confidence > gesture.lowThreshold || speech.confidence>speech.lowThreshold)
execute command // it is enough, since they are same
else
say "Again Please" // robot is not certain about either modality
else // commands are different
if (gesture.confidence > gesture.highThreshold && speech.confidence < speech.lowThreshold)
execute command // gesture is very certain, speech is not certain
else if (speech.confidence > speech.highThreshold && gesture.confidence < gesture.lowThreshold)
execute command // speech is very certain, gesture is not certain
else if (speech.confidence>speech.lowThreshold && gesture.confidence > gesture.lowThreshold)
say "Again Please" //both confidences are high. Cannot decide.Complementary Integrator
It is meant to integrate speech and gesture commands in a complementary fashion. Example of this kind of command is "Go there": saying gothere and pointing to certain direction. Possible improvements would be to include some other similar commands.
Note: We assume here that speech command is given before gesture command. This prerequisite was necessary to avoid synchronization conflict between speech and gesture commands.
Logic of complementary integrator is as follows:
- Speech command "Go" is received.
- Speech thread wait maximum for 4 seconds for corresponding gesture command to arrive.
- While the speech thread is waiting, if the gesture command arrives, then the complementary integrator sends two commands:
- Rotate in the direction of gesture.
- Move forward in that direction.
Note: The complementary integrator is called only when "Go" command is recognized by speech module otherwise primary integrator is called.