Sparsh Developer’s Guide

---

Version	Author	Date	Comments
Initial Draft	Prasad RS(prasadrs@iastate.edu)	10/02/08

1. Introduction:

This document gives a detailed description of the Sparsh UI architecture, the various modules and the protocol used between the clients and the Sparsh gesture Server. The target audience of this document is anybody interested in the development of the Sparsh UI architecture, Multi-touch application developers and enthusiasts.

2. Sparsh UI high level architecture

This section gives a high level overview of the Sparsh UI architecture. It briefly discusses the various modules involved in Sprash UI. We will later discuss in depth about each module.

Fig 1. High level architecture diagram of Sparsh UI.

Figure 1 shows the high level architecture diagram of Sparsh UI. A high level breakup of the multi-touch framework consists of the following components:

1)Input Device

Input device refers to the multi-touch device which is being used to obtain the input from the user. Sparsh UI is hardware agnostic in the sense it is compatible to any multi-touch hardware as long it follows the protocol specified by Sparsh UI.The protocol is discussed later.

2)Device Driver

The device driver is the software driver that interacts with the multi-touch hardware and provides the multi-touch events.Since this is dependent on the hardware that is being used it needs to be written for the specific multi-touch hardware one is using. We have provided the hardware for the hardware devices that we are currently using. (FTIR based multi-touch table , Infra-Red bezel ,Stantum multi-touch screen).

3)Sparsh Device Driver Adapter

To make the sparsh gestur recognition framework hardware agnostic , there is a need to standardize the input data format from the device driver. The native device driver may not comply with the format that Sparsh UI requires hence there is a need to write an adapter that sits on top of the device driver and supplies the data in the format required by sparsh UI.

4)Gesture Recognition Framework

This is the core engine of the Sparsh UI that takes in the input device data and analyses the gestures from the touch point co-oridinates. Multiple clients can register to the gesture recognition framework and obtain the gesture events for which the client registers.This module will be discussed in depth in later sections.

5)Client Adapter

This is the adapter that the client needs to implement to receive Sparsh UI events. The Client to server protcol adapter is provided as a part of Sparsh UI.The client application needs to implement an interface(abstract class) to receive the events. The handling of various events is done in this Client adapter.

6)Client

This is the Multi-touch application itself. The client listens on to the various multi-touch events(gesture) and acts accordingly.

In the subsequent sections we go over the details of each module in depth.

Protocols Used in Sparsh UI

Note: All the data sent over sockets is in Network Endian Order

1)Input Device Protocol

This is the protocol between the Input device and the Sparsh UI gesture recognition framework.The Sparsh Adapter (described earlier) is responsible for providing the information in this format.

Touch Point States

In multi-touch systems a touch point is normally associated with three states namely:

1. Touch Point birth (or Touch Point down) When place your finger on the multi-touch device a touch point is created. A unique id is associated with the touch point on its creation.
2. Touch Point move (or Touch point update) After the creation of touch point as you move your finger along the multi-touch surface , touch point state changes from birth to move and the touch point co-ordinates are constantly updated. #
3. Touch Point Death (Touch point UP) - Once you lift the finger from the multi-touch surface the touch point ceases to exist. This state is referred to as Touch Point Death or Touch Point UP.

In addition to this , each point is associated with an unique ID,which allows us to identify the touch point and its states.

Touch Point data stucture

The touch point data structure consists of the following fields in the given order:

1. touch point id ( 4 byte integer)
2. X co-ordinate ( 4 byte float )
3. Y co-ordinate ( 4 bytes float)
4. touch point state ( 1 byte value)

The touch point state is an enumerator with the following fields

enum Touch_Point_State

   POINT_BIRTH,
   POINT_DEATH,
   POINT_MOVE

end enum

The Input device driver and the Gesture framework communicate over a TCP / IP socket. The Gesture framework runs on port number 5945.The input device on connection sends a byte with value set to 1 indicating that it is an input device. After this the input device can start sending the data. The payload format is given below.

Number of Touch points ( byte integer )

Pay load ( Number of Touch Points X Touch_Point data structure )

2) Server to Client Protocol

The Server to client protocol consists of the various messages sent by the server to the client. The protocol consists of exchange of messages for the following :

1. Obtaining the group ID : This is the message sent by the server to the client to obtain the group ID information from the client , given the location information.

1 byte value	4 byte float	4 byte float
GET_GROUP_ID	X Co-ordinate	Y Co-ordinate

The first field is a single byte which is a member of enum MessageType. The second and third field are X and Y Co-ordinates respectively which represent the location of the touch-point screen. These two fields are 4 byte float values.

2. Obtaining the allowed gestures. This message is sent by the server to the client after obtaining a valid group ID.The server uses this information to process only the required gestures for that particular group ID.The message consists of the following fields shown below:

1 byte value	4 byte integer
GET_ALLOWED_GESTURES	Group ID

The first field is a single byte which is a member of enum MessageType indicates the type of the message. The second field is the 4 byte integer group Id for which the gestures are being queried for.

3. Gesture Events This message is sent by the server on the occurance of any gesture events. The message consists of the following fields

4 byte integer	4 byte integer	Gesture specific payload
Message Length	Group ID	Gesture speccific Information

The gesture specific fields are as follows:

Touch Event

4 bytes	4 bytes	4 bytes	4 bytes	4 bytes
Event Type	id	x co-ord.	y co-ord.	Touch State

Drag Event

4 bytes	4 bytes	4 bytes
Event Type	X direction displacement	Y direction displacement

Rotate Event

4 bytes	4 bytes	4 bytes	4 bytes
Event Type	rotation	x co-ord. of center	y co-ord. of center

Zoom Event

4 bytes	4 bytes	4 bytes	4 bytes
Event Type	scale	x co-ord. of center	y co-ord. of center

Spin Event

4 bytes	4 bytes	4 bytes	4 bytes
Event Type	rotation about X -axis	rotation about X -axis	rotation about Z -axis

3) Client to Server Protocol

The client to server protocol consists of the replies sent by the client to the server for the various queries that the server makes.

1) Group ID This message is sent in response to the Get Group ID message from the server. The message just consists of a single integer.

4 byte integer

GET GROUP ID

1) Allowed Gestures This message is sent in response to the Get Allowed Gestures message from the server. The message consists of the following fields:

4 byte integer	4 byte integer	4 byte integer
Message Length	Gesture1 Identifier	Gesture 2 identifier

The first field is the length of the message indicating how many bytes need to be read. The second field is the Gesture identifier. The number of gesture identifiers would be MessageLength / 4.

The Gesture identifiers are specified in the enumerator

enum GestureType {
	DRAG_GESTURE,
	MULTI_POINT_DRAG_GESTURE,
	ROTATE_GESTURE,
	SPIN_GESTURE,
	TOUCH_GESTURE,
	ZOOM_GESTURE
}

Based on the above discussion , the next diagram shows the sequence of exchange of messages between the client and the Sparsh Gesture server.