MediaDART Installation Guide

For each mediadart node, add the following lines to your /etc/apt/sources.list:

        deb http://www.notredam.org/stage-repository/lucid/  binary-i386/
        deb http://www.notredam.org/stage-repository/lucid/  binary-amd64/
 

Install the mediadart-mq package on each node hosting mediadart services, and change the default configuration on each node to match your installation:

        sudo apt-get update
        sudo apt-get install mediadart-mq
 

Mediadart is now installed and ready to go on a single node.

Configuring mediadart

Mediadart uses ConfigParser files to store its configuration.

The file mediadart.cfg contains a commented example of the configuration required by Mediadart.

Configuration is looked up in this order:

1. /etc/mediadart/mediadart.cfg
2. Other .cfg files in /etc/mediadart, read in alphabetic order.
3. $HOME/.mediadart/mediadart.cfg
4. Other .cfg files in $HOME/.mediadart, read in alphabetic order.

Files read later overwrite configuration options set in files read earlier.

The typical use is to put all definitions common to all mediadart nodes for example the address of the broker, (address of the broker) The additional files in $HOME/.mediadart are used to define options that are specific to user-developed servers, and that are not part of the mediadart package and that are not versioned in the mediadart repository.

The service classes

In the Mediadart framework, a server is a python class which extends MQServer, as described in the wiki page MessageQueues.

Services are started by the script launch.py. The list of the services to start is given in the section MQUEUE, by the concatenation of all options that start with "server_". So in order to start a new service, a configuration file must be created in one of the directories searched by Mediadart with a section MQUEUE and an option specifying a callable that can be used to start the server. Look at the existing src/config/mediadart.cfg in the source tree for examples.

All services are configured by setting options in the section of the service. The name of the section is the name of the class extending MQServer, uppercase.

The one item of configuration that it is important to specify is "concurrency_level". As explained in http://code.google.com/p/mediadart/wiki/ConcurrencyControl, this options determines the maximun number of requests that a service starts simultaneously. The default value is 10, which is too high for any service that starts external programs like ffmpeg etc.

Selecting and starting the message broker

Service classes export an RPC APIs implemented using the message queue provided by RabbitMQ (www.rabbitmq.org), a message broker that implements the AMQP protocol (see MediadartMessageQueues). A mediadart system should use the instance of the broker installed in the machine with the highest reliability.

In fact, the version of Mediadart available at the moment is geared toward performance, and uses non-persistent message queues, which means that in the event of broker failure, non-delivered messages are lost.

On ubuntu the broker is started automatically when you install mediadart-mq, so there is no need to do anything besides configuring each node with these four options:

• IP address and port
• User and password

All nodes in a mediadart system must point to the same broker, and all instances of RabbitMQ different from the chosen broker can be turned off. User and password are used for security and for defining independent contexts when a single broker is used by more than one mediadart system.

Starting and stopping a node

Assuming that the root of the installaion is /opt/mediadart, a mediadart node is started as follows:

 export PYTHONPATH=/opt/mediadart
 cd /opt/mediadart/mediadart
 python launch.py
 

To stop a node, simply kill launch.py.

Checking the installation

Every service class implements a "ping" method. For example to make a quick check is the service Adapter is up and running you can execute the following script:

 from mediadart.mqueue.mqclient_async import Proxy
 p = Proxy('Adapter')   # assuming you started the adapter
 p.ping()
 

Adding nodes to an existing installation

New instances of service classes can be made available by adding new nodes to and existing Mediadart network. Each new node can export all or a part of the available services, allowing different configurations that can adapt to varying service loads.

By installing the mediadart-mq package, a RabbitMQ instance in automatically started on each node, but all nodes must use the a single instance. This is achieved by setting the configuration options broker_ip, broker_port, username, password and virtual_host to the same value in each node of the network. Unused rabbitmq instances can be stopped using the command line tool rabbitmqctl, but this is not required as an idle rabbitmq instance does not consume resources.

So adding a new node requires three steps:

Install the mediadart-mq package as described above

Modify the section MQUEUE in the configuration so that the new node uses the chosen rabbitmq instance for the whole network

Choose the servers to start on the new node by commenting/uncommenting the options in the section MQUEUE that start with server_ .

Shared file system

The service classes FeatureExtractor and Adapter that come bundled with the Mediadart framework exchange data with their clients using shared filesystems. The service classes do not make any assumption on the file system used. You can use NFS, Samba or any other distributed filesystem. For example, instructions on how to mount a NFS share on a Unix machine can be found here.

All shared directories used by a mediadart node must be under a common root. The configuration has the option "cache_dir" for specifying the root of the tree of the shared directories

As the shared directories can have different mount points on different machines, all mediadart services should accept as argoments only paths relative to the root of the shared filesystem.

Beside the shared filesystem, a mediadart node uses a local filesystem for storing temporary files. The configuration option "temp_dir" specifies the absolute path to the temporary area.