My favorites | Sign in
Project Home Downloads Wiki Issues Source
READ-ONLY: This project has been archived. For more information see this post.
Search
for
Installation  
SSS Mapreduce Overview
ja , en
Updated Feb 27, 2013 by tatsuhik...@gmail.com

Download

Download the SSS Mapreduce package from the download Web page as below.

Download the packages of Tokyo Cabinet and Tokyo Tyrant from the following URLs.

Installation

Expanding the source files

Move to the directory which placed the downloaded package and then use the following command to expand the packages. (The sss-CCYYMMDD.tgz in the command is a package name.)

$ tar xfvz sss-CCYYMMDD.tgz
$ tar xfvz tokyocabinet-1.4.47.tar.gz
$ tar xfvz tokyotyrant-1.1.41.tar.gz
$ tar xfvz jtokyotyrant-0.1.1.tar.gz

Executing the above command will create a "sss-CCYYMMDD", "tokyocabinet-1.4.47", "tokyotyrant-1.1.41" and "jtokyotyrant-0.1.1" directories, and expand the those source files in those directories.

Note: Use the GNU tar to expand the package.

Apply the patch, running the configure and make

Move to the directory in which the source files of Tokyo Cabinet are expanded.

$ cd tokyocabinet-1.4.47
$ cat ../sss-CCYYMMDD/patches/tokyocabinet* | patch -p1
$ ./configure && make && sudo make install

Move to the directory in which the source files of Tokyo Tyrant are expanded.

$ cd ../tokyotyrant-1.1.41
$ cat ../sss-CCYYMMDD/patches/tokyotyrant* | patch -p1
$ ./configure && make && sudo make install

Move to the directory in which the source files of jtokyotyrant are expanded.

$ cd ../jtokyotyrant-0.1.1
$ make

Add jtokyotyrant.jar to your CLASSPATH. Add libjtokyotyrant.so to your LD_LIBRARY_PATH.

Example as sh:

$ export CLASSPATH=/DirPath/jtokyotyrant-0.1.1/jtokyotyrant.jar
$ export LD_LIBRARY_PATH=/DirPath/jtokyotyrant-0.1.1

Example as csh:

% setenv CLASSPATH /DirPath/jtokyotyrant-0.1.1/jtokyotyrant.jar
% setenv LD_LIBRARY_PATH /DirPath/jtokyotyrant-0.1.1

NOTE: The license of these patches is LGPLv2. See also sss-CCYYMMDD/patches/COPYING.

Running the Apache Ant

WARNING:

If msgpack-java 0.6.8 already has been released, since 'ant' downloads automatically, you need not read this.
SSS Mapreduce can not run normaly with usgpack-java 0.6.7 or earlier. But msgpack-java 0.6.8 has not been released yet as of Febrary 2013.
Thus it is necessary to get msgpack-java latest sorce code from the following GIT repogitory and register the package to maven2 local repogitory.
This work require the followings.
  • git
  • maven2
The command to get msgpack-java sorce code and register the package:
$ git clone https://github.com/msgpack/msgpack-java.git
$ cd msgpack-java
$ mvn install

Move to the directory in which the source files of SSS Mapreduce are expanded.

$ cd ../sss-CCYYMMDD
$ ant

Note: Our build.xml is used ivy http://ant.apache.org/ivy/. If your computer is inside proxy server, then set the ANT_OPTS environment variable collect.

As sh:

$ export ANT_OPTS="-Dhttp.proxyHost=YourProxyServer \
  -Dhttp.proxyPort=YourProxyPortNumber"

As csh:

% setenv ANT_OPTS "-Dhttp.proxyHost=YourProxyServer \
  -Dhttp.proxyPort=YourProxyPortNumber"

Setting up

Create two configuration files

Move to the directory in which the source files of SSS Mapreduce are expanded. Move to the conf directory. Create a file that describes the host name of your computers by file named "peers". Run genconf script and please answer four questions.

$ cd /DirPath/sss-CCYYMMDD/conf
$ cat > peers
node1.example.org
node2.example.org
^D
$ ./genconf
step1. select a file which contains your cluster node list:
> peers
step2. enter a filename which will be written as output:
> cluster.conf
step3. select which protocol you use:
> aist sss
step4. number of storage engine per node:
> 1

If you create a configuration file by hand, please make it a file name called "cluster.conf". Please refer to a cluster.conf.16node.

Set a environment variable

Set a environment variable "MAPREDUCE_HOME" to directory in which the source files of SSS Mapreduce are expanded.

As sh:

$ export MAPREDUCE_HOME=/DirPath/sss-CCYYMMDD

As csh:

% setenv MAPREDUCE_HOME /DirPath/sss-CCYYMMDD

Create directories

Create three directories db, logs and pids on remote machines $MAPREDUCE_HOME.

$ ssh node1.example.org mkdir $MAPREDUCE_HOME/{logs,db,pids}
$ ssh node2.example.org mkdir $MAPREDUCE_HOME/{logs,db,pids}

Start/Stop servers

Start TokyoTyrant and SSSserver

Run bin/clusterctl.sh script for start TokyoTyrant and SSSserver.

$ bin/clusterctl.sh start

Stop TokyoTyrant and SSSserver

$ bin/clusterctl.sh stop

Note: User must be able to execute ssh command for invoke remote job. In addition, it is recommended to configure user's ssh environments not to require user's input (e.g. password) for executions to avoid repetitious input while SSS Mapreduce application is executed. "ssh-agent" and "ssh-add" commands are usually used for such purposes.

The TokyoTyrant and SSSserver were created log file under $MAPREDUCE_HOME/logs directory.

Testing (optional)

You can run the testsuite (which may take a short time).

$ ant wordcount

This command execute application WordCount.

Powered by Google Project Hosting