|
Installation
SSS Mapreduce Overview
ja , en DownloadDownload the SSS Mapreduce package from the download Web page as below. Download the packages of Tokyo Cabinet and Tokyo Tyrant from the following URLs.
InstallationExpanding the source filesMove to the directory which placed the downloaded package and then use the following command to expand the packages. (The sss-CCYYMMDD.tgz in the command is a package name.) $ tar xfvz sss-CCYYMMDD.tgz $ tar xfvz tokyocabinet-1.4.47.tar.gz $ tar xfvz tokyotyrant-1.1.41.tar.gz $ tar xfvz jtokyotyrant-0.1.1.tar.gz Executing the above command will create a "sss-CCYYMMDD", "tokyocabinet-1.4.47", "tokyotyrant-1.1.41" and "jtokyotyrant-0.1.1" directories, and expand the those source files in those directories. Note: Use the GNU tar to expand the package. Apply the patch, running the configure and makeMove to the directory in which the source files of Tokyo Cabinet are expanded. $ cd tokyocabinet-1.4.47 $ cat ../sss-CCYYMMDD/patches/tokyocabinet* | patch -p1 $ ./configure && make && sudo make install Move to the directory in which the source files of Tokyo Tyrant are expanded. $ cd ../tokyotyrant-1.1.41 $ cat ../sss-CCYYMMDD/patches/tokyotyrant* | patch -p1 $ ./configure && make && sudo make install Move to the directory in which the source files of jtokyotyrant are expanded. $ cd ../jtokyotyrant-0.1.1 $ make Add jtokyotyrant.jar to your CLASSPATH. Add libjtokyotyrant.so to your LD_LIBRARY_PATH. Example as sh: $ export CLASSPATH=/DirPath/jtokyotyrant-0.1.1/jtokyotyrant.jar $ export LD_LIBRARY_PATH=/DirPath/jtokyotyrant-0.1.1 Example as csh: % setenv CLASSPATH /DirPath/jtokyotyrant-0.1.1/jtokyotyrant.jar % setenv LD_LIBRARY_PATH /DirPath/jtokyotyrant-0.1.1 NOTE: The license of these patches is LGPLv2. See also sss-CCYYMMDD/patches/COPYING. Running the Apache AntWARNING: If msgpack-java 0.6.8 already has been released, since 'ant' downloads automatically, you need not read this. SSS Mapreduce can not run normaly with usgpack-java 0.6.7 or earlier. But msgpack-java 0.6.8 has not been released yet as of Febrary 2013. Thus it is necessary to get msgpack-java latest sorce code from the following GIT repogitory and register the package to maven2 local repogitory. This work require the followings.
The command to get msgpack-java sorce code and register the package:$ git clone https://github.com/msgpack/msgpack-java.git $ cd msgpack-java $ mvn install Move to the directory in which the source files of SSS Mapreduce are expanded. $ cd ../sss-CCYYMMDD $ ant Note: Our build.xml is used ivy http://ant.apache.org/ivy/. If your computer is inside proxy server, then set the ANT_OPTS environment variable collect. As sh: $ export ANT_OPTS="-Dhttp.proxyHost=YourProxyServer \ -Dhttp.proxyPort=YourProxyPortNumber" As csh: % setenv ANT_OPTS "-Dhttp.proxyHost=YourProxyServer \ -Dhttp.proxyPort=YourProxyPortNumber" Setting upCreate two configuration filesMove to the directory in which the source files of SSS Mapreduce are expanded. Move to the conf directory. Create a file that describes the host name of your computers by file named "peers". Run genconf script and please answer four questions. $ cd /DirPath/sss-CCYYMMDD/conf $ cat > peers node1.example.org node2.example.org ^D $ ./genconf step1. select a file which contains your cluster node list: > peers step2. enter a filename which will be written as output: > cluster.conf step3. select which protocol you use: > aist sss step4. number of storage engine per node: > 1 If you create a configuration file by hand, please make it a file name called "cluster.conf". Please refer to a cluster.conf.16node. Set a environment variableSet a environment variable "MAPREDUCE_HOME" to directory in which the source files of SSS Mapreduce are expanded. As sh: $ export MAPREDUCE_HOME=/DirPath/sss-CCYYMMDD As csh: % setenv MAPREDUCE_HOME /DirPath/sss-CCYYMMDD Create directoriesCreate three directories db, logs and pids on remote machines $MAPREDUCE_HOME. $ ssh node1.example.org mkdir $MAPREDUCE_HOME/{logs,db,pids}
$ ssh node2.example.org mkdir $MAPREDUCE_HOME/{logs,db,pids}Start/Stop serversStart TokyoTyrant and SSSserverRun bin/clusterctl.sh script for start TokyoTyrant and SSSserver. $ bin/clusterctl.sh start Stop TokyoTyrant and SSSserver$ bin/clusterctl.sh stop Note: User must be able to execute ssh command for invoke remote job. In addition, it is recommended to configure user's ssh environments not to require user's input (e.g. password) for executions to avoid repetitious input while SSS Mapreduce application is executed. "ssh-agent" and "ssh-add" commands are usually used for such purposes. The TokyoTyrant and SSSserver were created log file under $MAPREDUCE_HOME/logs directory. Testing (optional)You can run the testsuite (which may take a short time). $ ant wordcount This command execute application WordCount. | |