|
|
UpAndRunningWithHadoop
Instructions on how to get Hypertable up and running with Hadoop
Starting the Servers
Step 1
Install and start Hadoop.
Step 2
Make sure you have extended attributes enabled on the partition that holds the root of the Hyperspace directory tree. This will be the same disk volume that holds the installation (e.g. ~/hypertable), unless you configured Hyperspace explicitly to point to somewhere else. This is not necessary on the mac, but is in general on Linux systems. To enable extended attributes on Linux, you need to add the user_xattr property to the relevant file systems in your /etc/fstab file. For example:
/dev/hda3 /home ext3 defaults,user_xattr 1 2
You can then remount the affected partitions as follows:
$ mount -o remount /home
Step 3
Create the directory /hypertable in HDFS and make it writeable by all.
$ hadoop fs -mkdir /hypertable $ hadoop fs -chmod 777 /hypertable
Step 4
Edit the config file conf/hypertable.cfg Change the following property to point to the Hadoop filesystem that got up and running in step 1 (assuming hdfs://motherlode000:9000):
HdfsBroker.fs.default.name=hdfs://motherlode001:9000
Change the following two properties to point to the location of the Hypertable Master and Hyperspace (assuming motherlode001):
Hyperspace.Master.Host=motherlode001 Hypertable.Master.Host=motherlode001
Step 5
Start the Master and Hyperspace
ssh motherlode001 ~/hypertable/bin/start-master.sh hadoop
Step 6
Start the Range Servers. Add all of the names of the machines that will be running range servers to the file conf/slaves relative to the installation directory. Here's what an example one looks like:
$ cat conf/slaves motherlode001 motherlode002 motherlode003 motherlode004 motherlode005 motherlode006 motherlode007 motherlode008
This slaves file, along with the entire installation should be pushed out to all of the machines using something like rsync. Once you've done this, you can launch all of the range servers with the following command (assuming installation dir is ~/hypertable on all of the machines):
./bin/slaves.sh ~/hypertable/bin/start-range-server.sh hadoop
Now you sould be able to run the ~/hypertable/bin/hypertable HQL command interpreter and start playing around.
Stopping the System
Execute the following commands to stop all of the servers:
ssh motherlode001 ~/hypertable/bin/kill-servers.sh ./bin/slaves.sh ~/hypertable/bin/kill-servers.sh
Sign in to add a comment
how to for windows?
[cmolina@dungun hypertable]$ bin/start-master.sh hadoop DfsBroker? (hadoop) hasn't come up yet, trying again in 5 seconds ... Num CPUs=2 HdfsBroker?.Port=38030 HdfsBroker?.Reactors=2 HdfsBroker?.Workers=20 HdfsBroker?.Server.fs.default.name=hdfs://localhost:9000 org.apache.hadoop.ipc.RPC$VersionMismatch?: Protocol org.apache.hadoop.dfs.ClientProtocol? version mismatch. (client = 14, server = 20)
Problem statring DfsBroker? (hadoop)Does this means I have a hadoop release that's too new? Could you add to this wiki the version of hadoop that the current hypertable release supports?.
Thanks a lot, Can't wait to get this thing up and running to migrate all our "vintage" systems.
Found the fix. Replace your hypertable_installation_dir/lib/java/hadoop-core-x-x-x.jar file with the one that comes in your hadoop_installation_dir/hadoop-core-x-x-x.jar.
Is it necessary to turn on extended attributes on all nodes?
@chrulle, Only the node running hyperspace needs it.