Machines
The cluster is comprised of 3 classes of machine:
| Type | CPU | Memory | Disk | No. Machines in Cluster |
| C1 | Intel(R) Xeon(R) CPU X3363 @ 2.83GHz, 2x6MB (quad) | 8GB | 2x500G SATA 5.4K | 10 |
| C2 | 2x Intel(R) Xeon(R) CPU E5630 @ 2.53GHz (quad) | 24GB | 6x250G SATA 7.2K | 3 |
| C3 | Intel(R) Xeon(R) CPU X3220 @ 2.40GHz (quad) | 4GB | 2x160G SATA 7.2K | 5 |
The configuration:
| Type | Daemons |
| C1 (1 master) | JobTracker, ZooKeeper, NameNode, HBaseMaster, PuppetMaster, Hue Master, Oozie |
| C1 (9 slaves) | DataNode, TaskTracker, RegionServer, PuppetClient |
| C2 | DataNode, TaskTracker, PuppetClient |
| C3 | DataNode, TaskTracker, PuppetClient |
Tuning M+R count based on memory
Map reduce can be limited by CPU, disk and network. However, the machines have a fixed amount of memory, and therefore there is a theoretical maximum allowed to ensure no instability due to memory issues are seen. These are given for a variety of scenarios below, but in operation, one would anticipate running below this limit after spotting IO wait, CPU load, or network issues.
#M and #R versus memory (Assuming DN, TT and OS take 1GB each, and remember -Xmx includes the io.sort.mb, no RS running)
| Type | Available memory | -Xmx500M | -Xmx768M | -Xmx1G |
| C1 | 5GB | Total M+R = 10 | Total M+R = 6 | Total M+R = 5 |
| C2 | 21GB | Total M+R = 42 | Total M+R = 27 | Total M+R = 21 |
| C3 | 1GB | Total M+R = 2 | Total M+R = 1 | Total M+R = 1 |
| | Total across cluster: | 246 | 140 | 113 |
#M and #R versus memory when RS is running on C2 (Assuming DN, TT and OS take 1GB each, and remember -Xmx includes the io.sort.mb, RS running with 6G heap)
| Type | Available memory | -Xmx500M | -Xmx768M | -Xmx1G |
| C1 | 5GB | Total M+R = 10 | Total M+R = 6 | Total M+R = 5 |
| C2 | 15GB | Total M+R = 30 | Total M+R = 19 | Total M+R = 15 |
| C3 | 1GB | Total M+R = 2 | Total M+R = 1 | Total M+R = 1 |
| | Total across cluster: | 190 | 118 | 95 |
#M and #R versus memory when RS is running on C1 (Assuming DN, TT and OS take 1GB each, and remember -Xmx includes the io.sort.mb, RS running with 4G heap)
| Type | Available memory | -Xmx500M | -Xmx768M | -Xmx1G |
| C1 | 1GB | Total M+R = 2 | Total M+R = 1 | Total M+R = 1 |
| C2 | 21GB | Total M+R = 42 | Total M+R = 27 | Total M+R = 21 |
| C3 | 1GB | Total M+R = 2 | Total M+R = 1 | Total M+R = 1 |
| | Total across cluster: | 154 | 95 | 77 |