hadoop+hbase集群搭建 詳細手冊V0.9

Hadoop，ZooKeeper，HBase分布式搭建前提準備注：hbase的安裝需要hadoop和zookeeper和hbase，生產環(huán)境下需要將zookeeper獨立安裝，hbase軟件選擇cloudera的cdh3u0，這樣不會出現版本不兼容等問題。apache的版本需要重新編譯hadoop0.20.2-appender版以保證沒有數據丟失。1、機器集群結構分布使用8臺曙光服務器搭建集群，ip為*.*.83.1-8，hostname為hadoop-node1到hadoop-node8，其中機器名IP所安裝的程序hadoop-node1 hadoopnamenode/hbaseHMasterhadoop-node2*.*.83.2 hadoopdatanode/hbaseHRegionServerhadoop-node3 *.*.83.3 hadoopdatanode/hbaseHRegionServerhadoop-node4 *.*.83.4hadoopdatanode/hbaseHRegionServerhadoop-node5 *.*.83.5hadoopdatanode/hbaseHRegionServerhadoop-node6 *.*.83.6hadoopdatanode HRegionServer zookeeperhadoop-node7 *.*.83.7hadoopdatanode HRegionServer zookeeperhadoop-node8 *.*.83.8hadoopsecondNamenode/hbaseHMasterzookeeper2.硬件環(huán)境：8臺曙光服務器，cpu8核，每臺機器都有兩個500GB硬盤。操作系統(tǒng)：SUSEenterprise11操作系統(tǒng)分區(qū)：硬盤1：sda/boot400MB/Swap10GB/余下的sda空間硬盤2：/sdb2都掛載在/sdb2上3.提前準備相關軟件Jdk： jdk-6u30-linux-x64.binHadoop： hadoop-0.20.2-cdh-3u**Zookeeper： zookeeper-3.3.5-cdh3u**Hbase： hbase-0.906-c*******詳細操作：當前用戶hadoop/home/hadoop/app/app/hadoop/hadoop1.0.0//app/hbase/hbase0.92.0//app/zookeeper/zoookeeper3.3.4/Step1安裝SUSE及搭建基本環(huán)境操作系統(tǒng)具體安裝可以隨意，盡量保證每臺機器一致，如每臺的機器名，和ip配置都要符合上面所規(guī)劃，這樣方便之后的操作在所有機器上，創(chuàng)建hadoop用戶保證每臺suse服務器上的ssh服務都已經啟動在每臺機器上/home/hadoop/下新建如下一些目錄/program/java;/app/hadoop;/app/hbase;/app/zookeeper;/app/zookeeperdata將相應的軟件上傳到實體機，在hadoop-node1上解壓hadoop和hbase#tarzxvfhadoop-0.20.2-cdh3u0.tar.gz#tarzxvfhbase-0.90.1-cdh3u0.tar.gzhadoop-node6在上解壓zookeeper#tarzxvfzookeeper-3.3.3-cdh3u0.tar.gz在節(jié)點1-5,8上修改/etc/profileexportJAVA_HOME=/home/hadoop/program/java/jdk1.6.0_30exportPATH=$JAVA_HOME/bin:$PATHexportCLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$CLASSPATHexportHADOOP_HOME=/home/hadoop/app/hadoop/hadoop-0.20.2-cdh3u4exportHBASE_HOME=/home/hadoop/app/hbase/hbase-0.90.6-cdh3u4在節(jié)點6,7上修改/etc/profileexportJAVA_HOME=/home/hadoop/program/java/jdk1.6.0_30exportOOKEEPER_HOME=/home/hadoop/app/zookeeper/zookeeper-3.3.5-cdh3u4exportPATH=$JAVA_HOME/bin:$ZOOKEEPER_HOME/conf:$PATHexportCLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$CLASSPATHssh免密碼登錄用hadoop用戶登錄所有機器，在/home/hadoop/下建立.ssh目錄；運行#ssh-keygen-tdsa-P''-f~/.ssh/id_dsa這樣會在~/.ssh/生成兩個文件：id_dsa和id_dsa.pub。#cat~/.ssh/id_dsa.pub>>~/.ssh/authorized_keys拷貝authorized_keys到2到8#scp/home/hadoop/.ssh/authorized_keyshadoop@*.*.83.2:/home/hadoop/.ssh/……#scp/home/hadoop/.ssh/authorized_keyshadoop@*.*.83.8:/home/hadoop/.ssh/修改hosts文件,每一臺實體機配置都如下#sudovim/etc/hosts從1登錄1,2…8用hostname（hadoop-nodeX），第一次需要確認，以后就可以直接登錄。#sshhadoop-node1#sshhadoop-node2#sshhadoop-node3#sshhadoop-node4#sshhadoop-node5#sshhadoop-node6#sshhadoop-node7#sshhadoop-node8部分操作截圖：1．上傳jdk最好是使用64系統(tǒng)2．上傳hadoop，hbase，zookeeperStep2安裝JDK#cdjava#chmod+xjdk-6u30-linux-x64.bin./jdk-6u24-linux-x64.binHadoop安裝1．在節(jié)點1和節(jié)點8上分別都創(chuàng)建/datamkdir/home/hadoop/data2.在節(jié)點2和5上分別都創(chuàng)建/disk1,/disk2,/disk3即mkdir/home/hadoop/disk1disk2disk33.vim/home/hadoop/app/hadoop/hadoop-0.20.2-cdh3u0/conf/hadoop-env.sh4.修改/home/hadoop/app/hadoop/hadoop-0.20.2-cdh3u0/conf/core-site.xml添加,截圖如下：5.修改/home/hadoop/app/hadoop/hadoop-0.20.2-cdh3u0/conf/hdfs-site.xml截圖：6.vihome/hadoop/app/hadoop/hadoop-0.20.2-cdh3u0/conf/mapred-site.xml截圖：7.修改/home/hadoop/app/hadoop/hadoop-0.20.2-cdh3u0/conf/masters截圖：8.修改/home/hadoop/app/hadoop/hadoop-0.20.2-cdh3u0/conf/slaves截圖：9.很關鍵的一步：拷貝節(jié)點1的hadoop到2-5,8#scp–r./hadoop-0.20.2-c**/hadoop@*.*.*.2:/home/hadoop/app/hadoop/…#scp–r./hadoop-0.20.2-c**/hadoop@*.*.*.5:/home/hadoop/app/hadoop/…10.格式化hadoop文件系統(tǒng)#hadoopnamenode–format11.啟動hadoop，在1上運行#start-all.sh注：運行jps查看各個節(jié)點運行情況截圖：節(jié)點1(主節(jié)點)：節(jié)點2或3、4、5(數據節(jié)點)：節(jié)點6或7(Zookeeper節(jié)點)：節(jié)點8(secondName和HbaseMaster節(jié)點)：查看集群狀態(tài):http://*.*.83.1:50070/dfshealth.jsp查看集群狀態(tài):http://*.*.83.1:50030/dfshealth.jspZookeeper安裝1.在節(jié)點6,7上創(chuàng)建/home/hadoop/zookeeperdata目錄修改6的/home/hadoop/app/zookeeper/zookeeper-3.3.3-cdh3u0/conf/zoo.cfg2.拷貝6的zookeeper到7#scp–r./zookeeper-3.3.3-*/hadoop@*.*.*.7:/home/hadoop/app/zookeeper/3.在6,7的/home/hadoop/zookeeperdata目錄下建myid文件，內容分別為1,2,3啟動zookeeper，4.在6,7上分別執(zhí)行#zkServer.shstart5.啟動后可以使用#zkServer.shstatus查看狀態(tài)Hbae安裝在節(jié)點1上修改/home/hadoop/app/hbae/hbase-0.90.1-cdh3u0/conf/hbase-env.shexportJAVA_HOME=/home/hadoop/program/java/jdk1.6.0_30exportHBASE_MANAGES_ZK=false2.在節(jié)點1上修改/home/hadoop/app/hbase/hbase-0.90.1-cdh3u0/conf/hbase-site.xml<configuration><property><name>hbase.rootdir</name><value>hdfs://hadoop-node1:9000/hbase</value></property><property><name>hbase.cluster.distributed</name><value>true</value></property><property><name>hbase.master.port</name><value>60000</value></property><property><name>hbase.zookeeper.quorum</name><value>hadoop-node6,hadoop-node7</value></property></configuration>3.在1上修改/home/hadoop/app/hbase/hbase-0.90.1-cdh3u0/conf/regionservers在regionservers中添加如下內容：hadoop-node2hadoop-node3hadoop-node4hadoop-node54.拷1的hbase到2,3,4,58#scp-r./hbase-0.90.1-cdh**/hadoop@*.*.*.2:/home/hadoop/app/hbae/…..5.啟動hbase在節(jié)點1上執(zhí)行#start-hbase.sh6.啟動hbase的第二個HM