hadoop學(xué)習(xí)筆記

1、一、Hadoop的結(jié)構(gòu)：NameNodeDataNodeSecondaryNameNodeLobTrackerTaskTracker二、Hadoop安裝流程:1、 新建用戶和組2、 配置服務(wù)器域名和主機(jī)名3、 SSH無密碼登陸4、 JDK的安裝5、 Hadoop配置6、 格式化與啟動(dòng)7、 WEB監(jiān)控Hadoop的運(yùn)行狀態(tài)三、下載：VirtualBoxCenterOSJDKHadoopHBasePutty四、Hadoop FS Shellhadoop fs command parameterhelpls 、 lsrdu、 duscountcopyFromLocal、copyToLocal、cpm

2、oveFromLocal、mv、putcatchgrp、chmod、chownexpungeget、getmergemkdirrm、rmrsetrep、stat、tail、test、text、touchzLinux的安裝在安裝時(shí)內(nèi)存大小如果小于652MB時(shí)默認(rèn)會(huì)進(jìn)入text model(文本安裝模式)。因此在此處需要將內(nèi)存調(diào)大些網(wǎng)絡(luò)：VirtualBox有4種網(wǎng)絡(luò)連接方式1、 NAT2、 Bridged Adapter3、 Internal4、 Host-only AdapterVMWare有三種連接方式（上面的除去Internal）1、 NATNAT:Network Address Tran

3、slation:網(wǎng)絡(luò)地址轉(zhuǎn)換NAT模式是最簡單的實(shí)現(xiàn)虛擬機(jī)上網(wǎng)的方式4、 Host-only Adapter(主機(jī)模型)主機(jī)模型是一種比較復(fù)雜的模式，需要有比較扎實(shí)的網(wǎng)絡(luò)基礎(chǔ)知識(shí)才能玩轉(zhuǎn)，可以說前面的幾種模式所實(shí)現(xiàn)的功能在這種模式下，通過虛擬機(jī)及網(wǎng)卡的設(shè)置都可以被實(shí)現(xiàn)網(wǎng)絡(luò)配置：DEVICE="eth0"BOOTPROTO="static"HWADDR="08:00:27:34:F4:69"NM_CONTROLLED="yes"ONBOOT="yes"TYPE="Ethernet&quo

4、t;UUID="35b4fc2a-f79b-4013-a465-0a6aa793b487"IPADDR=0NETMASK=GATEWAY=DNS1=DNS2=Hadoop的配置步驟：添加用戶和組rootlocalhost # groupadd hadoop#添加組rootlocalhost # useradd hadoop g hadoop#添加用戶rootlocalhost # su hadoop#切換hadooplocalhost root$ exi

5、t#退出exitHosts的配置：rootlocalhost #vi /etc/hosts追加0 master#本機(jī)IP地址 主機(jī)名rootlocalhost #vi /etc/sysconfig/network修改HOSTNAME=masterrootlocalhost #hostname#查看主機(jī)名（需要重啟才能修改主機(jī)名）rootlocalhost #hostname master#臨時(shí)把主機(jī)名修改為masterrootlocalhost #logout#登出SSH無密碼登陸rootmaster #yum y install openssh-clients#安裝

6、SSH客戶端#ssh默認(rèn)是需要輸入密碼的：root master #ssh localhostyes#接下來需要輸入密碼，我們按Ctrl+C結(jié)束，接下來我們?cè)O(shè)置ssh的無密碼登陸root master #su hadoop#切換到hadoop用戶hadoopmaster root$ cdhadoopmaster $ ssh-keygen t rsa P f /.ssh/id_rsa#設(shè)置無密碼登陸公鑰和私鑰對(duì)(除了rsa格式的公私鑰還有dsa格式的：ssh-keygen t dsa P f /.ssh/id_dsa)hadoopmaster $ pwd/home/hadoophadoopmas

7、ter $ cd .ssh/hadoopmaster .ssh$ ls alh#查看生成的公鑰(id_rsa.pub)和私鑰(id_rsa)hadoopmaster .ssh$ cat id_rsa.pub >> authorized_keyshadoopmaster .ssh$ ls -alhhadoopmaster .ssh$ chmod 600 authorized_keyshadoopmaster .ssh$ cd .hadoopmaster $ chmod 700 .ssh/ssh無密碼登陸設(shè)置成功。測試：hadoopmaster $ ssh masterexhadoop

8、master $ exit#退出sshJDK的安裝：hadoopmaster $ exit#退出hadoop用戶，進(jìn)入root用戶rootmaster $ cd software/#卻換到JDK安裝包目錄rootmaster software$ mkdir /usr/java#創(chuàng)建解壓目錄rootmaster software$ tar -zxvf jdk-7u25-linux-i586.tar.gz C /usr/java#解壓（rpm包的解壓：rpm ivh jdkxxx.rpm）rootmaster software$ cd /usr/javarootmaster java$ ls al

9、hrootmaster $ vi /etc/profile#配置PATHexport JAVA_HOME=/usr/java/jdkexport PATH=$JAVA_HOME/bin:$PATHexport CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jarrootmaster $ source /etc/profile#使配置生效hadoop的配置rootmaster # cd /software/rootmaster software# ls -alhrootmaster software#mkdir /software_

10、jyrootmaster software# tar -zxvf hadoop-1.2.1.tar.gz C /software_jy rootmaster software# cd /software_jy/rootmaster software_jy# ls -alh總用量 12Kdrwxr-xr-x. 3 root root 4.0K 4月 22 11:19 .dr-xr-xr-x. 27 root root 4.0K 4月 22 11:19 .drwxr-xr-x. 15 root root 4.0K 7月 23 2013 hadoop-1.2.1rootmaster software

11、_jy# chown -R hadoop:hadoop hadoop-1.2.1/rootmaster software_jy# mv hadoop-1.2.1/ hadooprootmaster software_jy# ls -alhrootmaster software_jy# su hadoophadoopmaster software_jy$ cd hadoop/confhadoopmaster hadoop$ ls -alhhadoopmaster conf$ vi hadoop-env.sh#配置環(huán)境變量export JAVA_HOME=/usr/java/jdkexport H

12、ADOOP_HEAPSIZE=256export HADOOP_PID_DIR=/home/$USER/pidshadoopmaster conf$ vi core-site.xml<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!- Put site-specific property overrides in this file. -><configuration> &l

13、t;property> <name></name> <value>hdfs:/master:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/$/tmp</value> </property></configuration>hadoopmaster conf$ vi hdfs-site.xml#配

14、置hdfs需要配置namenode的數(shù)據(jù)目錄、datanode的數(shù)據(jù)目錄、集群個(gè)數(shù)（dfs.replication）<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!- Put site-specific property overrides in this file. -><configuration> <property> <name>dfs.re

15、plication</name> <value>1</value> </property> <property> <name>.dir</name> <value>/home/$/dfs_name</value> </property> <property> <name>dfs.data.dir</name> <value>/home/$/dfs_data</value&

16、gt; </property></configuration>hadoopmaster conf$ vi mapred-site.xml<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!- Put site-specific property overrides in this file. -><configuration> <property&

17、gt; <name>mapred.job.tracker</name> <value>master:9001</value> </property> <!- mapred的系統(tǒng)目錄 -> <property> <name>mapred.system.dir</name> <value>/home/$/mapred_system</value> </property> <!- mapred的本地目錄 -> <prope

18、rty> <name>mapred.local.dir</name> <value>/home/$/mapred_local</value> </property></configuration>hadoopmaster conf$ vi masters#配置輔助名稱節(jié)點(diǎn)hadoopmaster conf$ vi slaves#配置slaves到此已配置完成。驗(yàn)證一下是否真正的配置成功：hadoopmaster conf$ cd ./binhadoopmaster bin$ ./hadoop nam

19、enode -formathadoopmaster bin$ ./start-all.shhadoopmaster bin$ jps查看命令是否能正常執(zhí)行。hadoopmaster bin$ ./hadoop dfsadmin report#執(zhí)行dfsadmin命令，-report作用：打印一個(gè)報(bào)告查看hdfs中有什么內(nèi)容：hadoopmaster bin$ ./hadoop fs -ls /hadoopmaster bin$ ./hadoop fs -ls /home/hadoop/mapred_system查看日志文件：hadoopmaster bin$ cd ./logshadoopma

20、ster logs$ ls alhHbase的配置：hadoopmaster logs$ cd ./confhadoopmaster conf$ vi hdfs-site.xml<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!- Put site-specific property overrides in this file. -><configuration> <

21、property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>.dir</name> <value>/home/$/dfs_name</value> </property> <property> <name>dfs.data.dir</name> <value>/home

22、/$/dfs_data</value> </property> <!- HBase的配置 -> <property> <name>dfs.support.append</name> <value>true</value> </property> <property> <name>dfs.datanode.max.xcievers</name> <value>4096</value> </property&

23、gt; <!- HBase配置結(jié)束 -></configuration>重啟hadoophadoopmaster conf$ cd ./binhadoopmaster bin$ ./stop-all.shhadoopmaster bin$ ./start-all.shhadoopmaster bin$ ./hadoop dfsadmin report#查看啟動(dòng)狀態(tài)0:50030/0:50070/開始安裝Hbase解壓Hbasehadoopmaster bin$ exitexitrootmaste

24、r # cd /softwarerootmaster software# tar -zxvf hbase-0.94.10.tar.gz -C /software_jy/rootmaster software# cd /software_jy/rootmaster software_jy# ls alhrootmaster software_jy# mv hbase-0.94.10/ hbaserootmaster software_jy# chown -R hadoop:hadoop hbase/rootmaster software_jy# cd hbase/rootmaster hbase

25、# ls alhrootmaster hbase# cd conf/rootmaster conf# ls -alhrootmaster conf# vi hbase-env.shexport JAVA_HOME=/usr/java/jdkexport HBASE_MANAGES_ZK=truerootmaster conf# vi hbase-site.xml<?xml version="1.0"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?

26、><configuration> <!- HBase的數(shù)據(jù)存放的位置 -> <property> <name>hbase.rootdir</name> <value>hdfs:/master:9000/hbase</value> </property> <!- 是否為分布式 -> <property> <name>hbase.cluster.distributed</name> <value>true</value> &l

27、t;/property> <!- 主節(jié)點(diǎn) -> <property> <name>hbase.master</name> <value>hdfs:/master:60000</value> </property> <!- Hbase臨時(shí)目錄 -> <property> <!- Hbase臨時(shí)目錄 -> <property> <name>hbase.tmp.dir</name> <value>/home/$

28、/tmp</value> </property> <!- zookeeper -> <property> <name>hbase.zookeeper.quorum</name> <value>master</value> </property> <property> <name>perty.clientPort</name> <value>2181</value> </property

29、> <property> <name>perty.dataDir</name> <value>/home/$/zookeeper</value> </property></configuration>rootmaster conf# vi regionservers覆蓋hadoop類庫：與hadoop集成時(shí)需要，hadoop類庫需要與Hbase類庫相同rootmaster conf# cd ./lib/#進(jìn)入hbase的lib目錄rootmaster

30、 lib# ls hadoop-core-1.0.4.jar#查看hbase使用的hadooprootmaster lib# rm -rf hadoop-core-1.0.4.jar#刪除hbase中使用的hadoop類庫rootmaster lib# cp /software_jy/hadoop/hadoop-core-1.2.1.jar .#將hadoop中的hadoop類庫復(fù)制到hbase中的lib目錄下rootmaster lib# chown -R hadoop:hadoop hadoop-core-1.2.1.jar#授權(quán)hadooprootmaster lib# su hadoop#切換hadoop用戶hadoop