Hadoop的部署方案設(shè)計

1、Hadoop的部署方案設(shè)計軟件包：hadoop-2.7.2.tar.gzhadoop-eclipse-plugin-2.7.2.jarhadoop-common-2.7.1-bin.zipeclipsejdk1.8.45hadoop-2.7.2 （linux 和 windows 各一份）Linux 系統(tǒng)（centos 或其它）Hadoop 安裝環(huán)境準(zhǔn)備環(huán)境：安裝Hadoop ,安裝步驟參見 Hadoop 安裝章節(jié)。安裝 eclipse。搭建過程如下：1.4等 hadoop-eclipse-plugin-2.7.2.jar拷貝至U eclipse/dropins 目錄下。.解壓 hadoop-2

2、.7.2.tar.gz 到 E 盤下。.下載或者編譯 hadoop-common-2.7.2 （由于 hadoop-common-2.7.1 可以兼容 hadoop-common-2.7.2 ,因此這里使用 hadoop-common-2.7.1 ）,如果想編譯可參考相關(guān)文章砌Thadocip-cmrncipi-2rJl bii2O1WB/1719W11；236W7I brxrr15/24 1052NOTlCE.bitREADIMMmkAnon牙堂麥曼的用戶克里叫變重值%EAhid*op-2確定orejM) i5峭210M CPU 2.50GHz新建第銅輯n刪除g3聞GB可用靠;544 9 1

3、9 孫 6/B/1T 1小X)16/l/6 9.20 20161/26 820 2011/26 820HADOOP HOME下的文件全部拷貝到E:hadoop-2.7.2binha-doop hadaop hadoop nadoop hadaop hadaop h內(nèi)門nDEBUG - I DEBUG - C Ex2fptionhadoop.dll在system32下面也要放一個，否則會報下圖的錯誤4.將 hadoop-common-2.7.1li d 口 口 p 亡 E|monh2 ： 口-日門.?日* r 考 ha d 0 口 ak，仃0用 indu . r hgdnp-connmcin 2

4、.7.1 hin.ripLKEN弘如文杵會明植RAR B5XWinRAR ZP,曬nvuu j. jl uiiL QLoija n-”LiJLii叼 t 9000 iiom stains qot valus2O16iffl/lT 11:232216/17 11:23J j. Ml l jl Ur 二 uU j dr， jji 幾 j u x wn L%J / 1 w i j.u -，上 Cliont (16420077 ccnr.ection to /1%1第.&1 :g=t FilTnfQ gok ? 3flT1514in ，2Ji3 - - . Pcir.r-KF.i:r icva.lan

5、gProceadBuiIcier (unknown source)util,.un7-;nin3d 三=I.1二丫/：4F3)u.ti bShFll. mn (She 11 . j av5 H561util FhAllSh&I ItoBunaRdExQ cu tor T sx 白 cut & iShemava:7W2)util. shall. ex&ccomiaiLd (sErIJ. . jaqn :e 151util .Shell.eKecCoimaEid (hs 11 ma ; V?0)fa. RauLocal ff 1 leEysten. set Perm! ss ion (RawLc

6、ua IF i luSyHtwm. j m vm冊c/Lzmj HXDOOF_HDME JWA_HDME MZVKN HIIHHIMOOPJiOHEE Ah-a口 up-芝 7 2MILKZ_PLUGIH_PATH E ： s otw ar elns t all ? ox i tFATHC; FHDSEAM FILES BEOABCOTR1MP的 RFRFRlFiFTr .R -nT 4 #: T n新建府mm .系統(tǒng)建E ： VbacVPortalCliaritsVEAhidoop-2.7.2E：jdU.B.45Kvsn_3 / 、. 啟動 eclipse,打開 windows-Prefer

7、ences 的 Hadoop Map/Reduce 中設(shè)置安裝目錄:type filter texttieneral、AntCoce RficommendersDats ManagpniFntHaooop Map/ReduuEHelpInstall/UpdateJavaJava EEJavs PersistenceJavaScriptMavenMylynOomphPlug-in DevelopmentRermote SystemsRun/DebugServerTeamTenriinalValidationHadoop Map/ReduceHadoop installation director

8、y:E:hadocp ,2.7,2Growse.OKCaned.打開 Windows-Open Perspective 中的 Map/Reduce ，在止匕 perspective下進(jìn)行hadoop程序開發(fā)。Open Perspective曲C5 Repository Exploring 輦; DebuglGit Repository Exploring餐,Jav3 default):，Java Erowsing蕾Java EEWJava Type Hierarchy爵1 JavaScript 利| Map/Reduce 件 Pliig-in Development tiResourcei1-

9、1 Tem SyncHirorrizing d WebX XMLOKCancel.打開 Windows-Show View中的 Map/Reduce Locations,如下圖右鍵選擇 New Hadoop location新建hadoop連接富 Edit Ha-doop location.Define Hadoop locationDefine the location q+ 4 Hddoop infrastructure 口running MspReduce appliedUser name; starsSOCKS projty點(diǎn)擊左側(cè)的DFSLocationsrriyhadoop（上一步

10、配罡的loc凱ion namuh如能看到user,表示安裝成功力 Project Exploit 鼠T DFS LocationsmyhadoopTO (1)日USB9.新建工程并添加WordCount類:10. 把 10perties 和 hadoop 集群中的 core-site.xml 力口入至U classpath中。我的示例工程是 maven 組織，因此放到 src/main/resources 目錄。：i /.跳，后m而三鼻后口耳 cgric-Htcjcm-1Iog4j.prc pcrtei JRf iysttMffl Ubrif)i ：j ：5 .1. Mawn CrpndcfK

11、Wk- Ub. J tirgrt hddoopEndnit3 pgrn.arnl. perties文件內(nèi)容如下:10g4j.rootLogger=debug,stdout,R10g4j.appender.stdout=org.apache.1og4j.Conso1eAppender10g4j.appender.stdout.1ayout=org.apache.1og4j.PatternLayout10g4j.appender.stdout.layout.ConversionPattern=%5P - %m%n10g4j.appender.R=org.apache.1og4j.Ro11ingFi

12、1eAppender10g4j.appenderRFi1e-mapreduce_test.log10g4j.appender.R.MaxFi1eSize=1MB10g4j.appender.R.MaxBackupIndex=110g4j.appender.R.1ayout=org.apache.1og4j.PatternLayout10g4j.appender.R.1ayout.ConversionPattern=%p %t %c - %m%n10.codefutures=DEBUG.在HDFS上創(chuàng)建目錄inputhadoop dfs -mkdir input.拷貝本地 README.txt

13、到 HDFS 的 input 里hadoop dfs -copyFromLocal /usr/local/hadoop/README.txt input. hadoop集群中hdfs-site.xml中要添加下面的配置，否則在 eclipse中無法向hdfs中上傳文件： dfs.permissionsfalseVM. 若 碰至 UCannot connect tocom.sun.jdi.connect.TransportTimeoutException ,貝關(guān)閉防火墻。.書寫代碼如下：復(fù)制代碼package com.hadoop.example;import java.io.IOExcepti

14、on;import java.util.StringTokenizer;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduc

15、e.Reducer;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.GenericOptionsParser;public class WordCount public static class TokenizerMapper extendsMapper private final static IntWritable one = new

16、 IntWritable(1);private Text word = new Text();public void map(Object key, Text value, Context context)throws IOException, InterruptedException StringTokenizer itr = new StringTokenizer(value.toString();System.out.print(-map: + value.toString() + n); while (itr.hasMoreTokens() word.set(itr.nextToken

17、();System.out.print(-map token: + word.toString() n);context.write(word, one);System.out.print(-context: + word.toString() + ”, + one.toString() + n);public static class IntSumReducer extendsReducer private IntWritable result = new IntWritable();public void reduce(Text key, Iterable values,Context c

18、ontext)InterruptedException int sum = 0;for (IntWritable val : values) sum += val.get();result.set(sum);context.write(key, result);System.out.print(-reduce:throws IOException,+ key.toString() + , +result.toString() + n);public static void main(String args) throws Exception System.setProperty(hadoop.

19、home.dir, E:hadoop-2.7.2);Configuration conf = new Configuration();String otherArgs = new GenericOptionsParser(conf, args) .getRemainingArgs();if (otherArgs.length != 2) System.err.println(Usage: wordcount );System.exit(2);)Job job = new Job(conf, word count);job.setJarByClass(WordCount.class);job.s

20、etMapperClass(TokenizerMapper.class);job.setCombinerClass(IntSumReducer.class);job.setReducerClass(IntSumReducer.class);job.setNumReduceTasks(2);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);FileInputFormat.addInputPath(job, new Path(otherArgs0);FileOutputFormat.setOutputPath(job, new Path(otherArgs1);System.exit(job.waitForCompletion(true) ? 0 : 1);).點(diǎn)擊 WordCount.java ,右鍵，點(diǎn)擊 Run As Run Configurations ,配 置運(yùn)行參數(shù)，即輸入和輸出文件夾，java application 里面如果沒有wordcount 就先把當(dāng)前 project run-java applationX Run ConngundoniCrcatfrf manage and run configurationsRun t java 口口