Hadoop與數(shù)據(jù)分析概述

1、技術(shù)創(chuàng)新，變革未來(lái)Hadoop與數(shù)據(jù)分析概述OutlineHadoop基本概念Hadoop的應(yīng)用范圍Hadoop底層實(shí)現(xiàn)原理Hive與數(shù)據(jù)分析Hadoop集群管理典型的Hadoop離線分析系統(tǒng)架構(gòu)常見(jiàn)問(wèn)題及解決方案關(guān)于打撲克的哲學(xué)打撲克與MapReduceInput split shuffle output 分牌各自齊牌交換再次理牌搞定統(tǒng)計(jì)單詞數(shù)The weather is goodThis guyis a good manToday is goodGood manis goodthe 1weather 1is 1good 1today 1is 1good 1this 1guy 1is 1a

2、1good 1man 1good 1man 1is 1good 1a 1 good 1good 1good 1good 1good 1man 1man 1the 1weather 1today 1guy 1is 1is 1is 1is 1this 1a 1good 5guy 1is 4man 2the 1this 1today 1weather 1流量計(jì)算6趨勢(shì)分析7/截圖用戶推薦8分布式索引910Hadoop 核心Hadoop Common分布式文件系統(tǒng)HDFSMapReduce框架并行數(shù)據(jù)分析語(yǔ)言Pig 列存儲(chǔ)NoSQL數(shù)據(jù)庫(kù) Hbase分布式協(xié)調(diào)器Zookeeper數(shù)據(jù)倉(cāng)庫(kù)Hive(使用

3、SQL)Hadoop日志分析工具ChukwaHadoop生態(tài)系統(tǒng)11DataData data data data dataData data data data dataData data data data dataData data data data dataData data data data dataData data data data dataData data data data dataData data data data dataData data data data dataData data data data dataData data data data da

4、taData data data data dataResultsData data data dataData data data dataData data data dataData data data dataData data data dataData data data dataData data data dataData data data dataData data data dataHadoop ClusterDFS Block 1DFS Block 1DFS Block 2DFS Block 2DFS Block 2DFS Block 1DFS Block 3DFS B

5、lock 3DFS Block 3MAPMAPMAPReduceHadoop實(shí)現(xiàn)作業(yè)執(zhí)行流程/ MapClass1中的map方法 public void map(LongWritable Key, Text value, OutputCollector output, Reporter reporter) throws IOException String strLine = value.toString(); String strList = strLine.split(); String mid = strList3; String sid = strList4;String timest

6、r = strList0;try timestr = timestr.substring(0,10);catch(Exception e)return;timestr += 0000; / 省略數(shù)十行 output.collect(new Text(mid + “” + “sid” + timestr , .);Hadoop案例(1) public static class Reducer1 extends MapReduceBase implements Reducer private Text word = new Text(); private Text str = new Text()

7、; public void reduce(Text key, Iterator values, OutputCollector output, Reporter reporter) throws IOException String t = key.toString().split(); word.set(t0);/ str.set(t1); output.collect(word,str);/uid kind /reduce /Reduce0b Hadoop案例(2) public static class MapClass2 extends MapReduceBase implements

8、 Mapper private Text word = new Text(); private Text str = new Text(); public void map(LongWritable Key, Text value, OutputCollector output, Reporter reporter) throws IOException String strLine = value.toString(); String strList = strLine.split(s+);word.set(strList0);str.set(strList1);output.collect

9、(word,str); Hadoop案例(3) public static class Reducer2 extends MapReduceBase implements Reducer private Text word = new Text(); private Text str = new Text(); public void reduce(Text key, Iterator values, OutputCollector output, Reporter reporter) throws IOException while(values.hasNext() String t = v

10、alues.next().toString(); / 省略數(shù)十行代碼 / 省略數(shù)十行代碼 output.collect(new Text(mid + “” + sid + “”) + ., .) Hadoop案例(4)BADAACBCBCDGroupCo-groupFunctionAggregate FilterFilterThinking in MapReduce(1)Thinking in MapReduce(2)Magics of Hive:SELECT COUNT(DISTINCT mid) FROM log_tableHive的魔力為什么淘寶采用Hadoop?webalizeraws

11、tat般若Atpanel時(shí)代日志最高達(dá)250GB/天最高達(dá)約50道作業(yè)每天運(yùn)行20小時(shí)以上Hadoop時(shí)代當(dāng)前日志470GB/天當(dāng)前366道作業(yè)平均67小時(shí)完成還有誰(shuí)在用Hadoop?雅虎北京全球軟件研發(fā)中心中國(guó)移動(dòng)研究院英特爾研究院金山軟件百度騰訊新浪搜狐IBMFacebookAmazonYahoo!Web ServersLog Collection ServersFilersData Warehousing on a ClusterOracle RACFederated MySQLWeb站點(diǎn)的典型Hadoop架構(gòu)HadoopRich ClientMetaStore ServerMysqlSchedulerThrift ServerWebJobClientCLI/GUIClientProgramWeb Server淘寶Hadoop與Hive的使用標(biāo)準(zhǔn)輸出,標(biāo)準(zhǔn)出錯(cuò)Web顯示(50030, 50060, 50070)NameNode,JobTracker, DataNode, TaskTracker日志本地重現(xiàn): Local RunnerDistributedCache中放入調(diào)試代碼調(diào)試目的：查性能瓶頸，內(nèi)存泄漏，線程死鎖等工具： jmap, jstat, hpr