10、flume應(yīng)用場(chǎng)景原理基本架構(gòu)與案例分析

1、Flume OG基本架Flume OG基本架Flume NG基本架Flume案例分6Flume OG基Flume OG基本架7Flume OG基Flume OG基本架8通常由source和部V通常由source和部V組Source用于獲取數(shù)據(jù)，從文本文件，syslog，HTTPSink將Source獲得的數(shù)據(jù)進(jìn)一步傳輸給面的Flume自帶了很多source和sinksyslogTcp(5140) |)|9collectorSource(35853) | console collectorSource(35853) | collectorSink(file:/tmp/flume/collecte

2、d,syslog); collectorSource(35853) |collectorSink(hdfs:/namenode/user/flume/ AgentPCollector對(duì)應(yīng)AgentPCollector對(duì)應(yīng)關(guān)AgentPCollector對(duì)應(yīng)關(guān)AgentPCollector對(duì)應(yīng)關(guān)2Flume集群的制器借zookeeper管理管理多容錯(cuò)機(jī)制容錯(cuò)機(jī)制容錯(cuò)機(jī)制容錯(cuò)機(jī)制Flume OG基本架Flume Flume OG基本架Flume NG基本架Flume案例分Flume NG基Flume NG基本架Flume NG心概CFlume NG心概C其他組件erceto1Cannl lo1iE

3、vent是Flume數(shù)據(jù)傳輸?shù)幕鵈vent是Flume數(shù)據(jù)傳輸?shù)幕締蝒件的形式將數(shù)傳送到最Event由選的header和載有數(shù)據(jù)的一個(gè)bytearray成載有的數(shù)據(jù)對(duì)flume是O透 e在PQ文路由中使用擴(kuò)展Cnt是一個(gè)將原始log包裝成events個(gè)agent的實(shí)體Cnt是一個(gè)將原始log包裝成events個(gè)agent的實(shí)體在flume的拓?fù)浣Y(jié)構(gòu)中O是必ntFlume log4j C使用.apache.flume.api)定制特定ntSDKC一個(gè)Agent包含Source,一個(gè)Agent包含Source, Channel, Sink和其他組件agent是flume流的基礎(chǔ)部Agent之Ag

4、ent之Agent之Source負(fù)責(zé)接收event或通過特殊機(jī)制產(chǎn)生eventAgent之Source負(fù)責(zé)接收event或通過特殊機(jī)制產(chǎn)生eventO類型的P系統(tǒng)集成的Source: Syslog, 自生成件的Source: 用于Agent和Agent之間通信的IPC Source: Source必至少和一個(gè)channel關(guān)Agent之PEvent Agent之PEvent Event Agent之Agent之O的Channel提供的持久化水平也是O一的MemoryChannel: FileChannel基于WAL預(yù)寫式日志W(wǎng)rite-Ahead JDBC Channel: 基于嵌入Datab

5、aseChannel支持，提供較弱的和任何數(shù)量的Source和Sink工Agent之Sink負(fù)責(zé)將event傳輸?shù)絈Agent之Sink負(fù)責(zé)將event傳輸?shù)絈一跳或最終目的，成成將event從channel移除O類型的儲(chǔ)event到最終目的的終端Sink. 如: HDFS, 自消耗的Sink. 如: Null 用于Agent間通信的IPC sink: 必作用于一個(gè)確W的其他幾個(gè)作用于Source，按照預(yù)設(shè)的其他幾個(gè)作用于Source，按照預(yù)設(shè)的序在必要地方裝飾和過濾Channel允許Source基于預(yù)設(shè)的標(biāo)準(zhǔn)，從所有Channel中，選擇一個(gè)或alSink多個(gè)一個(gè)sor通過組中所有實(shí)現(xiàn)負(fù)載均

6、衡也在一個(gè)Sink失敗時(shí)轉(zhuǎn)移到另一個(gè)Flume OG基本架Flume OG基本架Flume NG基本架Flume案例分Q步驟2修改etc/profileexport FLUME_HOME=/opt/software/flume-0.9.4- export步驟3驗(yàn)證安步驟5運(yùn)行命啟OG集群態(tài)OG集群態(tài)在已經(jīng)啟的master節(jié)點(diǎn)P，依次輸入localhost s如執(zhí)行exec config a1 tailDir(/data/logfile lume.collector.port兩個(gè)參數(shù)決定和點(diǎn)，填寫source1sink選OG集群拓?fù)渑渲肙G集群拓?fù)渑渲胊gentA:src|agentB : sr

7、c | agentSink(collector,35853); agentC : src | agentSink(collector,35853); agentD :src | agentSink(collector,35853); agentE : src | agentSink(collector,35853); agentF : src | agentSink(collector,35853); collector : collectorSource(35853) |OGO種靠性級(jí)end to end，這個(gè)級(jí)OGO種靠性級(jí)end to end，這個(gè)級(jí)別是sonanacknowledgem

8、ent, andretry if no acknowledgement is 效率最好，agentO寫入到本地任何數(shù)據(jù)，如果在直接刪除消息發(fā)現(xiàn)處理失敗，AgentSink 是agentE2ESink 的別OG集群拓?fù)渑渲肙G集群拓?fù)渑渲胊gentA:src|agentB : src | agentE2ESink(collectorA,35853); agentC : src | agentE2ESink(collectorB,35853); agentD : src |agentE2ESink(collectorB,35853); agentE : src | agentE2ESink(col

9、lectorC,35853); agentF : src | agentE2ESink(collectorC,35853);collectorA :collectorSource(35853) | collectorSink(hdfs:/.,src); collectorB : collectorSource(35853) | collectorSink(hdfs:/.,src); collectorC : collectorSource(35853) | collectorSink(hdfs:/.,src);OG集群拓?fù)渑渲肙G集群拓?fù)渑渲胊gentA:src|agentB : src |

10、agentE2EChain(collectorA:35853,collectorC:35853); agentC : src | agentE2EChain(collectorB:35853,collectorA:35853); agentD : src |agentE2EChain(collectorB:35853,collectorC:35853); agentE : src | agentE2EChain(collectorC:35853,collectorA:35853); agentF : src | agentE2EChain(collectorC:35853,collectorB

11、:35853); collectorA : collectorSource(35853) | collectorSink(hdfs:/.,src); collectorB : collectorSource(35853) | collectorSink(hdfs:/.,src); collectorC : collectorSource(35853) | collectorSink(hdfs:/.,src);Q步驟2修改etc/profileexport FLUME_HOME=/opt/software/flume-1.4.0.tar.gz export PATH=.:$PATH:$FLUME

12、_HOME/bin步驟3驗(yàn)證安步驟5運(yùn)行命啟NG部署-配置文件格Java PropertiesNG部署-配置文件格Java Properties 文件格# 注釋 key1valuekey2=multi-line層級(jí)配agent1.channels.myChannel.type = = 使用配關(guān).mySource.type= .mySource.channels=NG部署-簡(jiǎn)單配置實(shí)# 定OsourceNG部署-簡(jiǎn)單配置實(shí)# 定Osource1channel和sink=agent1.channels = agent1.sinks=.src1.type= .src1.channels=.src1.

13、bind=.src1.port = #定O并配置 sink1 agent1.sinks.sink1.type = logger agent1.sinks.sink1.channelch1# 定O并配置 ch1 agent1.channels.ch1.typememoryNG部署-實(shí)例NG部署-實(shí)例NG部署-實(shí)例第一層1配置實(shí)例a1.channels=a1.sinks.k1.channelNG部署-實(shí)例第一層1配置實(shí)例a1.channels=a1.sinks.k1.channel=a1.sinks.k1.type = AVRO a1.sinks.k1.hostname=a21.ex a1.sin

14、ks.k1.port = 41414=a1.sinks=k1sor.type=sor.backoff = truea1.sinks.k2.channel=a1.sinks.k2.type = AVRO a1.sinks.k2.hostname=a22.ex a1.sinks.k2.port = 41414.r1.channels= .r1.type=.r1.port=NG部署-實(shí)例第層2配置實(shí)例NG部署-實(shí)例第層2配置實(shí)例a2.channels = =a2.sinks = a2.channels.c1.type = .r1.channels=.r1.type=.r1.bind=.r1.port

15、 = a2.sinks.k1.channel = a2.sinks.k1.type = HDFS a2.sinks.k1.hdfs.path = hdfs:/namenode.ex a2.sinks.k1.hdfs.fileType=DataStreamNG部署-實(shí)例NG部署-實(shí)例NG部署-實(shí)例a1.channels=a1.sinks=NG部署-實(shí)例a1.channels=a1.sinks=k1a1.sinkgroups = g1 a1.sinkgroups.g1.sinks=k1k2sor.type=sor.backoff = truea1.channels.c1.type=第一層實(shí)1配.r

16、1.channels=.r1.host=a1.sinks.k1.channel=a1.sinks.k1.type = AVRO a1.sinks.k1.hostname=a21.ex a1.sinks.k1.port = 41414 a1.sinks.k2.channel = c1 a1.sinks.k2.type = AVRO a1.sinks.k2.hostname=a22.ex a1.sinks.k2.port = 41414NG部署-實(shí)例第層2配置實(shí)例.r1.channels=c1.r1.type = NG部署-實(shí)例第層2配置實(shí)例.r1.channels=c1.r1.type = .r

17、1.bind= .r1.port =.r1.selector.header= .r1.selector.default= =a2.sinks = k1 k2 a2.sinkgroups.g1.sinks=k1sor.type=.0=c1 .1=c1 .2=c1 .3=c1 sor.backoff = true Dir=/var/run/flume-ng/flume/ch-1/checkpo a2.channels.c1.dataDirs = /var/run/flume-ng/ flume/ch-1/dataa2.sinks.k1.channel=a2.sinks.k1.type = HDFS a2.sinks.k1.hdfs.path = hdfs:/nn.exa2.channels.c2.type=Dir=/var/run/flume-ng/flume/ch-2/checkpo a2.channels.c2.dataDirs = /var/run/flume-ng/ flume/ch-2/data/demo/%Y-a2.sinks.k1.hfds.filePrefix = FlumeData-a2.sinks.k1.hdfs.fileType=DataStream a2.sin