大數(shù)據(jù)環(huán)境軟件

大數(shù)據(jù)環(huán)境軟件操作系統(tǒng)CentOS7#zipunzip命令安裝yuminstallzipunzip#netstat等命令網(wǎng)絡(luò)工具包安裝yuminstallnet-tools軟件安裝包-rw-r--r--.1rootroot209853354月-rw-r--r--.1rootroot1535128795月drwxr-xr-x.10rootroot40967月-rw-r--r--.1rootroot715038924月-rw-r--r--.1rootroot 13581905月-rw-r--r--.1rootroot 18773805月-rw-r--r--.1rootroot176993067月3010:14apache-storm-0.9.3.tar.gz16:15jdk-7u79-linux-x64.tar.gz1011:55jzmq1710:06kafka_2.10-.zip11:26redis-3.0.1.tar.gz122011zeromq-2.1.7.tar.gz1002:50zookeeper-3.4.6.tar.gzJAVAvi/etc/profile在文件末尾增加如下配置內(nèi)容#setjavaenvironmentJAVA_HOME=/usr/local/jdk1.7.0_79/CLASSPATH=.:$JAVA_HOME/lib/tools.jarPATH=$JAVA_HOME/bin:$PATHexportJAVA_HOMECLASSPATHPATH配置生效source/etc/profile驗(yàn)證配置結(jié)果[root@localhostlocal]#java-versionjavaversion"1.7.0_79"

Java(TM)SERuntimeEnvironment(build1.7.0_79-b15)JavaHotSpot(TM)64-BitServerVM(build24.79-b02,mixedmode)[root@localhostlocal]#javac-versionjavac1.7.0_79zookeeper重命名$Zookeeper_home/conf/zoo_sample.cfg為zoo.cfgZookeeper默認(rèn)啟動(dòng)使用zoo.cfg配置文件進(jìn)入zookeeper安裝目錄cd/usr/local/zookeeper-3.4.6/啟動(dòng)zookeeper./bin/zkServer.shstart通過連接命令確認(rèn)是否連接成功./fallX'LtCll.i-Ji l￡t2.J￡a.l.2D0：21^Laais-D^-io/20/ED14GOTZaiE-07-lU11917iZ1jBE13015-07-1011=17；Z1^D313QL5-m-10ll；11；aais-D^-io/20/ED14GOTZaiE-07-lU11917iZ1jBE13015-07-1011=17；Z1^D313QL5-m-10ll；11；31.,n3Jaai5-07-iij2015-07-10114,215弱[□yid!]-INTtl[DTiili]-INF][nyid;]-IMro[ayid;]-INFO[nyid;]-IHP)[nyid;]-Um[iiaiiL!EriviioiM-ntplOO][皿OUE±iVll4UKli>：8100][lain;FiivinjmwirfilOCi][iBin.:Fiivin]nD?ii：BinO](min.;EiivinjDMiit-ElLij]Cllear.eitTilonMADLL.?E3iiin-3.A.6-1S996SPbuiltaijOS(lain.;EnrvinjnMJit-ElLij]-Cliwi匕CLiaii：ea*7iEDniuDt:liadT-iCliMireii^.EDni.?Qter]職土^t3iao=Ja7?0_7^euKedh■■皂工mql-eIeCnEpaiatininClieaiJ:-eix^itonunt;jnua.hoae?JmcFlcicL/jdJtal.*cavacnnMnt;jsua,cJ933,pai^-/u3i71DcaL/3：DDJfeeepeE-3.4.6fb...■. : ；..ii： -.：:■.■■I.,-■ ■…I：.i.-.：.：......■111>：■；.：■■■.1.1 .：：■. - | 1 .：.-..■..：>■：.:-i1>:...:..-..yiLCij-dax.i1LOCAl.i'ZOC^AApA￡-3-4.b.yilii/flllJl]-Apl-L-t-t/U^XI1LOCAl.^OOiE&GfEi^￡-3-4.b/bm/../]3SznfiCL5!-3."1.ClFiniL.]yujt.'1□iialXEoa'kwt'tE-a.-4- _/lib/lou4ji-L-E.Ifi.j*E_/uar/Jocal.toaYtttSEE-S.4.SjT>in.?!../]ib.11jlJtit-fl. :j'liSE/loral/Eacjkttpei-S.4(b/buL*1!1 €g]u:i/U5E/lac-al.,ED41：ttj>eE-3.4u6/Iiid/b.ami.i'lib/3'-3-irt.■■TuiyiooiL.i'tocttteper-3a6/buL1--i/cosE]?1Aiseen.^rnni.cQtcj日喧亂，JdtE3iyiparh=rt.T5E/3i,J-3',PBrk-iipc-5ZllbfqdJlocWzidkKT-匕科『iih/tcDlmw：2015-07-1011;17;Z1^D34[ayid;]-en.^rnni.cQtcj日喧亂，JdtE3iyiparh=rt.T5E/3i,J-3',PBrk-iipc-5Zllbfqd2015-07-102OL￡-D?-1Dll=10：±l^B5daais-D^-io2a.L5-D7-lD]1217:21^0572015-07-1011917l31jBE72015-07-102OL￡-D?-1Dll=10：±l^B5daais-D^-io2a.L5-D7-lD]1217:21^0572015-07-1011917l31jBE7aaiS-D7-1011317i31jBE7201-5-07-1011;1?;21^37￡115-07-10[oryid;]-IHP)[ayid:]-INFO[[■Vid:]-IHFO[b￥ld!]-INFO[Eiyid:]-iwra[DTiili]-INF][nyicLi]-INF][tiyid;]-INFO[nyid;]-IMF)(lain.;EinvinjnMJit-ElLij]-CHent(sain.:EnvixcraMTihgluO]-CLL?nh|>Aih.:EJi^lxouKncBluQ]-CLLsihl[^AIIl!EJiVllAIW^CPlOCl]-ClidH此[kaiiL：Erivixoi-CLlsnc[^Bza9EliVli4UH3ic81OO]-Clieait[ZlEIZU!EDV1X4UK31c81L^]-[iBin:FiiviirnQ?ii：BinO]-r1Lhi￡小或，io-tipdjx-/t*p■n*neanunL:jhuh.. eOH.?it：onMKi[.：Da,h.-MK--LihUh亡I曜」Ad.a.￡Cti-iUidjc-4亡ilKLankAFjt.as.V￡E^iaii-3.JD.D-2S.±17.K8fi_fi4e11*71匚口皿e口位：aEgx■H8tea*ziEi>ni.4Dt:iuci,ert'satnnMDt-nisei.djcsyujHJ'lnciL/EDnteer-s-E-a.4.5[sain;Epp^jecptc]-Initiatirurcllcm-ccameGt-LcCc.rnrx>sr'：5t-ilDg=i3Z：i155.1.S3P-Z15Lse3-jLcaiiTiicoijt-300Cijnat-chstBDEq,.apache.wo>Eepen：BErjnKc?peEHaLnfiMyWatd^EEEll&3Jd!6duwto2ddUh?P-ieI2￡ILE-D7-lfl [Q^ld!]-INF)(>aih.-S￡-ndTh￡wvEl[i￡>2.1.200：21^1>i:ClliaKiLCiUHtjwtridThXwwCl^^-S]-叩如L明doakAl：此hMT匚ionLa[亡heeISZ.lfia.1.2QD/192.Jfid.1.2OQjL.liilltiocarutipcto占ucn亡iLdRid匚亡wwq9心Li.uiiktio'imexxoc)20L5-D7-1CI11117!21.,903[Qyid:]-INPO[>aiiL-StHiniLi：e4Ll(192.]SB.1.2DD.Z131).riieDtDiKii&^iTniie?]^IB52]-3ackscWHjyxitm—wkf電-kf電-itSlEP-Cl-：- 匚bli^heiiE1E>Z116$ulu200/192^16$^1.ZOO121B1riniti^TLDg3EflJLoai?illEiftsupportlet31115-07-1011:17;Z1^9ZJ|[nyid;]-IMHi[iain-5?nfTtiEead(13Z.150.i.3DD-ZiaiJ-ClientCnKn45?ntfTliie-Kl&lZ35]-況EmmInstL也him口CCOplBT-CCOSCIIPET13E,皿山1. ?101!^L￡00=r?E33innjd■SiJ-lETndCrllifldOflOi^nEgt■偵ccdtjieaat■DDOCCifXTCHEP”TiD-JitdEvetiLSTiDti3ynLCctmtcijrftypt:IQctfitpd.Eli.null[￡|￡=192.LSS.1.2CCI121B1iOhmcm'iD]|C-x*nK?dtiIKL]?Iggl惑@GCCyum-yinstallgccyum-yinstallgcc-c++stormZeroMQtar-xzfzeromq-2.1.7.tar.gzcdzeromq-2.1.7./configuremakesudomakeinstallzeroMQ需要安裝UUID包支持yum-yinstalluuidyuminstalllibuuid-develJZMQ手動(dòng)編譯zeromq內(nèi)的java文件$cdsrc/org/zeromq/$/jzmq/src/org/zeromq]$javac*.java$cd..$make...success!$sudomakeinstall3.paythonpython-Vstorm集群部署進(jìn)入storm安裝目錄cd/usr/local/apache-storm-0.9.3/nimbus、stormUI為00supervisor為01、02以下是啟動(dòng)Storm各個(gè)后臺(tái)進(jìn)程的方式：Nimbus:在Storm主控節(jié)點(diǎn)上運(yùn)行"bin/stormnimbus>/dev/null2>&1&”啟動(dòng)Nimbus后臺(tái)程序，并放到后臺(tái)執(zhí)行；Supervisor:在Storm各個(gè)工作節(jié)點(diǎn)上運(yùn)行"bin/stormsupervisor>/dev/null2>&1&"啟動(dòng)Supervisor后臺(tái)程序，并放到后臺(tái)執(zhí)行；UI:在Storm主控節(jié)點(diǎn)上運(yùn)行"bin/stormui>/dev/null2>&1&"啟動(dòng)UI后臺(tái)程序，并放到后臺(tái)執(zhí)行，啟動(dòng)后可以通過http://{nimbushost}:8080觀察集群的worker資源使用情況、Topologies的運(yùn)行狀態(tài)等信息。

kafkaKAFKA_HOME/config/perties下面為00機(jī)器的配置broker.id=200=00zookeeper.connect=00:2181卜面為01機(jī)器的配置broker.id=201=01zookeeper.connect=00:2181下面為02機(jī)器的配置broker.id=202=02zookeeper.connect=00:2181三臺(tái)集群機(jī)器必須使用相同的zookeeper服務(wù)00:2181啟動(dòng)命令cdKAFKA_HOME/cd/usr/local/kafka_2.10-/nohup./bin/kafka-server-start.sh./config/perties&redis10#進(jìn)入redis目錄cd$Redis_HOME#編譯makeMALLOC=libc#安裝測(cè)試工具yum-yinstallctl#測(cè)試maketest#安裝makeinstall啟動(dòng)redisredis-server/etc/redis_6379.confredis-server/etc/redis_7379.conf啟動(dòng)sentinelredis-sentinel/etc/sentinel.confORACLEoracle管理員密碼linkAge1234510:1158/em啟動(dòng)Oracle示例#切換到Oracle用戶su-oracle#登錄sqlplussqlplus/assysdba#啟動(dòng)Oracle實(shí)例SQL>startup;#退出sqlplusSQL>exit;#啟動(dòng)Oralce監(jiān)聽程序$lsnrctlstart#停止Oralce監(jiān)聽程序$lsnrctlstop#顯示Oralce監(jiān)聽狀態(tài)$lsnrctlstatus常見問題:發(fā)布topologies到遠(yuǎn)程集群時(shí)，出現(xiàn)Nimbushostisnotset異常原因是Nimbus沒有被正確啟動(dòng)起來，可能是storm.yaml文件沒有配置，或者配置有問題。解決方法：打開storm.yaml文件正確配置：nimbus.host:"xxx.xxx.xxx.xxx”，重啟nimbus后臺(tái)程序即可。發(fā)布topologies到遠(yuǎn)程集群時(shí)，出現(xiàn)AlreadyAliveException(msg:xxxisalreadyactive)異常原因是提供的topology與已經(jīng)在運(yùn)行的topology重名。解決方法：發(fā)布時(shí)換一個(gè)拓?fù)涿Q即可。啟動(dòng)Supervisor時(shí)，出現(xiàn)java.lang.UnsatisfiedLinkError:/usr/local/lib/libjzmq.so.0.0.0:libzmq.so.1:cannotopensharedobjectfile:Nosuchfileordirectory異常原因是未找到zmq動(dòng)態(tài)鏈接庫(kù)。解決方法1：exportLD_LIBRARY_PATH=/usr/local/lib解決方法2：編輯/etc/ld.so.conf文件，增加一行：/usr/local/lib。再執(zhí)行sudoldconfig命令，重啟Supervisor。發(fā)布drpc類型的topologies到遠(yuǎn)程集群時(shí)，出現(xiàn)空指針異常，連接drpc服務(wù)器失敗原因是未正確配置drpc服務(wù)器地址。解決方法：在conf/storm.yaml文件中增加drpc服務(wù)器配置，啟動(dòng)配置文件中指定的所有drpc服務(wù)。內(nèi)容如下：drpc.servers:-"drpc服務(wù)器ip"客戶端調(diào)用drpc服務(wù)時(shí)，worker的日志中出現(xiàn)Failingmessage，而bolt都未收到數(shù)據(jù)錯(cuò)誤日志如下所示：2011-12-0209:59:16task[INFO]Failingmessagebacktype.storm.drpc.DRPCSpout$DRPCMessageId@3770bdf7:source:1:27,stream:1,id:{-5919451531315711689=-5919451531315711689},[/blog/1,{"port”:3772,"id”:"5”,"host”:"4"}]解決方法：主機(jī)名，域名，hosts文件配置不正確會(huì)引起這類錯(cuò)誤。檢查并修改storm相關(guān)機(jī)器的主機(jī)名，域名，hosts文件。重啟網(wǎng)絡(luò)服務(wù):servicenetworkrestart,重啟storm，再次調(diào)用drpc服務(wù)，成功。Hosts文件中必須包含如下內(nèi)容：[nimbus主機(jī)ip][nimbus主機(jī)名][nimbus主機(jī)別名][supervisor主機(jī)ip][supervisor主機(jī)名][supervisor主機(jī)別名][zookeeper主機(jī)ip][zookeeper主機(jī)名][zookeeper主機(jī)別名]發(fā)布topologies時(shí)，出現(xiàn)不能序列化log4j?Logger的異常原因是日志系統(tǒng)無法正確支付序列化。解決方法：使用slf4j代替log4j。bolt在處理消息時(shí)， worker的日志中出現(xiàn)Failingmessage原因可能是因?yàn)門opology的消息處理超時(shí)所致。解決方法：提交Topology時(shí)設(shè)置適當(dāng)?shù)南⒊瑫r(shí)時(shí)間，比默認(rèn)消息超時(shí)時(shí)間(30秒)更長(zhǎng)。conf.setMessageTimeoutSecs(60);storm啟動(dòng)時(shí)報(bào)nojzmqinjava.library.path錯(cuò)誤原因是找不到j(luò)zmq，默認(rèn)情況下在執(zhí)行install_zmq.sh時(shí)，那些.so文件安裝路徑在/usr/local/lib，但是實(shí)際安裝時(shí)可能裝在其他的路徑下了。解決方法：在storm.yaml中添加：java.library.path:"/opt/storm/jzmq/lib:/opt/storm/zeromq/lib:/usr/local/lib:/opt/local/lib:/usr/lib"安裝jzmq時(shí)遇到Noruletomaketarget'classdist.noinststamp'的make錯(cuò)誤具體的make錯(cuò)誤信息：make[1]:***Noruletomaketarget'