ClouderaManager大數(shù)據(jù)平臺部署指南

部署指南ClouderaManagerCDH官方共給出了3中安裝方式。第一種方法必需要求全部機器都能連網(wǎng)，由于CMyum1、ClouderaManagerCDH不支持在容器中部署。OracleJDK71.7u80，C5.3.x及以上版本也支持使用oracleJDK8CDHJDK版本必需全都。Python2.4Python3IPv4IPv6disabledtar壓縮包的離線安裝方式，會在6.0.0中停頓支持該方式。支持selinux，但假設(shè)不生疏使用方法，建議關(guān)閉。治理端口為7180，此外還有很多根底應用使用的端口，可以運行iptables，需要確認有恰當?shù)氖跈?quán)掌握，例如至少要對同網(wǎng)段的訪問放開全部端口。2、ClouderaManager部署的幾個階段和可選擇的方式y(tǒng)umPATHC，RPMPATHB3、主機節(jié)點信息和根底配置主機信息NameNode1TB/dfs/dn1RAIDRAID掌握器，則直接對每塊盤設(shè)置一個單獨的RAID0，仍舊當作單盤使用。/dfs/dn2，更多的數(shù)據(jù)盤則按命名規(guī)章和掛盤規(guī)章擴展即可。RAID性。DataNodeRAID1CDH0cluster-701cluster-712cluster-723cluster-734cluster-745cluster-75CentOS6.9Minimal/etc/hosts文件中。免密登錄cluster-70ssh：ssh-keygen將該密鑰分發(fā)至其它各節(jié)點，以實現(xiàn)免密鑰登錄，如：ssh-copy-id cluster-71selinuxiptables全部節(jié)點配置為關(guān)閉selinux。全部節(jié)點的iptables設(shè)置為對同網(wǎng)段地址放開全部端口的訪問權(quán)限。中即可。cat<<EOF>>/etc/sysctl.confvm.swappiness=0net.ipv6.conf.all.disable_ipv6=1net.ipv6.conf.default.disable_ipv6=1net.ipv4.ip_local_port_range=102465000EOFsysctl-pechonever>/sys/kernel/mm/transparent_huge/defragechonever>/sys/kernel/mm/transparent_huge/enabledcat<<EOF>>/etc/rc.localechonever>/sys/kernel/mm/transparent_huge/defragechonever>/sys/kernel/mm/transparent_huge/enabledEOF4JDK下載并分發(fā)OracleJDK安裝包至全部節(jié)點，執(zhí)行安裝和配置命令。rpm-ivhjdk-7u80-linux-x64.rpm直接刷下面腳本完成配置：cat<<EOF>>/etc/profileJAVA_HOME=/usr/java/jdk1.7.0_80PATH=\$JAVA_HOME/bin:\$PATH:.CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarexportJAVA_HOMEexportPATHexportCLASSPATHEOFsource/etc/profile以上JDK配置，需要在每個節(jié)點上都處理。5、安裝和配置數(shù)據(jù)庫主要有以下業(yè)務會使用到數(shù)據(jù)庫效勞：ClouderaManager，存放各種效勞、配置數(shù)據(jù)，空間大約100MB，很重要，應常常OozieServer-ContainsOozieworkflow,coordinator,andbundledata.Cangrowverylarge.SqoopServer-Containsentitiessuchastheconnector,driver,linksandjobs.Relativelysmall.ActivityMonitor-Containsinformationaboutpastactivities.Inlargeclusters,thisdatabasecangrowlarge.ConfiguringanActivityMonitordatabaseisonlynecessaryifaMapReduceserviceisdeployed.Reports Manager - Tracks disk utilization and processing activities over time.Medium-sized.HiveMetastoreServer-ContainsHivemetadata.Relativelysmall.HueServer-Containsuseraccountinformation,jobsubmissions,andHivequeries.Relativelysmall.SentryServer-Containsauthorizationmetadata.Relativelysmall.ClouderaNavigatorAuditServer-Containsauditinginformation.Inlargeclusters,thisdatabasecangrowlarge.ClouderaNavigatorMetadataServer-Containsauthorization,policies,andauditreportmetadata.Relativelysmall.由于預備使用cluster-70作為NameNodeMaster，所以把mysql數(shù)據(jù)庫安cluster-71，以分散下風險?？桑簓uminstallmysql-servermysqlchkconfigmysqldonservicemysqldstart完成數(shù)據(jù)庫初始化：/usr/bin/mysql_secure_installation優(yōu)化數(shù)據(jù)庫配置servicemysqldstopInnoDBlog/var/lib/mysql/ib_logfile0and/var/lib/mysql/ib_logfile1outof/var/lib/mysql/參照下面調(diào)整f參數(shù)，以更適合CDH效勞使用：[mysqld]datadir=/var/lib/mysqlsocket=/var/lib/mysql/mysql.sockuser=mysql#Disablingsymbolic-linksisrecommendedtopreventassortedsecurityriskssymbolic-links=0transaction-isolation=READ-COMMITTEDkey_buffer_size=32Mmax_allowed_packet=32Mthread_stack=256Kthread_cache_size=64query_cache_limit=8Mquery_cache_size=64Mquery_cache_type=1max_connections=550#binlog_format=mixedread_buffer_size=2Mread_rnd_buffer_size=16Msort_buffer_size=8Mjoin_buffer_size=8M#InnoDBsettingsinnodb_file_per_table=1innodb_flush_log_at_trx_commit =2innodb_log_buffer_size=64Minnodb_buffer_pool_size=2Ginnodb_thread_concurrency=8innodb_flush_method=O_DIRECTinnodb_log_file_size=512M[mysqld_safe]log-error=/var/log/mysqld.logpid-file=/var/run/mysqld/mysqld.pidsql_mode=STRICT_ALL_TABLES啟動mysql，假設(shè)有報錯，可依據(jù)日志信息排查。創(chuàng)立需要使用的幾個庫createdatabasedatabaseDEFAULTCHARACTERSETutf8;grantallondatabase.*TO”user”@”%”IDENTIFIEDBY”password”;latin時日志中會報錯創(chuàng)立索引時超長度限制。mysqljdbcdriver一下。Serverhost,ActivityMonitor,ReportsManager,HiveMetastoreServer,HueServer,SentryServer,ClouderaNavigatorAuditServer,andClouderaNavigatorMetadataServer從以下網(wǎng)址下載需要使用的驅(qū)動包：“://mysql/downloads/connector/j/5.1.html“://mysql/downloads/connector/j/5.1.htmltar-zxvf mysql-connector-java-5.1.43.tar.gzmkdir-p/usr/share/java/cpmysql-connector-java-5.1.43/mysql-connector-java-5.1.43-bin.jar/usr/share/java/mysql-connector-java.jar6NTP的系統(tǒng)時間嚴格同步。yum-yinstallntpservicentpdrestartchkconfigntpdon節(jié)點均配置/etc/ntp.confserver7ClouderaManagerServer軟件ClouderaManagerServercluster-70下載和分發(fā)安裝資源clouderamanagers://archive.cloudera/cm5/cm/5/cdhparcels下載地址：“://archive.cloudera/cdh5/parcels/“://archive.cloudera/cdh5/parcels/wget-cs://archive.cloudera/cm5/cm/5/cloudera-manager-el6-cm5.12.0_x86_64.tar.gzwget-c“://archive.cloudera/cdh5/parcels/5.12/CDH-5.12.0-1.cdh5.12.0“://archive.cloudera/cdh5/parcels/5.12/CDH-5.12.0-1.cdh5.12.0.p0.29-el6.parcelwget-c“://archive.cloudera/cdh5/parcels/5.12/CDH-5.12.0-1.cdh5.1“://archive.cloudera/cdh5/parcels/5.12/CDH-5.12.0-1.cdh5.12.0.p0.29-el6.parcel.sha1wget-c“://archive.cloudera/cdh5/parcels/5.12/manifest.json“://archive.cloudera/cdh5/parcels/5.12/manifest.jsonsha，否則上傳到系統(tǒng)中后，CDHshacloudera-manager-el6-cm5.12.0_x86_64.tar.gz文件放到全部節(jié)點的/opt/cloudera-manager名目下。mkdir/opt/cloudera-managercd/opttarzxvfcloudera-manager-el6-cm5.12.0_x86_64.tar.gzCMServeruseradd--system--home=/opt/cloudera-manager/cm-5.12.0/run/cloudera-scm-server--no-create-home--shell=/bin/false--comment“ClouderaSCMUser“cloudera-scmCMServercluster-70：mkdir-p/var/lib/cloudera-scm-serverchowncloudera-scm:cloudera-scm/var/lib/cloudera-scm-serverClouderaManagerAgentssed-i”s/server_host=localhost/server_host=cluster-70/”/opt/cloudera-manager/cm-5.12.0/etc/cloudera-scm-agent/config.iniCMServercluster-71cluster-71：JDBCcp mysql-connector-java-5.1.43-bin.jar cm-5.12.0/share/cmf/lib初始化腳本語法如下：scm_prepare_database.shdatabase-type[options]database-nameusernamepassword/opt/cloudera-manager/cm-5.12.0/share/cmf/schema/scm_prepare_database.shmysql-hlocalhost-uroot-pHstest2017scmscmscm屏幕輸出信息：JAVA_HOME=/usr/java/jdk1.7.0_80Verifyingthatwecanwriteto/opt/cloudera-manager/cm-5.12.0/etc/cloudera-scm-serverCreatingSCMconfigurationfilein/opt/cloudera-manager/cm-5.12.0/etc/cloudera-scm-serverExecuting: /usr/java/jdk1.7.0_80/bin/java-cp/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera-manager/cm-5.12.0/share/cmf/schema/../lib/*com.cloudera.enterprise.dbutil.DbCommandExecutor/opt/cloudera-manager/cm-5.12.0/etc/cloudera-scm-server/pertiescom.clouderaf.db.[main]DbCommandExecutor INFOSuccessfullyconnectedtodatabase.Alldone,yourSCMdatabaseisconfiguredcorrectly!mysql，增加一條對CMServergrantallonscm.*TO”scm”@”cluster-70”IDENTIFIEDBY”scm”;cluster-70CMServer/opt/cloudera-manager/cm-5.12.0/etc/cloudera-scm-server/perties需要依據(jù)實際狀況更以下參數(shù)。com.clouderaf.db.host=cluster-71com.cloudera=scmcom.clouderaf.db.user=scmcom.clouderaf.db.password=scmcom.clouderaf.db.setupType=EXTERNALCDHParcelcluster-70mkdir-p/opt/cloudera/parcel-repo將以下已經(jīng)預備好的parcel資源文件放入該名目：CDH-5.12.0-1.cdh5.12.0.p0.29-el6.parcelCDH-5.12.0-1.cdh5.12.0.p0.29-el6.parcel.sha1manifest.jsonchown-Rcloudera-scm:cloudera-scm/opt/cloudera/parcel-repo在其它全部節(jié)點上：mkdir-p/opt/cloudera/parcelschowncloudera-scm:cloudera-scm/opt/cloudera/parcelsclouderamanagerserverandagents直接使用root運行啟動腳本即可，腳本中已經(jīng)默認設(shè)定為會切換到cloudera-scm在全部節(jié)點上，先調(diào)整下文件屬主權(quán)限：chown-Rcloudera-scm:cloudera-scm/opt/cloudera-managercluster-70CMServer：/opt/cloudera-manager/cm-5.12.0/etc/init.d/cloudera-scm-serverstartCMagent：/opt/cloudera-manager/cm-5.12.0/etc/init.d/cloudera-scm-agent tart檢查../../log/下面的日志文件，分析錯誤緣由。8ClouderaManagerCDH訪問以下地址：://0:7180/cmf初始治理賬號信息為：admin/admin就選免費版，連續(xù)。66CDHparcels數(shù)配置上的優(yōu)化建議，遵照處理后重做檢驗檢查即可?！币部梢?。進入集群角色安排環(huán)節(jié)，這一步比較關(guān)鍵，留意要把NameNode我們設(shè)計的cluster-70節(jié)點，SecondaryNameNode安排到cluster-71節(jié)點上。留意HDFSDataNodecluster-72到cluster-75這4個cluster-70下一步，進入數(shù)據(jù)庫設(shè)置。按實際配置信息填寫，然后下一步。DataNode/dfs/dn；。這里要特別留意，假設(shè)給數(shù)據(jù)節(jié)點預備了“DataNode/dfs/dn2,盤時，按這樣的規(guī)章掛載到系統(tǒng)中的。在進入下一步之前有一個細節(jié)格外重要，由于我們是使用的獨立的數(shù)CDHHDFSchown-Rh