GPFS配置手冊(cè)

1、江西省農(nóng)信社GPFS安裝手冊(cè)V1.0目錄1.江西省農(nóng)信社GPFS規(guī)劃21.1江西省農(nóng)信社GPFS集群規(guī)劃21.2集群Quorum方式31.3NSD磁盤(pán)組成31.4文件系統(tǒng)基本信息31.5GPFS系統(tǒng)環(huán)境示意圖52.GPFS安裝前準(zhǔn)備52.1安裝介質(zhì)及補(bǔ)丁準(zhǔn)備52.2Openssl及Openssh62.3Linux節(jié)點(diǎn)安裝ksh62.4Linux節(jié)點(diǎn)修改網(wǎng)絡(luò)參數(shù)63.GPFS安裝配置73.1配置ssh信任關(guān)系73.2AIX安裝GPFS軟件103.3Linux安裝GPFS軟件113.4GPFS集群配置124.GPFS日常維護(hù)194.1GPFS啟動(dòng)194.2GPFS停止194.3查看GPFS使用狀況

2、201. 江西省農(nóng)信社GPFS規(guī)劃1.1 江西省農(nóng)信社GPFS集群規(guī)劃IP地址節(jié)點(diǎn)名操作系統(tǒng)集群角色8AIX 7100-02-02-1316QAIX 7100-02-02-1316QRedhat 6.4 for PRedhat 6.4 for Powerclient10.0.147

3、.51Redhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for Powerclient6gpfs56.abs.j

4、Redhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat

5、6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for Powercl

6、Redhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for Powerclient2

7、Redhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for Powerclient7gpfs77.abs.jxnx

8、Redhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4

9、 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for Powerclien

10、Redhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for Powerclient3gpf

11、Redhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for PRedhat 6.4 for Powerclient1.2 集群Quorum方式集群Quorum方式采用Node Quorum + Tiebreaker disks,兩個(gè)AIX節(jié)點(diǎn)加上一塊專(zhuān)用的SAN磁盤(pán),目前只有一個(gè)AIX

12、節(jié)點(diǎn),第二個(gè)AIX節(jié)點(diǎn)和SAN磁盤(pán)需要后續(xù)添加。1.3 NSD磁盤(pán)組成PV_NAMESIZENSD_NAMEFS_NAMENSD SERVERS LISThdisk2500Gbjxrccb_hdisk3500Gbjxrccb_1.4 文件系統(tǒng)基本信息1.5 GPFS系統(tǒng)環(huán)境示意圖GPFS Server: 2個(gè)AIX分區(qū)連接V7000SAN Storage: SVCGPFS Client: 48個(gè)Power Linux分區(qū)GPFS管理網(wǎng)絡(luò): 千兆交換機(jī)2. GPFS安裝前準(zhǔn)備2.1 安

13、裝介質(zhì)及補(bǔ)丁準(zhǔn)備安裝包:GPFS v補(bǔ)丁：GPFS-12.2 Openssl及Openssh確認(rèn)AIX上openssl及openssh是否已安裝,Linux默認(rèn)已經(jīng)安裝;lslpp -l |grep open2.3 Linux節(jié)點(diǎn)安裝ksh#rpm -ivh ksh-20100621-19.el6.ppc64.rpm2.4 Linux節(jié)點(diǎn)修改網(wǎng)絡(luò)參數(shù)修改/etc/sysctl.conf文件,需修改的網(wǎng)絡(luò)參數(shù)如下:# increase Linux TCP buffer limitsnet.core.rmem_max = 8388608net.core.wmem_max

14、 = 8388608# increase default and maximum Linux TCP buffer sizesnet.ipv4.tcp_rmem = 4096 262144 8388608net.ipv4.tcp_wmem = 4096 262144 8388608# increase max backlog to avoid dropped dev_max_backlog = 2500修改完畢后執(zhí)行sysctl -p /etc/sysctl.conf讓配置生效3. GPFS安裝配置3.1 配置ssh信任關(guān)系n 在節(jié)點(diǎn)gpfs98.abs.

15、上生成并配置ssh密鑰對(duì)#ssh-keygen -t rsa# cd .ssh# lsid_rsa id_rsa.pub# cat id_rsa.pub authorized_keys# chmod 600 authorized_keys# echo StrictHostKeyChecking no config# chmod 600 confign 修改各節(jié)點(diǎn)的/etc/hosts文件,加入對(duì)集群中各節(jié)點(diǎn)的別名和ip地址9 0 1 gp

16、2 3 4 5 6 7 8 9 0 gpfs60.abs.jxn

17、1 2 3 4 5 6 7 8 9 10.0.1

18、47.70 1 2 3 4 5 6 7 8 9 gpfs79

19、.0 1 2 3 4 5 6 7 8 gpfs88.abs.jxnxs.c

20、om9 0 1 2 3 4 5 6 7

21、8 n 驗(yàn)證本機(jī)無(wú)需密碼的ssh登錄#ssh hostnamen 將.ssh目錄復(fù)制到其他各節(jié)點(diǎn)#scp -r /.ssh :/root/#scp -r /.ssh :/root/#scp -r /.ssh :/root/#scp -r /.ssh :/root/驗(yàn)證各節(jié)點(diǎn)間無(wú)密碼的ssh登錄3.2 AIX安裝GPFS軟件1） gpfs3.5.0安裝包放在/home/gpfsaix/iso目錄下2

22、）切換/home/gpfsaix/iso目錄下,通過(guò)smitty installp安裝GPFS軟件3) 安裝補(bǔ)丁gpfs3.5版本，需要升級(jí)到1以上，補(bǔ)丁下載地址：/support/fixcentral/swg/selectFixes?parent=Cluster+software&product=ibm/power/IBM+General+Parallel+File+System&release=3.5.0&platform=AIX&function=all下載后上傳到服務(wù)器后解壓：gzip -d -c filename.tar.gz

23、|tar -xvf -使用smitty update_all安裝補(bǔ)丁4）安裝后檢查#lslpp -l|grep gpfs如果之前已經(jīng)創(chuàng)建了cluster，在所有node都down的狀態(tài)下，加載GPFS版本到最新版本：mmchconfig release=LATEST;如果是新裝的GPFS,則可以忽略此步。3.3 Linux安裝GPFS軟件1) 上傳安裝介質(zhì)安裝文件位于/home/gpfsrpm/，補(bǔ)丁文件位于/home/gpfsrpm/updte；2) 安裝GPFS軟件在安裝介質(zhì)所在目錄下執(zhí)行：#rpm -ivh gpfs*.rpm3) 升級(jí)GPFS軟件版本在補(bǔ)丁文件所在目錄下執(zhí)行：#rpm

24、-Uvh gpfs*.rpm4) 檢查安裝結(jié)果#rpm -qa |grep gpfs5) Building the GPFS portability layer#cd /usr/lpp/mmfs/src#make Autoconfig#make World#make InstallImages3.4 GPFS集群配置n 修改系統(tǒng)環(huán)境變量AIX平臺(tái)下修改/etc/environment在PATH后增加/usr/lpp/mmfs/bin#vi /etc/environmentLinux平臺(tái)下修改/root/.bashprofile,在PATH后增加/usr/lpp/mmfs/binn 創(chuàng)建集群按照

25、規(guī)劃，本次集群中包含1個(gè)AIX節(jié)點(diǎn)和48個(gè)Linux client；首先在1個(gè)AIX節(jié)點(diǎn)平臺(tái)創(chuàng)建GPFS集群創(chuàng)建集群定義文件#cd /home#mkdir gpfs#cd gpfs#vi jxrcc_cluster8:quorum-manager a) 創(chuàng)建GPFS集群#mmcrcluster N jxrccb_cluster p 8 -r /usr/bin/ssh -R /usr/bin/scp -C jxrccb_cluster A# mmcrcluster -N jxrccb_cluster -p 8 -r /usr/bin/s

26、sh -R /usr/bin/scp -C jxrccb_cluster -AWarning: Permanently added (RSA) to the list of known hosts.Tue Nov 19 20:01:38 CST 2013: 6027-1664 mmcrcluster: Processing node mmcrcluster: Command successfully completedmmcrcluster: 6027-1254 Warning: Not all nodes ha

27、ve proper GPFS license designations. Use the mmchlicense command to designate licenses as needed.b) 配置許可# mmchlicense server -accept -N 8# mmchlicense server -accept -N 8The following nodes will be designated as possessing GPFS server licenses: mmchlicense: Co

28、mmand successfully completedc) 添加Linux節(jié)點(diǎn)通過(guò)如下命令依次添加48個(gè)Linux節(jié)點(diǎn)：# mmaddnode N # mmchlicense server -accept -N # mmaddnode N # mmchlicense server -accept -N 查看集群狀態(tài)#mmlsclusterd) 修改集群參數(shù)#mmchconfig prefetchThreads=1000 N 1

29、8# mmchconfig pagepool=4G N 8# mmchconfig failureDetectionTime=15# mmchconfig MaxMBps=4096e) 啟動(dòng)集群#mmstartup -a啟動(dòng)后，查看集群狀態(tài)#mmgetstate -a確保所有節(jié)點(diǎn)的狀態(tài)為Activen 創(chuàng)建NSD磁盤(pán)a) 修改fscsi參數(shù)在所有直接訪問(wèn)SAN盤(pán)的節(jié)點(diǎn)上執(zhí)行#lsdev |grep fscsi|awk print chdev -l,$1,-a dyntrk=yes -a fc_err_recov=fast_fail|shb) 識(shí)別外置存儲(chǔ)磁

30、盤(pán)在所有直接訪問(wèn)SAN盤(pán)的節(jié)點(diǎn)上執(zhí)行#cfgmgrc) 修改hdisk參數(shù)在所有直接訪問(wèn)SAN盤(pán)的節(jié)點(diǎn)上執(zhí)行#lsdev -Cc disk|grep MPIO |awk print chdev -l,$1,-a algorithm=round_robin -a queue_depth=8 -a reserve_policy=no_reserve|shd) 創(chuàng)建NSD描述文件#cd /gpfs#vi nsd.confhdisk2: 8:jxrccb_nsd1:hdisk3: 8:jxrccb_nsd2:備注文件格式：DiskName:ServerList:

31、DiskUsage:FailureGroup:DesiredName:StoragePool在本集群中，將AIX節(jié)點(diǎn)定義為NSD Server，如果NSD較多可以將每塊盤(pán)的首選NSD Server設(shè)置為不同，以均衡NSD Sever的負(fù)載；e) 生成NSD盤(pán)#mmcrnsd -F nsd.conff) 查看NSD盤(pán)#mmlsnsd# mmlsnsd File system Disk name NSD servers - (free disk) jxrccb_nsd1 (free disk) jxrccb_nsd2 gpfs98.abs.jxnxs.co

32、m n 定義Tiebreaker disks集群處于Down狀態(tài)時(shí)才能定義；#mmshutdown -a# mmchconfig tiebreakerDisks= 定義后可用如下命令查看；# mmlsconfign 創(chuàng)建GPFS文件系統(tǒng)#mmcrfs gpfs -F nsd.conf -A yes -B 512K -T /gpfs# mmcrfs gpfs -F nsd.conf -A yes -B 512K -T /gpfsGPFS: 6027-531 The following disks of gpfs will be formatted on node ABSGPFS1: jxrccb_nsd1: size 536870912 KB jxrccb_nsd2: size 536870912 KBGPFS: 6027-540 Formatting file system .GPFS: 6027-535 Disks up to size 4.4 TB can be added to storage pool system.Creating Inode FileCreating Allocation MapsCreating Log FilesClearing In