CEPH分布式存儲部署.

1、CEPH分布式存儲部署PS:本文的所有操作均在 mon節(jié)點的主機進行，如有變動另有注釋 作者：網(wǎng)絡(luò)技術(shù)部徐志權(quán)日期：2014年2月10日更新歷史：2014210:VERSION 1.0首次完成ceph部署文檔，塊設(shè)備及對象存儲的配置隨后添加。部署前網(wǎng)絡(luò)規(guī)劃1.1環(huán)境部署主機名公網(wǎng) IP（ eth0）私網(wǎng) IP（eth1）操作系統(tǒng)運行服務(wù)n ode10101Cen tOS6.5mor、mdsn ode20202Cen tOS6.5osdno de303172.16.100

2、.103Cen tOS6.5osd操作系統(tǒng)使用CentOS6.5，因為系統(tǒng)已經(jīng)包含 xfs的支持可以直接使用不需要再次 編譯。由于CentOS6.5系統(tǒng)的內(nèi)核為2.6.32，因此要關(guān)閉硬盤的寫入緩存，若高于此版本不需要關(guān)閉。#hdparm -W 0 /dev/sdb 0本次部署一共有一個監(jiān)控節(jié)點、一個元數(shù)據(jù)節(jié)點、兩個數(shù)據(jù)節(jié)點，每個數(shù)據(jù)節(jié)點擁 有兩個硬盤作為數(shù)據(jù)盤。1.2網(wǎng)絡(luò)拓撲1.3配置服務(wù)器、安裝 ceph添加ceph的rpm庫key#rpm -import/git/?p=ceph.git;a=blob_pla in; f=keys/release.asc#rp

3、m -import/git/?p=ceph.git;a=blob_pla in; f=keys/autobuild.asc添加 ceph-extras 庫#vi /etc/yum.repos.d/ceph-extrasceph-extrasn ame=Ceph Extras Packagesbaseurl=/packages/ceph-extras/rpm/ce ntos6/$basearchen abled=1priority=2gpgcheck=1type=rpm-mdgpgkey=/git/?p=ce

4、ph.git;a=blob_pla in; f=keys/release.ascceph-extras-no archn ame=Ceph Extras n oarchbaseurl=/packages/ceph-extras/rpm/ce ntos6/noarchen abled=1priority=2gpgcheck=1type=rpm-mdgpgkey=/git/?p=ceph.git;a=blob_pla in; f=keys/release.ascceph-extras-sourcen ame=Ceph Extras Sour

5、cesbaseurl=/packages/ceph-extras/rpm/ce ntos6/SRPMSen abled=1priority=2gpgcheck=1type=rpm-mdgpgkey=/git/?p=ceph.git;a=blob_pla in; f=keys/release.asc 添加ceph庫#rpm -Uvh /rpms/el6/noarch/ceph-release-1-0.el6 .n oarch.rpm 添加epel庫#rpm -Uvhhttp:/dl.fedoraproject.

6、Org/pub/epel/6/x86_64/epel-release-6-8. no arch.rpm 安裝ceph#yum update -y & yum in stall ceph -y二、部署ceph2.1創(chuàng)建一個集群創(chuàng)建一個集群，集群名為ceph，在mon節(jié)點，這里是主機nodel，執(zhí)仃以下操作2.1.1 獲取一個 uuid#uuidge n2.1.2編輯配置文件#vi /etc/ceph/ceph.c onfglobalauth_service_required = cephx#filestore_xattr_use_omap = true#該選項只針對 ext4文件系統(tǒng)格式auth

7、_clie nt_required = cephx auth_cluster_required = cephx auth supported = nonefsid = a500528d-fc02-464e-b944-bddacbb5c3d1#填入剛才獲取到的uuidpublic network = /24cluster network = /24osd pool default size = 2osd pool default min size = 1max ope n files = 65535mon osd full ratio = .95

8、mon osd n earfull ratio = .85mo nmon data = /data/$cluster-$idosdosd jour nal size = 1024osd journal = /data/$ name/jo urnalosd data = /data/$ name2.2創(chuàng)建一個mon2.2.1編輯配置文件，添加mo n參數(shù)#vi /etc/ceph/ceph.c onfmon.no de1host = no de1mon addr = 01:67892.2.2 創(chuàng)建一個 mon keyring#ceph-authtool /etc/cep

9、h/ceph.mon.keyring -create-keyring -gen-key -n mon.2.2.3創(chuàng)建mon數(shù)據(jù)目錄#mkdir -p /data/ceph-no de1#ceph-m on -i no de1 -mkfs -keyri ng /etc/ceph/ceph.m on .keyri ng2.2.4啟動并查看集群狀態(tài)#/etc/i nit.d/ceph start mon.no de1#ceph - s出現(xiàn)以下提示，啟動成功：root no de1 # ceph -scluster a500528d-fc02-464e-b944-bddacbb5c3d1health

10、HEALTH_ERR 192 pgs stuck in active; 192 pgs stuck un clea n; no osds monmap e1: 1 mo ns at n ode1=01:6789/0, electio n epoch 2, quorum 0 no de1osdmap e1: 0 osds: 0 up, 0 inpgmap v2: 192 pgs, 3 pools, 0 bytes data, 0 objects0 kB used, 0 kB / 0 kB avail192 creat ing2.3創(chuàng)建一個osd首先為節(jié)點添加一塊硬盤（這

11、里是主機n ode2）PS:硬件支持熱插拔的情況下可以使用以下命令，在線檢測新增硬盤# echo /sys/class/scsi_host/host2/sca n2.3.1 獲取 osd 的 ID#ceph osd create/記錄得到的編號，該編號是下面創(chuàng)建的osd的ID2.3.2編輯配置文件#vi /etc/ceph/ceph.c onf添加osd.0public addr =02cluster addr=02233同步配置文檔到節(jié)點node2#scp -r root01:/etc/ceph/ root192.1

12、68.100.102:/etc/2.3.4部署osd節(jié)點#ssh root02#parted /dev/sdb mktable gpt#parted /dev/sdb mkpart osd.0 1 20g/新加的硬盤為 20g,并將所有空間劃分為一個分區(qū)#mkfs -t xfs /dev/sdb1#mkdir /data/osd.0#mou nt /dev/sdb1 /data/osd.0#ceph-osd -i 0 -mkfs -mkkey#ceph auth add osd.0 osd allow * mon allow rwx -i /data/osd.0/ke

13、yring#ceph osd crush add osd.0 0.2 root=default host =no de22.3.5啟動osd進程#ceph-osd -i 02.3.6查看進程#ps -ef|grep ceph-osd以下提示啟動成功root no de2 # ps -ef|grep ceph-osdroot136410 08:42 ?00:00:07 ceph-osd -i 02.3.7查看osd狀態(tài)#exit#ceph osd statosd添加成功root no de1 # ceph osd statosdmap e41: 1 osds: 1 up, 1 in2.3.8開機

14、自動啟動#vi /etc/fstab添加/dev/sdb1/data/osd.0xfsdefaults0 0# echo ceph-osd -i 0 /etc/rc.local2.4創(chuàng)建一個mds倉U建mds# ceph-mds -i nodel -n mds. nodel -c /etc/ceph/ceph.co nf -m 01:6789 檢查mds狀態(tài)，成功啟動如下root no de1 # ceph mds state19: 1/1/1 up 0=node仁up:active三、配置ceph客戶端3.1確定操作系統(tǒng)內(nèi)核3.1.1查看內(nèi)核版本#un ame -r

15、若內(nèi)核版本低于 2.6.32時，建議升級到2.6.34或以上3.1.2更新內(nèi)核，編譯ceph模塊這里以升級到3.10.29為例#wget /pub/li nu x/kernel/v3.x/li nux-3.10.29.tar.xz#tar -Jxvf lin ux-3.10.29.tar.xz #cd lin ux-3.10.29#cp /boot/config-2.6.32-431.3.1.el6.x86_64 ./.config將當(dāng)前系統(tǒng)內(nèi)核的配置文件復(fù)制到kernel源文件的目錄內(nèi)#make menuconfig依次選擇下面選項，ceph必須編譯

16、為模塊MFile systems-Network File Systems-Ceph distributed file systemr二ll-i str iuted file s1RPC: Enable dprintk debugging_jgging I FSsupportCaavarTcean e t wo r k保存并退出#make#make modules_ in stall#make in stall確定boot木有下存在以下文件root no de1 lin ux-3.10.29# ll /boot/|grep 3.10ini tramfs-3.10.29.imgSystem.ma

17、p - /boot/System.map-3.10.29System.map-3.10.29vml inuz - /boot/vmli nuz-3.10.29vmli nu z-3.10.29更改 grub.conf#vi /etc/grub.c onf添加title Cen tOS (3.10.29)root (hd0,0)kernel /vmlinuz-3.10.29 ro root=/dev/mapper/V olGroup-lv_root rd_NO_LUKSLANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=V olGroup/lv_swapSYSFONT=lat

18、arcyrheb-su n16 crashkernel=auto rd_LVM_LV=VolGroup/lv_root KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quietini trd /in itramfs-3.10.29.img3.1配置文件目錄共享客戶端mds節(jié)點PS:確定已經(jīng)運行一個 ceph集群，并且集群包含至少一個或以上 掛載ceph分布式存儲文件目錄共享以下在客戶端上操作#yum in stall ceph-fuse -y#ceph-fuse -m 01:6789 /mnt測試寫入1G文件#dd if=/dev

19、/zero of=/mn t/test.1g bs=8192 cou nt=125000四、ceph擴容4.1增加osd節(jié)點 （以下操作在nodel執(zhí)行，即mon節(jié)點）根據(jù)第一章在新的節(jié)點node3服務(wù)器安裝ceph，并配置好環(huán)境4.1.1 獲取 osd 的 ID#ceph osd create/記錄得到的編號，該編號是下面創(chuàng)建的osd的ID4.1.2編輯配置文件#vi /etc/ceph/ceph.c onf添加osd.1public addr =03cluster addr=034.1.3同步配置文檔到節(jié)點node3#scp -r root

20、01:/etc/ceph/ root03:/etc/4.1.4部署osd節(jié)點#ssh root03#parted /dev/sdb mktable gpt#parted /dev/sdb mkpart osd.1 1 20g/新加的硬盤為 20g，并將所有空間劃分為一個分區(qū)#mkfs -t xfs /dev/sdb1#mkdir -p /data/osd.1#mou nt /dev/sdb1 /data/osd.1#ceph-osd -i 1 -mkfs -mkkey#ceph auth add osd.1 osd

21、allow * mon allow rwx -i /data/osd.1/keyring#ceph osd crush add osd.1 0.2 root=default host =no de24.1.5啟動osd進程#ceph-osd -i 14.1.6查看進程#ps -ef|grep ceph-osd以下提示啟動成功root no de3 # ps -ef|grep ceph-osdroot136410 08:42 ?00:00:07 ceph-osd -i 04.1.7查看osd狀態(tài)#exit#ceph osd statosd添加成功root no de1 # ceph osd st

22、at osdmap e41: 2 osds: 2 up, 2 in4.2 增力口 mon （以下操作在主機node2執(zhí)行）PS:每臺主機只能為 ceph集群啟動一個 mon進程PS2：集群中只有兩個mon節(jié)點時，其中一個斷開會導(dǎo)致集群不能訪問，建議mon總數(shù)3個或以上。本次mon將部署在主機node24.2.3創(chuàng)建新的monitor數(shù)據(jù)目錄#mkdir /data/ceph-no de24.2.4部署新的mon#ceph auth get mon. -o /data/ceph-no de2/ceph.m on .keyri ng#ceph mon getmap -o /data/ceph-no

23、 de2/ceph.m on. map#ceph-m on -i no de2 -mkfs -mon map /data/ceph-no de2/ceph.m on. map -keyri ng/data/ceph-no de2/ceph.m on .keyri ng#ceph mon add node2 02:67894.2.1啟動新的mon#ceph-mon -i node2 -public-addr 02:6789五、檢查ceph集群狀態(tài)#ceph health/ceph健康狀態(tài)#ceph status/ceph當(dāng)前全部狀態(tài)#ceph

24、-w/實時監(jiān)控ceph狀態(tài)及變化#ceph osd dump/所有osd詳細狀態(tài)#ceph osd tree/osd所在位置，及狀態(tài)#ceph quorum_status/mon優(yōu)先級狀態(tài)#ceph mon dump/mon節(jié)點狀態(tài)#ceph mds dump/mds詳細狀態(tài)六、Openstack 與 Ceph 整合6.1 整合 openstack 與 ceph6.1.1創(chuàng)建池#ceph osd pool create volumes 128#ceph osd pool create images 128#ceph osd pool create backups 1286.1.2獲取ceph配

25、置文件PS:應(yīng)用于openstack所有節(jié)點#scp -r 3:/etc/ceph /etc6.1.3所有openstack節(jié)點安裝ceph庫文件#yum in stall -y pyth on-ceph ceph6.1.4創(chuàng)建ceph用戶#ceph auth get-or-create clie nt.ci nder mon allow r osd allow class-read object_prefixrbd_childre n, allow rwx pool=volumes, allow rx pool=images#ceph auth get-or-creat

26、e clie nt.gla nee mon allow r osd allow class-read object_prefix rbd_childre n, allow rwx pool=images#ceph auth get-or-create clie nt.ci nder-backup mon allow r osd allow class-read object_prefix rbd_childre n, allow rwx pool=backups6.1.5獲取ceph用戶的key加入到ceph配置文件中PS:應(yīng)用于openstack所有節(jié)點#ceph auth get-or-c

27、reate clie nt.gla nee |tee /etc/ceph/ceph.clie nt.gla nce.keyri ng#chown glance:glance /etc/ceph/ceph.client.glance.keyring#ceph auth get-or-create clie nt.ci nder |tee /etc/ceph/ceph.clie nt.ci nder.keyri ng#chow n cin der:c in der /etc/ceph/ceph.clie nt.ci nder.keyri ng#ceph auth get-or-create cli

28、e nt.ci nder-backup |tee /etc/ceph/ceph.clie nt.c in der-backup.keyri ng#chow n cin der:c in der /etc/ceph/ceph.clie nt.ci nder-backup.keyri ng6.1.6配置計算節(jié)點的 nova-compute使用ceph PS：應(yīng)用于所有計算節(jié)點#ceph auth get-key clie nt.ci nder | tee clie nt.ci nder.key#uuidge n3a37c634-3b97-698c-c0b0-cb6f01fa4533#cat sec

29、ret.xml EOF3a37c634-3b97-698c-c0b0-cb6f01fa4533clie nt.ci nder secret/nameEOF#virsh secret-define -file secret.xmlSecret 3a37c634-3b97-698c-c0b0-cb6f01fa4533 created將上述所得 uuid值保存至/etc/nova/nova.conf rbd_secret_uuid=3a37c634-3b97-698c-c0b0-cb6f01fa4533#virsh secret-set-value -secret 3a37c634-3b97-698c-c0b0-cb6f01fa4533 -base64 $(cat clie nt.ci nder.key)6.1.7glance、cinder、nova 的 ceph 酉己置Glance:default_store=rbd rbd_store_user=gla nee rbd_store_pool=images show image direct url=True Cin der:volume_