GPFS雙活數(shù)據(jù)中心建設(shè)方案

1、IBM ATS, China IBM CorporationPower GPFS Active-Active SolutionATS: Ni Gang第1頁 雙活數(shù)據(jù)中心思索 GPFS雙活處理方案及實(shí)施 客戶案例分享 內(nèi)容提要 GPFS雙活數(shù)據(jù)中心處理方案2第2頁 1雙活數(shù)據(jù)中心思索雙活數(shù)據(jù)中心需求“0”切換高資源利用率雙活數(shù)據(jù)中心技術(shù)實(shí)現(xiàn)條件基礎(chǔ)架構(gòu)要求第3頁4關(guān)鍵業(yè)務(wù)運(yùn)行依賴信息系統(tǒng)Gartner and DRJ BCM/DR survey各行業(yè)關(guān)鍵業(yè)務(wù)RTO各行業(yè)宕機(jī)損失商業(yè)運(yùn)行每小時營收30秒數(shù)據(jù)價值制造業(yè)$1,610,000 $13,417 金融$1,495,000 $12,458

2、零售$1,107,000 $9,225 制藥$1,082,000 $9,017 醫(yī)療$636,000 $5,300 公用事業(yè)$643,000 $5,358 天災(zāi)人禍難以防止災(zāi)備市場 & 增加率IDC中國業(yè)務(wù)連續(xù)性與災(zāi)難恢復(fù)市場-預(yù)測與分析第4頁5恢復(fù)時間目標(biāo)15 Min.1-4 Hr.4 -8 Hr.8-12 Hr.12-16 Hr.24 Hr.Days成本/價值BC Tier 4 快照復(fù)制， 用于備份/恢復(fù)BC Tier 3 磁帶庫，電子VaultingBC Tier 2 熱備站點(diǎn)，磁帶恢復(fù)BC Tier 7 服務(wù)器或者存放復(fù)制，且端到端服務(wù)器自動恢復(fù)BC Tier 6 實(shí)時連續(xù)數(shù)據(jù)復(fù)制，服

3、務(wù)器或者存放BC Tier 1 磁帶恢復(fù)從磁盤恢復(fù)從磁帶拷貝恢復(fù)數(shù)據(jù)容災(zāi)級別（BC Tier）恢復(fù)時間目標(biāo) RTO需要均衡成本與性能客戶目標(biāo): 均衡 恢復(fù)時間 和 合理成本BC Tier 5 應(yīng)用/數(shù)據(jù)庫級復(fù)制和集成RTO=0 & RPO=0 雙活數(shù)據(jù)中心能提供更高BC Tier第5頁雙活系統(tǒng)中心”0”切換當(dāng)前非雙活系統(tǒng)中心模式”0”切換當(dāng)?shù)馗呖捎孟到y(tǒng)異地容災(zāi)系統(tǒng)應(yīng)急保障系統(tǒng)雙活系統(tǒng)中心=“0”切換系統(tǒng)切換風(fēng)險和成本用戶滿意度運(yùn)行/KPI考評應(yīng)急情況處理能力技術(shù)創(chuàng)新點(diǎn)用戶滿意度運(yùn)行/KPI考評 雙活系統(tǒng)“0”切換運(yùn)行風(fēng)險第6頁雙活系統(tǒng)中心高資源利用率ServerSAN存放Active Site

4、SAN存放Active Site not standbyServer第7頁Fabric AFabric B支持跨站點(diǎn)集群功效應(yīng)用存放虛擬化，實(shí)現(xiàn)跨站點(diǎn)數(shù)據(jù)讀寫和數(shù)據(jù)冗余雙活數(shù)據(jù)中心實(shí)現(xiàn)條件8第8頁設(shè)計雙活數(shù)據(jù)中心考慮原因不論客戶或者廠家實(shí)施雙活數(shù)據(jù)中心都必須處理以下挑戰(zhàn)：雙活數(shù)據(jù)中心之延遲和穩(wěn)定性:因?yàn)楣馑傧拗?，?100km 所產(chǎn)生數(shù)據(jù)往返延遲約為1ms。該延遲已經(jīng)跟普通硬盤讀寫可比。所以，會對實(shí)際應(yīng)用性能組成影響, 尤其是兩數(shù)據(jù)中心數(shù)據(jù)交互親密業(yè)務(wù)。另外，數(shù)據(jù)中心之間網(wǎng)絡(luò)更難以維護(hù)和掌控。Quorum / Tie-Breaker 之需求:為了防止雙活數(shù)據(jù)中心產(chǎn)生腦裂(Split Brain

5、)或場地分割(site isolation) 情況，處理方案需要提供有效Quorum / Tie-Breaker方式來確保數(shù)據(jù)完整性.工作負(fù)載之考慮:業(yè)務(wù)交易中，應(yīng)用所產(chǎn)生之寫操作 (INSERT, UPDATE, DELETE)百分比越高，則越多數(shù)據(jù)需要跨數(shù)據(jù)中心傳送。這類型業(yè)務(wù)交易不利于雙活數(shù)據(jù)中心設(shè)計。推薦業(yè)務(wù)劃分，讀寫分離等有效躲避數(shù)據(jù)中心間交互架構(gòu)。9第9頁IBM GPFS雙活處理方案GPFS相關(guān)技術(shù)介紹GPFS雙活+Oracle RACGPFS雙活+DB2 Purescale(GDPC)與EMC VPLEX競爭分析 2第10頁11storage networkIP network

6、并行文件系統(tǒng) shared disk storage/DB1_data/user_homeclients server cluster softwarecluster softwarecluster software/DB1_data/user_home/DB1_data/user_home/DB1_data/user_home第11頁IBM ConfidentialIBM GPFSGeneral Parallel File System (GPFS) 悠久歷史1998HPCGPFS General File ServingStandardsPortable operating system

7、interface (POSIX) semantics-Large blockDirectory and Small file perfData managementVirtual Tape Server (VTS)Linux Clusters (Multiple architectures)IBM AIX Loose ClustersGPFS 2.1-2.3HPCResearchVisualizationDigital MediaSeismicWeather explorationLife sciences32 bit /64 bitInter-op (IBM AIX & Linux)GPF

8、S MulticlusterGPFS over wide area networks (WAN)Large scale clustersthousands of nodesGPFS 3.1-3.2First called GPFSGPFS 3.4Enhanced Windows cluster support- Homogenous Windows ServerPerformance and scaling improvementsEnhanced migration and diagnostics support GPFS 3.3Restricted Admin FunctionsImpro

9、ved installationNew license modelImproved snapshot andbackupImproved ILM policy engine Ease of administrationMultiple-networks/ RDMADistributed Token ManagementWindows Multiple NSD serversNFS v4 SupportSmall file performanceInformation lifecycle management (ILM)Storage PoolsFile setsPolicy EngineGPF

10、S 3.5Active File ManagementGPFS Native RAIDGPFS Shared Nothing Cluster12第12頁 | GPFS是基于共享存放分布式、高性能集群文件系統(tǒng)存放Server網(wǎng)絡(luò)集群內(nèi)可多達(dá)8192個節(jié)點(diǎn)，256個FS支持多平臺PowerPowerX86IntaniumX86AIXAIXRedhat/SUSEWindowsWindows支持異構(gòu)存放（IBM DS Sotrage / IBM XIV / EMC DMX ）實(shí)測環(huán)境中，文件系統(tǒng)容量高達(dá) 4 PB網(wǎng)絡(luò)支持EthernetInfinBandSANSAN能夠無限擴(kuò)展在線靈活擴(kuò)展第13頁支持

11、存放產(chǎn)品In addition to IBM Storage, IBM GeneralParallel File System (GPFS) supportsstorage hardware from these vendors:GPFS supports many storage systems, and the IBM support team can help customers using storage hardware solutions not on this list of tested devices.EMCHitachiHewlett PackardDDN第14頁GPFS處

12、理方案GPFS is certified for RACon/solutions/sap/doc/content/landingdtw/1605307130.html?g_type=pspotPowered by GPFSIBM Virtualization Engine TS7700TS7510 Virtualization EngineVirtual Tape Server Model B10Virtual Tape Server Model B20IBM family ofVirtualization Tape Systems GPFS insideSAS Grid is support

13、ed on GPFSGPFS V9.8 & GPFS第15頁NSD ServerNSD ServerSANNSD ClientAppNSD ClientAppNSD ClientAppNSD ClientAppNSD ClientAppNSD ClientAppNSD ClientAppNSD ClientAppNSD ClientAppNSD ServerNSD Server僅NSD服務(wù)器 連接SAN存放盤，NSD Client經(jīng)過LAN連接Server全部節(jié)點(diǎn)連接SAN存放盤，NSD Server*NSD ServerApp*NSD ServerApp*NSD ServerApp*NSD

14、ServerAppSAN*Optionally configured to enhance availabilityGPFS存放管理-NSD（Network shared disk）GPFS中全部Disk定義成NSD，是否直連SAN存放是區(qū)分NSD Server和Client標(biāo)志NSD即GPFS中集群用于描述各Disk全局唯一名字一個節(jié)點(diǎn)上定義則全部節(jié)點(diǎn)可用NSDs能夠在各種存放設(shè)備類型上創(chuàng)建(hdisk,vpath,Emcpower,Lv)第16頁IBM ConfidentialFailure Group:在一個Storage pool中，經(jīng)過不一樣failure groups對pool內(nèi)D

15、isks進(jìn)行分離;文件復(fù)制時，兩份數(shù)據(jù)自動分布到兩個不一樣group元數(shù)據(jù)同時及數(shù)據(jù)復(fù)制預(yù)防存放故障引發(fā)數(shù)據(jù)高可用支持定義失效組，預(yù)防存放單點(diǎn)故障（如數(shù)據(jù)位于同一存放陣列）失效組數(shù)據(jù)自動同時多至2份拷貝支持復(fù)制整個文件系統(tǒng)，一組文件或單個文件失效組恢復(fù)日志自動同時支持Local 讀支持異地容災(zāi) 支持跨域GPFS復(fù)制 支持同時Metro Mirror支持異步Global Mirror17GPFSGPFS存放管理-Failure Group第17頁IBM ConfidentialReplication:復(fù)制，當(dāng)GPFS復(fù)制功效啟用時,GPFS文件系統(tǒng)數(shù)據(jù)和元數(shù)據(jù)將被保留兩個分布在不一樣失效組副本,

16、當(dāng)其中一個副本不可用時,另一個副本將使文件系統(tǒng)保持可用.18GPFS Replication over failure groupsInode, indirect block, and/or data blocks may be replicatedEach disk address: List of pointers to replicasEach pointer: disk id + sector no.GPFS文件系統(tǒng)寫數(shù)據(jù)時候會同時寫入不一樣失效組內(nèi)盤中, 這么就實(shí)現(xiàn)了數(shù)據(jù)高可用. 數(shù)據(jù)類型能夠包含結(jié)構(gòu)化及非結(jié)構(gòu)化數(shù)據(jù).另外GPFS支持當(dāng)?shù)刈x, 這么對于讀操作, GPFS優(yōu)先選擇本數(shù)據(jù)

17、中心存放進(jìn)行讀取, 防止了因讀取遠(yuǎn)端存放而引發(fā)性能損失.第18頁此quorum機(jī)制是文件系統(tǒng)關(guān)鍵仲裁,和failure group一起保障了存放層高可用.創(chuàng)建時: 若有個不一樣failure groups,就會創(chuàng)建份復(fù)制若有個不一樣Disk,就會創(chuàng)建份復(fù)制若只有個或個Disk,則每個Disk上創(chuàng)建一份復(fù)制這些Disk中必需超出半數(shù)復(fù)制可用來維持文件系統(tǒng)操作可經(jīng)過創(chuàng)建descOnly NSD增加mmlsdisk device -L remark field: desc or excl Disk Descriptor QuorumNode 1Node 2File System Descriptor

18、 Quorum第19頁IBM ConfidentialQuorum NodeNon-Quorum NodePrimary Configuration ServerSecondary Configuration ServerIP NetworkSANSANGPFS Synchronous ReplicationPrimary SiteSecondary SiteTie-Breaker SiteSingle GPFS Cluster20Active-ActiveGPFS Active-Active Solution第20頁 | IP networkSite ASite BSite C (tiebr

19、eaker)Shared disk access (SAN / NSD)Site A storage(failure group 1)Sample disk descriptor listgpfs1nsd:siteAnode1:dataAndMetadata:1gpfs2nsd:siteAnode1:dataAndMetadata:1gpfs3nsd:siteAnode3:dataAndMetadata:1gpfs4nsd:siteAnode4:dataAndMetadata:1gpfs5nsd:siteBnode1:dataAndMetadata:2gpfs6nsd:siteBnode2:d

20、ataAndMetadata:2gpfs7nsd:siteBnode3:dataAndMetadata:2gpfs8nsd:siteBnode4:dataAndMetadata:2gpfs9nsd:siteCnode1:descOnly:3Site B storage (failure group 2)non-quorum nodequorum nodeprimary cluster configuration data serversecondary cluster configuration data server仲裁站點(diǎn)(C) 用于確保一個數(shù)據(jù)中心出現(xiàn)故障時，維持整個cluster可用性

21、Holds an additional quorum node Holds a “file system descriptor-only” (descOnly) disk in a third failure groupSingle replicated GPFS file system: mmcrfs /gpfs/fs0 fs0 F diskDescList -r 2 -R 2 -m 2 -M 2GPFS Active-Active Solution Configuration第21頁 22IBM ConfidentialGPFS Cluster心跳機(jī)制第22頁 23IBM Confiden

22、tialGPFS Cluster心跳相關(guān)參數(shù) leaseDurationTime a disk-lease is granted from CfgMgr to any node.Default: 35 seconds leaseRecoveryWaitAdditional time to wait in case a node failed to renew the lease. The purpose allows for I/Os that were started just before the lease ran out to finish first.Default: 35 seco

23、nds PingPeriodTime period between two pings (second)Default: 2 seconds TotalPingTimeoutTotal amount of time pinging a node before giving up (second)Default: 120 seconds第23頁 24IBM ConfidentialGPFS Persistent Reserve Function Persistent Reserve: Persistent Reserve refers to a set of Small Computer Sys

24、tems Interface-3 (SCSI-3) standard commands and command options. These PR commands and command options give SCSI initiators the ability to establish, preempt, query, and reset a reservation policy with a specified target disk. The functions provided by PR commands are a superset of current reserve a

25、nd release mechanisms. These functions are not compatible with legacy reserve and release mechanisms. Target disks can only support reservations from either the legacy mechanisms or the current mechanisms.Note: GPFS manages reserve_policy and PR_key_value using reserve_policy=PR_shared when Persiste

26、nt Reserve support is enabled and reserve_policy=no_reserve when Persistent Reserve is disabled.第24頁 GPFS Persistent Reserve HW Support List 25IBM Confidential第25頁 26IBM ConfidentialGPFS Local Read Configuration Local Read: 在GPFS Replication環(huán)境下，能夠經(jīng)過指定NSD優(yōu)先級來實(shí)現(xiàn)雙數(shù)據(jù)中心當(dāng)?shù)刈x需求.普通情況下，GPFS優(yōu)先使用SAN提供IO服務(wù). # mm

27、fsadm dump disk | grep relAcctime dtype nsd, directly attached relAcctime 0 dtype nsd, directly attached relAcctime 1 dtype nsd, LAN attached relAcctime 2 默認(rèn)下，SAN Attached 優(yōu)先級為0，LAN Attached 優(yōu)先級為2. Command: mmfsadm test diskrelacc oradata nsd1 1 #修改遠(yuǎn)端數(shù)據(jù)中心nsd1優(yōu)先級為1.第26頁IBM ConfidentialIP NetworkSANSA

28、NGPFS Synchronous ReplicationPrimary SiteSecondry SiteTie-Breaker Site Single GPFS Cluster27More Requirement 兩地三中心IP NetworkSANMetro or Global MirrorBackup SiteBackup GPFS Cluster第27頁IBM Confidential RAC Public NetworkRAC01RAC02IP networkSAN SWITCHSAN SWITCHDWDNDWDNGPFS Cluster using Replication Pri

29、vate networkActive Site AActive Site BTie-Breaker Site CGroup 1PrimaryGroup 2Group 2Primary Group 1GPFS AA + Oracle Extended RAC28第28頁GPFS for Oracle RAC 11g on AIXThe GPFS v3.2 Concepts, Planning and Installation Guide contains a “GPFS use with Oracle” chapter that contains Oracle configuration and

30、 tuning considerations.Oracle MetaLink Note 302806.1 contains a “GPFS Tuning Requirements for Oracle” section which describes the latest tuning information.IBM Whitepaper: Oracle Database 11g and Oracle RAC 11g on IBM AIX Tips and Considerations/support/techdocsGPFS is the only IBM file system certi

31、fied for RACon29第29頁Oracle RAC & GPFSRAC ON GPFS 基本配置GPFS當(dāng)?shù)貜?fù)制,增加RAC可用性GPFS經(jīng)過SAN或IP異地復(fù)制方案為RAC提供災(zāi)備處理方案第30頁Oracle RAC仲裁: 31IBM ConfidentialOracle RAC仲裁機(jī)制: OCSSD 這個進(jìn)程是Oracle Clusterware中最關(guān)鍵進(jìn)程，假如這個進(jìn)程出現(xiàn)異常，會造成系統(tǒng)重啟，這個進(jìn)程提供CSS(Cluster Synchronization Service)服務(wù)。 CSS 服務(wù)經(jīng)過各種心跳機(jī)制實(shí)時監(jiān)控集群狀態(tài)，提供腦裂保護(hù)等基礎(chǔ)集群服務(wù)功效. CSS 服務(wù)有

32、2種心跳機(jī)制： 一個是經(jīng)過私有網(wǎng)絡(luò)Network Heartbeat，另一個是經(jīng)過Voting DiskDisk Heartbeat.這2種心跳都有最大延時，對于Disk Heartbeat， 這個延時叫作IOT (I/O Timeout);對于Network Heartbeat, 這個延時叫MC(Misscount);重啟起最大允許被驅(qū)逐次數(shù)reboottime默認(rèn)為3次，Disk timeout 默認(rèn)200秒，Misscount默認(rèn)值:第31頁Oracle RAC仲裁和GPFS集群仲裁: 32IBM ConfidentialOracle不提議使用共享文件系統(tǒng)來保留Voting disk.

33、實(shí)施中提議取兩臺存放raw device做為voting1和voting2， 仲裁站點(diǎn)仲裁Sever創(chuàng)建NFS為做voting3.在容災(zāi)場景中，GPFS 恢復(fù)時間應(yīng)該小于Oracle RACtimeout.GPFS使用網(wǎng)絡(luò)做為整個IO集群心跳，OraclePrivate Network也對網(wǎng)絡(luò)有很高要求，提議分別布署dedicated network.第32頁DB2 PureScale 處理方案 Geographically-Dispersed PureScale Cluster (GDPC)33Mbr1Mbr3Mbr2Mbr4CFsecCFpriInfiniband switch Datab

34、ase Log Log Log LogEthernetInfinibandFiber ChannelGPFS Shared Storage (logical view)Physical StorageSingle-sitepureScaleconfigurationGDPCpureScaleconfigurationN kmIB range extenderMbr1Mbr3Mbr2Mbr4CFsecCFpriDark Fiber or WAN Database Log Log Log LogFiber ChannelGPFS Shared Storage (logical view)Physi

35、cal Storage site APhysical Storage site BInfiniband + IPoIBGPFS replication between sites3rd Site tie-breakerEthernet第33頁 LAN BackboneSANswitchesHeterogeneous storagesystemOracle RAC cluster(primary)Node 1RAC interconnectLAN BackboneNode 2Mirror A (active)Mirror B (active)VPLEX interconnect(dark fib

36、re)Primary I/O pathPrimary I/O pathNote:No Mirror I/O path requiredVPLEXClustersVPLEXWitness(virtual machine)Dedicated IP linkDedicated IP link“Identical” virtual volumes(actually the same volumes)Physical volumesEMC VPLEX雙活數(shù)據(jù)中心方案 第34頁經(jīng)過VPLEX Metro提供分布式聯(lián)合能力，擴(kuò)展了同時距離內(nèi)兩個位置間訪問能力。VPLEX Metro 利用AccessAnyw

37、here支持跨遠(yuǎn)距離共享、訪問單個數(shù)據(jù)拷貝。實(shí)現(xiàn)主備主機(jī)同時訪問同一套卷愿望，也就是所謂“雙活”架構(gòu)，35EMC VPLEX雙活數(shù)據(jù)中心方案 第35頁IBM GPFS VS EMC VPLEX指標(biāo)對比指標(biāo)EMC VPLEXIBM GPFS應(yīng)用案例5月才公布早在2.X版本就支持AA架構(gòu) ()支持盤陣與主機(jī)類型有限（IBM i系列不支持）絕大多數(shù)廠商都支持物理安裝全部設(shè)備必須安裝在同一EMC機(jī)柜能夠分開安裝在任何標(biāo)準(zhǔn)機(jī)柜最大虛擬卷數(shù)量每個集群8000個只受OS限制和存放限制最大虛擬卷大小96TB只受OS限制和存放限制Point-in-time copy不支持有snapshot功效遠(yuǎn)程異步復(fù)制不支持

38、不支持寫緩存不支持（只支持讀緩存）GPFS本身提供寫緩存(Oracle使用DirectIO)copies之間在線復(fù)制/同時支持支持是否支持向兩地三中心擴(kuò)展使用存放異步復(fù)制實(shí)現(xiàn)使用存放異步復(fù)制實(shí)現(xiàn)數(shù)據(jù)不一樣時后是否支持增量同時不支持支持當(dāng)?shù)刈x支持支持價格低，性價比高36第36頁客戶案例分享交通銀行(GDPC)山東移動(win)渤海銀行(ongoing)武漢社保(ongoing) 3第37頁交通銀行GDPC項(xiàng)目 - 項(xiàng)目需求項(xiàng)目目標(biāo)把試點(diǎn)應(yīng)用改造為在雙中心同時處理雙中心運(yùn)行時業(yè)務(wù)交易性能控制到可接收范圍內(nèi)恢復(fù)時間目標(biāo)（RTO）為分鐘級恢復(fù)點(diǎn)目標(biāo)（RPO）需趨近于0設(shè)計方案不要求應(yīng)用有太多改動試點(diǎn)考

39、查應(yīng)用系統(tǒng)一套網(wǎng)銀支付類應(yīng)用一套貸款流程管理類應(yīng)用依據(jù)實(shí)際效果推廣 381/2 of clientconnections1/2 of clientconnectionsClientAppsClientAppsClientAppsClientAppsClientAppsClientAppsM1M3M2M4CFSCFPSite ASite B70 kmConnections balanced第38頁生產(chǎn)中心和同城副生產(chǎn)中心之間光纖距離70公里客戶與其合作搭檔了解 DataSharing/z原理實(shí)現(xiàn)了MultiSite DataSharing，采取讀寫分離等辦法到達(dá)了期望性能年部分實(shí)施了GDPS客戶

40、開放平臺試點(diǎn)應(yīng)用改造前采取了WAS Cluster, DB2 ESE +HA CMP和EMC SRDF. 客戶選擇了GDPC參考架構(gòu)作為實(shí)現(xiàn)項(xiàng)目目標(biāo)基礎(chǔ)交通銀行GDPC項(xiàng)目 - 項(xiàng)目背景39GDPCActive-Active第39頁故障類型場景應(yīng)用故障轉(zhuǎn)移時間提供業(yè)務(wù)連續(xù)性Member服務(wù)器故障重啟約3秒非故障Member DB2進(jìn)程瓦解約5秒非故障MemberInfiniband網(wǎng)絡(luò)故障約2秒非故障Member以太網(wǎng)故障(非GPFS心跳網(wǎng))約9秒非故障MemberGPFS心跳網(wǎng)絡(luò)故障約38秒非故障Member暫掛約18秒后繼續(xù)服務(wù)主CF服務(wù)器故障重啟應(yīng)用幾乎不受影響全部MemberCF進(jìn)程瓦

41、解應(yīng)用幾乎不受影響全部MemberInfiniband網(wǎng)絡(luò)故障應(yīng)用幾乎不受影響全部Member以太網(wǎng)故障(非GPFS心跳網(wǎng))應(yīng)用幾乎不受影響全部MemberGPFS心跳網(wǎng)絡(luò)故障應(yīng)用幾乎不受影響全部Member交通銀行GDPC項(xiàng)目 實(shí)測結(jié)果40切換測試第40頁山東移動Oracle雙活項(xiàng)目 介紹41GPFSActive-Active第41頁RAC私網(wǎng)GPFS心跳以太網(wǎng)絡(luò)生產(chǎn)與DR主機(jī)DR與生產(chǎn)主機(jī)生產(chǎn)存放分區(qū)節(jié)點(diǎn)站點(diǎn)全部網(wǎng)絡(luò)生產(chǎn)節(jié)點(diǎn)第42頁類型描述被操作設(shè)備執(zhí)行操作現(xiàn)象描述數(shù)據(jù)恢復(fù)網(wǎng)絡(luò)故障模擬一個節(jié)點(diǎn)RAC私網(wǎng)中止Power770_3ifconfig en1 downPower770_3數(shù)據(jù)庫IO

42、終止而且該實(shí)例退出.其它節(jié)點(diǎn)數(shù)據(jù)庫數(shù)據(jù)庫IO暫停40秒左右后繼續(xù)兩份數(shù)據(jù)一致, 數(shù)據(jù)無需恢復(fù)模擬一個節(jié)點(diǎn)GPFS心跳通訊網(wǎng)絡(luò)故障Power770_4ifconfig en2 downPower770_4被從GPFS集群中踢出,數(shù)據(jù)庫實(shí)例退出，其它三個節(jié)點(diǎn)數(shù)據(jù)庫IO暫停20秒左右后繼續(xù)兩份數(shù)據(jù)一致, 數(shù)據(jù)無需恢復(fù)模擬站點(diǎn)間網(wǎng)絡(luò)通訊故障Power770_3/Power770_4ifconfig en2 down;ifconfig en1 downPower770_3,4從集群中踢出，其它兩個節(jié)點(diǎn)IO暫停60秒后繼續(xù)數(shù)據(jù)一致，無需恢復(fù)存放故障兩個站點(diǎn)間FC鏈路故障DS8300-08/09關(guān)掉生產(chǎn)站點(diǎn)

43、存放與DR站點(diǎn)主機(jī)相連光纖端口一個站點(diǎn)停頓服務(wù)，另一個站點(diǎn)在暫停40-50秒后恢復(fù)服務(wù)兩份數(shù)據(jù)不一致, 經(jīng)過增量方式實(shí)現(xiàn)數(shù)據(jù)同時關(guān)掉DR站點(diǎn)存放與生產(chǎn)站點(diǎn)主機(jī)相連光纖端口生產(chǎn)存放故障DS8300-08關(guān)掉生產(chǎn)站點(diǎn)存放與4個節(jié)點(diǎn)相連光纖端口四個節(jié)點(diǎn)業(yè)務(wù)暫停20秒左右后繼續(xù)兩份數(shù)據(jù)不一致, 經(jīng)過增量方式實(shí)現(xiàn)數(shù)據(jù)同時山東移動Oracle雙活項(xiàng)目 實(shí)測結(jié)果143切換測試第43頁類型描述被操作設(shè)備執(zhí)行操作現(xiàn)象描述數(shù)據(jù)恢復(fù)節(jié)點(diǎn)故障模擬節(jié)點(diǎn)故障Power770_3halt -q其它三個點(diǎn)數(shù)據(jù)庫IO暫停30秒左右后繼續(xù)兩份數(shù)據(jù)一致, 數(shù)據(jù)無需恢復(fù)站點(diǎn)故障模擬站點(diǎn)間通訊故障Power770_3/Power770

44、_4,DS8308/09ifconfig en1/en2 down;一個站點(diǎn)停頓服務(wù)，另一個站點(diǎn)在暫停40-50秒后恢復(fù)服務(wù)兩份數(shù)據(jù)不一致, 經(jīng)過增量方式實(shí)現(xiàn)數(shù)據(jù)同時關(guān)掉生產(chǎn)站點(diǎn)存放與DR站點(diǎn)主機(jī)相連光纖端口關(guān)掉DR站點(diǎn)存放與生產(chǎn)站點(diǎn)主機(jī)相連光纖端口模擬生產(chǎn)站點(diǎn)故障Power770_1,Power770_2,DS8300-08Power770_1/2: halt -qPower770_3/4數(shù)據(jù)庫IO暫停60秒左右后繼續(xù)兩份數(shù)據(jù)不一致, 經(jīng)過增量方式實(shí)現(xiàn)數(shù)據(jù)同時關(guān)掉生產(chǎn)站點(diǎn)存放與4個節(jié)點(diǎn)相連光纖端口山東移動Oracle雙活項(xiàng)目 實(shí)測結(jié)果244切換測試第44頁天津渤海銀行Oracle雙活項(xiàng)目 介

45、紹45GPFSActive-Active第45頁類型被操作設(shè)備執(zhí)行操作現(xiàn)象描述數(shù)據(jù)恢復(fù)1節(jié)點(diǎn)故障RAC02halt -qRAC01數(shù)據(jù)庫IO暫停17秒后繼續(xù)兩份數(shù)據(jù)一致, 數(shù)據(jù)無需恢復(fù)2仲裁節(jié)點(diǎn)(RAC03)halt -q沒有任何影響兩份數(shù)據(jù)一致, 數(shù)據(jù)無需恢復(fù)3存放故障生產(chǎn)站點(diǎn)存放關(guān)掉生產(chǎn)節(jié)點(diǎn)存放連到交換機(jī)上全部端口(2個)兩個節(jié)點(diǎn)數(shù)據(jù)庫IO均暫停76秒后繼續(xù)暫停76秒受PowerPath判斷路徑失敗機(jī)制影響, 卸載PowerPath之后切換時間降低到15秒兩份數(shù)據(jù)不一致, 經(jīng)過增量方式實(shí)現(xiàn)數(shù)據(jù)同時mmnsddiscover -a -N allmmchdisk oradata start -

46、d data_dr4網(wǎng)絡(luò)故障RAC02ifconfig en4 down模擬一個GPFS網(wǎng)絡(luò)通訊故障RAC02數(shù)據(jù)庫IO終止而且該實(shí)例退出RAC01數(shù)據(jù)庫IO暫停20秒后繼續(xù)兩份數(shù)據(jù)一致, 數(shù)據(jù)無需恢復(fù)需要重新開啟第二個節(jié)點(diǎn)數(shù)據(jù)庫5RAC02ifconfig en3 down模擬RAC私有網(wǎng)絡(luò)通訊故障RAC02節(jié)點(diǎn)會Crash并rebootRAC01節(jié)點(diǎn)數(shù)據(jù)庫IO暫停58秒后繼續(xù)兩份數(shù)據(jù)一致, 數(shù)據(jù)無需恢復(fù)需要重新開啟第二個節(jié)點(diǎn)數(shù)據(jù)庫6RAC02ifconfig en3 down;ifconfig en4 down;模擬站點(diǎn)間網(wǎng)絡(luò)通訊故障與上一致與上一致7RAC02RAC03(仲裁站點(diǎn))RAC

47、02: ifconfig en4 down;RAC03: ifconfig en1 down;模擬兩個GPFS節(jié)點(diǎn)通訊網(wǎng)絡(luò)故障GPFS集群因?yàn)槌霭霐?shù)節(jié)點(diǎn)失效,故GPFS文件系統(tǒng)不能訪問,Oracle被終止,兩個節(jié)點(diǎn)數(shù)據(jù)庫IO均中止兩份數(shù)據(jù)一致, 數(shù)據(jù)無需恢復(fù)網(wǎng)絡(luò)恢復(fù)后需要重新開啟兩個節(jié)點(diǎn)數(shù)據(jù)庫天津渤海銀行Oracle雙活項(xiàng)目 實(shí)測結(jié)果146切換測試第46頁類型被操作設(shè)備執(zhí)行操作現(xiàn)象描述數(shù)據(jù)恢復(fù)備注8FC鏈路故障RAC01關(guān)掉fcs0連在交換機(jī)上端口兩個節(jié)點(diǎn)數(shù)據(jù)庫IO暫停79秒后繼續(xù)兩份數(shù)據(jù)不一致, 經(jīng)過增量方式實(shí)現(xiàn)數(shù)據(jù)同時暫停79秒受PowerPath判斷路徑失敗機(jī)制影響, 卸載PowerP

48、ath之后切換時間降低到15秒9兩個站點(diǎn)間FC鏈路故障關(guān)掉生產(chǎn)站點(diǎn)存放與DR站點(diǎn)主機(jī)相連光纖端口關(guān)掉DR站點(diǎn)存放與生產(chǎn)站點(diǎn)主機(jī)相連光纖端口兩個節(jié)點(diǎn)立刻沒有數(shù)據(jù)庫IO, 第一個節(jié)點(diǎn)Oracle進(jìn)程退出, 第二個節(jié)點(diǎn)數(shù)據(jù)庫IO暫停76秒后繼續(xù)兩份數(shù)據(jù)不一致, 經(jīng)過增量方式實(shí)現(xiàn)數(shù)據(jù)同時暫停76秒受PowerPath判斷路徑失敗機(jī)制影響, 卸載PowerPath之后切換時間降低到15秒10站點(diǎn)故障生產(chǎn)站點(diǎn)(RAC02)及生產(chǎn)站點(diǎn)存放RAC02: halt -q關(guān)掉生產(chǎn)節(jié)點(diǎn)存放連到交換機(jī)上全部端口(2個)RAC01數(shù)據(jù)庫IO暫停76秒后繼續(xù)兩份數(shù)據(jù)不一致, 經(jīng)過增量方式實(shí)現(xiàn)數(shù)據(jù)同時暫停76秒受PowerPath判斷路徑失敗機(jī)制影響, 卸載PowerPath之后切換時間降低到15秒11存放故障(卸載PowerPath驅(qū)動)生產(chǎn)站點(diǎn)存放關(guān)掉生產(chǎn)節(jié)點(diǎn)存放連到交換機(jī)上全部端口(2個)兩個節(jié)點(diǎn)數(shù)據(jù)庫IO均暫停15秒后繼續(xù)(PowerPath影響了3,8,9,10測試場景切換時間)兩份數(shù)據(jù)不一致, 經(jīng)過增量方式實(shí)現(xiàn)數(shù)據(jù)同時采取AIX Native MPIO能加緊路徑切換時間天津渤海銀行Or