EMC Isilon橫向擴展存儲解決方案

云計算

大數(shù)據(jù)EMCIsilon橫向擴展存儲解決方案技術(shù)顧問經(jīng)理：楊峰Yang.feng有計算環(huán)境的挑戰(zhàn)計算系統(tǒng)以超級計算機、集群或者網(wǎng)格為主多核計算迅速發(fā)展，物理機能力越來越強，原有的資源管理與調(diào)度器基于物理機，對于調(diào)度準確度要求更高靜態(tài)調(diào)度：基于系統(tǒng)負載狀態(tài)和作業(yè)信息將調(diào)度作業(yè)到物理機器上難以滿足突發(fā)、批處理、CPU密集型、數(shù)據(jù)密集型等不同類型的作業(yè)對于不同的物理資源的需求如果作業(yè)類型與機器資源不匹配，會導致作業(yè)運行效率低。如果采用多隊列調(diào)度，又會導致隊列之間的共享問題計算應用越來越多樣化，但是傳統(tǒng)調(diào)度系統(tǒng)的接口單一、不易擴展，嚴重限制了應用推廣基于集群或者網(wǎng)格構(gòu)建分布式“私有云”，以按需定制的模式提供服務，可滿足應用日益增長的多樣性和廣泛性的需求遺留程序越來越多，老操作系統(tǒng)和軟件與新硬件無法配合，全部移植幾乎不可能如不能兼容，原有寶貴的數(shù)據(jù)無法讀取和分析引入虛擬化技術(shù)現(xiàn)有存儲環(huán)境的挑戰(zhàn)存儲以高性能并行文件系統(tǒng)為主容量管理PB成為基本配置，10PB-20PB已經(jīng)非常常見文件管理文件數(shù)量越來越多，數(shù)千萬至10億級別設(shè)備老化硬盤、電源等部件頻繁出錯，導致服務中斷元數(shù)據(jù)問題可靠性性能瓶頸擴容問題大規(guī)模數(shù)據(jù)遷移引入云存儲技術(shù)！EMCIsilon對云存儲的需求海量數(shù)據(jù)存儲，輕松管理PB級乃至數(shù)十PB的存儲空間全局命名空間，所有應用看到統(tǒng)一的文件系統(tǒng)視圖支持標準接口，應用無需修改直接運行讀寫性能優(yōu)異，聚合帶寬高達數(shù)GB乃至數(shù)十GB易于管理維護，無需中斷業(yè)務即可輕松實現(xiàn)動態(tài)擴展基于開放架構(gòu)，可以運行于任何開放架構(gòu)的硬件之上多級數(shù)據(jù)冗余，支持硬件與軟件冗余保護，數(shù)據(jù)高可靠多級存儲備份，靈活支持SSD、SAS、SATA、磁帶庫統(tǒng)一管理提綱EMCIsilon產(chǎn)品介紹EMCIsilon存儲解決方案總結(jié)內(nèi)容概括您將從中了解：EMCIsilon產(chǎn)品技術(shù)特性如果您的業(yè)務系統(tǒng)需要使用橫向擴展架構(gòu)，Isilon哪個型號/功能是您所需要的大數(shù)據(jù)時代已經(jīng)到來地球物理

勘探醫(yī)學成像視頻

監(jiān)視移動傳感器視頻渲染基因測序智能電網(wǎng)社交網(wǎng)絡(luò)電子支付大數(shù)據(jù)時代的橫向擴展架構(gòu)VS縱向擴展架構(gòu)縱向擴展架構(gòu)橫向擴展架構(gòu)Faster…………….Slower…………….TRAINSTOPS!!!!!!!!!!!!TRAINKEEPSGOING!!OneFS操作系統(tǒng)最佳的企業(yè)級大數(shù)據(jù)存儲——OneFS7.02012/11/26GA超大容量（20PB+）安全性高卓越性能數(shù)據(jù)保護可預期的性能互操作性操作簡便應對快速增長EnterpriseSCALE-OUTCoreInnovation…

IsilonOneFSScale-Out系統(tǒng)單一文件系統(tǒng)，單一磁盤卷，最大支持20PB+大于80%的存儲利用率高性能，全對稱的集群系統(tǒng)易管理，易擴展Multi-tier單一文件系統(tǒng)/單一集群

Scale-OutNASPerformanceOneFS7.0性能新特性106GB/s并發(fā)吞吐降低50%延遲724MB/s單流EnterpriseIT性能考量隨機/IO密集型BigData

性能考量吞吐1.6MIO/sSPECsfs2008

benchmark線性擴展性

>20PB單一文件系統(tǒng)

(144nodes，4TBSATA盤)最大并發(fā)吞吐性能>=106GB/secIsilon典型架構(gòu)NFS,CIFS,FTP,HTTP客戶端/應用端層標準的網(wǎng)絡(luò)層千兆/萬兆Isilon集群存儲層集群存儲內(nèi)部通訊InfiniBand層WindowsUNIX/LINUXMAC(optional2ndswitchforadditionalsubnet)(optional2ndswitchforhighavailability)Isilon產(chǎn)品家族Scale-outNASproductfamily性能和容量同時線性擴展適用于IOPS密集型應用的高性能解決方案S200高并發(fā)高吞吐的解決方案X200X400成本優(yōu)化的大容量擴展解決方案NL400S-SeriesX-SeriesNL-SeriesIsilonS200配置可選項2RU2.5”10KRPMSASHDDDualIBPorts2x10GECard2x1GEportsVGA(HD15)IPMIServicePortConsolePort(DB9)USB(x2)PowerReceptaclesIsilonS200–前后視圖IsilonX200可選項配置:2RU3.5”7.2KRPMSATAHDDDualIBPorts2x10GECard2x1GEportsVGA(HD15)PowerReceptacles(Nema5-15R)IPMIServicePortConsolePort(DB9)USB(x2)IsilonX200–前后視圖IsilonX400可選項配置:IsilonX400–后視圖1243567812435678PowersuppliesBatteriesPowerswitchUSBportsSerialportGigabitEthernetexternalnetworkportsOptionalexternalnetworkportsInfinibandinternalnetworkportsIsilonNL400可選項配置:IsilonPlatform擴展節(jié)點

性能加速器2顆4核CPU8/32GB內(nèi)存無硬盤2個萬兆高速端口或2個千兆端口加入集群時間低于15秒用于增加集群對外服務的吞吐/IOPSIsilonPlatform擴展節(jié)點

備份加速器2顆4核CPU8GB內(nèi)存無硬盤4個4GbFC端口支持NDMPV3/V4提供可高達1.7TB/小時的備份速度IsilonOneFS產(chǎn)品架構(gòu)OneFS為EMCIsilon專利的操作系統(tǒng)，負責IO調(diào)度，集群管理Smartconnect?提供應用訪問負載均衡和容錯SmartPools?提供自動分層功能SnapshotIQ?提供本地數(shù)據(jù)保護/恢復SyncIQ?提供存儲間的數(shù)據(jù)復制（本地/異地），保證業(yè)務連續(xù)性SmartQuotas?報告/管理存儲資源的使用，同時擁有精簡供應功能SmartLock?提供WORM技術(shù)InsightIQ?提供存儲性能報表和使用趨勢分析HDFS特性支持Hadoop大數(shù)據(jù)應用AsperaforIsilon?適用于遠程數(shù)據(jù)復制，提高鏈路帶寬利用率強大但簡易——Scale-out60秒完成擴展無停機時間Isilon–線性,可預期的可擴展性根據(jù)需求增長容量和性能無需超量購買存儲增加無需增加管理的時間和精力減少管理時間和成本高效的業(yè)務的成果

無需數(shù)據(jù)移植單一,可擴展的卷可預期的SLATheAnatomyofaWriteServers(optional2ndswitch)NFS,SMB,HDFSFTP,HTTPServersServersRAM(Cache)WriteCoalescerL2CacheNVRAMInfinibandHardDriveWritesarecacheduntilwritecoalescerisfull,timelimitreached,orprotocolrequestsconfirmationofdeliveryAtthesametime,datablocksassignedtoothernodesgothroughIBintotheirL2cacheOnceallnodeshaveallthedataandparityblocksinNVRAM,confirmationfromallnodesissenttothenodebeingwrittento,andacommitisreturnedtotheclientDataisthenwrittenontothespindlesWriteplanandparitycalculatedDatablock(s)assignedtothisnodestaycachedinL2forfuturereadsofthatfileDataisthencopiedintobatterysuppliedNVRAMfromL2CacheTheAnatomyofaReadServers(optional2ndswitch)NFS,SMB,HDFSFTP,HTTPServersServersRAM(Cache)L1CacheL2CacheNVRAMInfinibandHardDriveIfdataisn’talreadyinL2cache,datablockiscopiedfromtheHDDintoL2cache,thenIntoL1cacheDataisthenreconstructedInL1cacheRAM(Cache)L1CacheL2CacheNVRAMInfinibandHardDriveInthisexamplethough,sincewerecentlywrotethedata,itwasalreadyinL2cache.Wedon’tneedtoloaditfromtheHDD.ItjustloadsinfromtheL2cacheintotheL1cacheNode1Node2Node3DataLayoutDetailExample:3Node,N+1,1024KBfile,Concurrent1024KBFile128KB128KB128KB128KB128KB128KB128KB128KB128KB128KB128KB2+1FECProtectionGroups(Stripes)128KB128kDataStripeUnit128kProtectionStripeUnitNode1Node2Node3DataLayoutDetailExample:3Node,N+1,1024KBfile,Streaming1024KBFile128KB128KB128KB128KB128KB128KB128KB128KB128KB128KB128KB2+1FECProtectionGroups(Stripes)128KB128kDataStripeUnit128kProtectionStripeUnitNode1Node2Node3DataLayoutDetailExample:3Node,N+2:1,1024KBfile,Concurrent1024KBFile128KB128KB128KB128KB128KB128KB128KB128KB128KB128KB128KB2+1FECProtectionGroups(Stripes)128KBStripedepthisdoubledBlockswithinthesameStripenowarewrittentoseparatedrivesoneachnode128kDataStripeUnit128kProtectionStripeUnitOneFSFlexProtect–磁盤重構(gòu)數(shù)據(jù)ECC校驗碼分布在各個節(jié)點的各個磁盤中相對傳統(tǒng)存儲，橫向擴展架構(gòu)有更多的CPU和磁盤參與數(shù)據(jù)重構(gòu)Isilon傳統(tǒng)存儲16hourrebuild1hourrebuild最可靠的存儲系統(tǒng)——FlexProtect10GigEdedicatedAcceleratorXnodesNFSFailoverrequiredProcessingITFullaccess,maintenanceinterfaceCorporateDNS,noSCStatic(well-known)IPsrequiredBizzDevRenamedsub-domainCIFSclients(staticIP)10.10EngSharedsubnetSeparatesub-domainNFSFailoverVLAN(confidentialtraffic,isolated)SamephysicalLAN10.30FinanceStoragenodesNFSclients,nofailoverEditors10.20ServiceIPDNSDelegationRecordIsilonNodeIPBalancePolicySmartConnectzonenameDynamicIPs252627282930313226X1GigE10GigE應用負載均衡——SmartConnect數(shù)據(jù)分層存放FAST_POOLOLD_DATASSD/SASDiskpoolSATADiskpool數(shù)據(jù)分層存放新文件寫入存儲30天之后…文件分布到FAST_POOL數(shù)據(jù)塊自動遷移到OLD_POOL37配額管理及精簡供應——SmartQuotas基于目錄的Quota基于用戶的Quota基于組的QuotaThinProvisionSnapshotIQ

快照/恢復快速有效簡單業(yè)務連續(xù)性解決方案——SnapshotIQ病毒意外刪除(optional2ndswitch)ServersNFS,CIFS,FTP,HTTP(optional2ndswitch)ServersServers備份服務器SAN交換機磁帶機FCAccelerator備份服務器上可運行的備份軟件業(yè)務連續(xù)性解決方案——NDMP備份業(yè)務連續(xù)性解決方案——SyncIQ北京上海本地/遠程數(shù)據(jù)復制基于IP鏈路改善數(shù)據(jù)鏈路復制效率一對多/多對一方式增強RPO/RTO異步復制方式正向/反向復制一鍵式恢復Failover/FailbackSIMPLE多認證域的支持例如：整合法律部和財務部;財務部擁護登錄到LDAP1；法律部擁護登錄到LDAP3；存儲端數(shù)據(jù)各自獨立，互相無法訪問；SmartConnectZone1SystemZoneSmartConnectZone2LDAP1ActiveDirectory2LDAP3NISNISLocalUserGroupDatabaseAuthZone1AuthZone2FinanceLegalLDAP2ActiveDirectory1GroupDatabase對虛擬化的支持CIFSNFSFTPHTTPHDFS

(Hadoop)VAAIVASAOneFSAPIVAAI–VstorageAPIsForArrayIntegrationVASA–VsphereAPIsForStorageAwareness可寫虛機克隆網(wǎng)絡(luò)延遲大大減小與Vmware完美整合

簡單PlatformOneFSRESTAPI管理和整合更加容易統(tǒng)一化管理和整合第三方應用靈活HadoopVS.Google技術(shù)架構(gòu)的比較數(shù)據(jù)結(jié)構(gòu)化管理組件：Hbase→BigTable并行計算模型：MapReduce→MapReduce分布式文件系統(tǒng)：HDFS→GFSHadoop缺少分布式鎖服務ChubbyZookeeper是Chubby的克隆，為Hadoop高可用性服務提供保障.HBaseMapReduceHDFSBigTableMapReduceGFSHadoop云計算應用ChubbyGoogle云計算應用StorageMapReduceCompute業(yè)界唯一的內(nèi)嵌HDFS的橫向擴展存儲解決方案內(nèi)嵌支持HDFS管理簡單

HDFSTechnologyChallengesofHadoopHadoopDASEnvironment1DedicatedStorageInfrastructureOne-offforHadooponly2SinglePointofFailureNameNode3LackingEnterpriseDataProtectionNoSnapshots,replication,backup4PoorStorageEfficiency3Xmirroring5FixedScalabilityRigidcomputetostorageratio6ManualImport/ExportNoprotocolsupportNameNodeTechnologyChallengesofHadoopHadoopDASEnvironment1DedicatedStorageInfrastructureOne-offforHadooponly2SinglePointofFailureNameNode3LackingEnterpriseDataProtectionNoSnapshots,replication,backup4PoorStorageEfficiency3Xmirroring5FixedScalabilityRigidcomputetostorageratio6ManualImport/ExportNoprotocolsupport1x1x2x2x3x2x3x3x1xNameNode1Scale-OutStoragePlatformMultipleapplications&workflows2NoSinglePointofFailureDistributedNameNode3End-to-EndDataProtectionSnapshotIQ,SyncIQ,NDMPBackup4Industry-LeadingStorageEfficiency>80%StorageUtilization5IndependentScalabilityAddcompute&storageseparately6Multi-ProtocolIndustrystandardprotocolsNFS,CIFS,FTP,HTTP,HDFSHDFSEMCIsilonHadoop解決方案提綱EMCIsilon產(chǎn)品介紹EMCIsilon垂直行業(yè)解決方案介紹總結(jié)能源行業(yè)UpstreamProcessesandIndustryApplicationPartnersExplorationDevelopmentProductionEngineeringSeismicacquisitionSeismicprocessingSeismic&GeologicalinterpretationReservoirModelingDrilling&testingProductionDevelopmentandOptimizationReservoirSimulationFacilities&ReservoirEngineeringDrillingprogramProduction

LandmarkProMAXSeisSpaceLDIParadigmEchosSchlumbergerOmegaLandmarkR5000SeisWorksStratWorksGeoProbeDecisionSpaceParadigmSeisEarthStratEarthSchlumbergerPetrelGeoFrameSMTKingdom

LandmarkNexusPowerGridVIPParadigmSKUASchlumbergerEclipseIntersectPetrel

LandmarkCompassAssetPlannerWellborePlannerParadigmSysdrillSchlumbergerPetrelDrillingLandmarkOpenWellsSperryDrillingParadigmSysdrillSchlumbergerOspreyOperation

LandmarkDecisionSpaceforProductionTOWSchlumbergerAvocetAutomatedIntelligentDataTieringEMCExplorationSolutionsSeismicprocessingSeismicInterpretationGeologicalInterpretationReservoirModelingReservoirSimulationPetroVault:IntelligentScale-outTier2StorageforArchiveandStagingStorageSolutionsforSeismicProcessingIntelligentScale-outTier1StorageforInterpretationandModelingMakesnapshotseasy

UsetheFUSEIMplug-intocreatePetrelsnapshotsfromthesimple-to-useOceanuserinterfaceAccessaSnapshotLibrary

Browseyourlibraryofsnapshots:InsidePetrelorwithinaweb-browserByprojectchronologySelectasnapshottosee:DetailedprojectattributesStatisticsandhistoricalinformation

OnlyEMCoffersaSnapshotLibraryFeaturewiththePetrelSnapshotplug-in云存儲+

EMCISILONSCALE-OUTNASJeetuPatel,SyncplicityGM,VPJeffSchultz,Syncplicity,MarketingVinodMuralidhar,IsilonProductMgmt10December2012Streamingrecordinglink:/isilon/ldr.php?AT=pb&SP=MC&rID=117802012&rKey=b053de0490a4422dDownloadrecordinglink:/isilon/lsr.php?AT=dw&SP=MC&rID=117802012&rKey=e7a70b1cd947b62eOUTSIDEFIREWALLIPHONEANDROIDIPADIPADMACBOOKAIRIPHONEACROSSALLDEVICESSYNCFILESITSecurityandControlSyncAnyFolderSYNCPLICITYDIFFERENCEDISTRIBUTIONGROUPFILEITSecurityandControlSyncandNotificationsReadOnlyPermissionsSYNCPLICITYDIFFERENCEISILONTOMORROW:

CloudService/OnPremStorageFILESAUTHENTICATIONSyncplicity”O(jiān)rchestration”AuthenticationMetaDataEncryptionKeysSyncplicity”StorageandCompute”FilesON-PREMISECLOUD媒體

TypicalArchitecture–BroadcastMedia&Entertainment

TypicalArchitecture–OriginStorageforContentDeliveryMedia&Entertainment–Streaming

TypicalArchitecture–Film-scanforArchivalandBroadcastMedia&Entertainment