大數(shù)據(jù)分析存儲解決方案

從企業(yè)數(shù)據(jù)向大數(shù)據(jù)的擴展TraditionalApproachStructured,analytical,logicalSystemsofRecordNewApproach

Creative,holisticthought,intuitionSystemsOfEngagementMultimediaSystemsofInsightEnterpriseIntegration

andContextAccumulationStructured

Repeatable

LinearUnstructured

Exploratory

DynamicDataWarehouseWebLogsSocialDataTextData:

emailsSensordata:

imagesRFIDInternalAppDataTransactionDataMainframeDataOLTPSystemDataHadoopand

StreamsTraditionalSourcesNewSourcesERP

data具備洞悉才干的系統(tǒng)SystemsofInsight對新式根底架構(gòu)的需求在可靠和平安的環(huán)境中處置關(guān)鍵業(yè)務(wù)運用存取和處置海量數(shù)據(jù)——包括構(gòu)造化和非構(gòu)造化數(shù)據(jù)速度及時呼應(yīng)隨時能夠出現(xiàn)的商業(yè)時機，這就需求靈敏、實時性的根底架構(gòu)ThedynamicsofSoRandSoE:經(jīng)過負(fù)載及資源部署的優(yōu)化，來加強靈敏性和效益經(jīng)過采用包括基于開放規(guī)范的技術(shù)等新技術(shù)來改善ITeconomicsSystemofRecord〔SoR〕SystemsofEngagement〔SoE〕對的決策對的地方對的時間點BigData&Analytics大數(shù)據(jù)分析的新型架構(gòu)處理方案IBMBigData&AnalyticsInfrastructureDataZoneApplicationZone4SmartMeteringGridOperations電網(wǎng)管理FieldService外勤現(xiàn)場效力ResourcePlanning資源規(guī)劃CustomerService/CustomerOperations實現(xiàn)真正的有效的法規(guī)服從及時發(fā)現(xiàn)能源損耗問題、以及偷電和欺詐行為提高客戶稱心度電量運用預(yù)測更為準(zhǔn)確電網(wǎng)運維優(yōu)化減少停電次數(shù)和時間案例:SmartMetering智慧電力計費大數(shù)據(jù)分析運用可以帶來真正的業(yè)務(wù)價值法規(guī)服從案例:用大數(shù)據(jù)分析來加強SmartMetering數(shù)據(jù)分析的高可用性，以確保隨時了解用戶喜好跨運用的TB級的數(shù)據(jù)需求–通用虛擬化存儲平臺實時搜集、存儲并分析數(shù)據(jù)，最快可達(dá)50,000datapoints/sec歷史用電形狀數(shù)據(jù)的復(fù)雜查詢處置數(shù)據(jù)在加載到數(shù)據(jù)倉庫前的清洗、驗證，這些數(shù)據(jù)能夠來自很多的用戶、收費系統(tǒng)或斷電維護(hù)系統(tǒng)關(guān)系掌控

構(gòu)建和維護(hù)電網(wǎng)的獨一試圖對整個企業(yè)的構(gòu)造化和非構(gòu)造化數(shù)據(jù)t做全局導(dǎo)覽Navigation，從中發(fā)現(xiàn)Discover價值分析用戶用電情況，偵測偷電、改表等行為預(yù)測哪些用戶適宜于哪些分時時段電價或需求/呼應(yīng)效力分時時段電價的實時定價或

提供及時的需求/呼應(yīng)效力IBMBigData&AnalyticsReferenceArchitectureBigDataPlatformCapabilitiesInformationIngestReal-timeAnalyticsWarehouse&DataMartsAnalyticAppliancesAllDataSourcesAdvancedAnalytics/

NewInsightsNew/

EnhancedApplicationsCognitive認(rèn)知LearnDynamically?Prescriptive規(guī)范BestOutcomes?Predictive預(yù)測WhatCouldHappen?Descriptive

描畫WhatHasHappened?ExplorationandDiscoveryWhatDoYouHave?StreamingDataTextDataApplicationsDataTimeSeriesGeoSpatialRelationalSocialNetworkVideo&ImageAutomatedProcessCaseManagementAnalyticApplicationsWatsonCloudServicesISVSolutionsAlertsNewInfrastructureLeveragesDataTypesDatain

MotionDataat

RestDatain

ManyFormsInformationIngestionandOperationalInformationDecision

ManagementBIandPredictiveAnalyticsNavigation

andDiscoveryIntelligence

AnalysisRawDataStructuredDataTextAnalyticsDataMiningEntityAnalyticsMachineLearningLandingArea,AnalyticsZoneandArchiveVideo/AudioNetwork/SensorEntityAnalyticsPredictiveReal-timeAnalyticsExploration,IntegratedWarehouse,andMartZonesDiscoveryDeepReflectionOperationalPredictiveStreamProcessingDataIntegrationMasterDataStreamsInformationGovernance,SecurityandBusinessContinuityBigInsightsStreamsWarehouseInfoSphereBigInsightsHadoop-based低延遲分析，針對多樣化的、海量靜態(tài)數(shù)據(jù)Data-At-RestNetezzaHighCapacityAppliance基于構(gòu)造化數(shù)據(jù)的可查詢歸檔Netezza1000基于構(gòu)造化數(shù)據(jù)的

BI+定制化分析DataSmartAnalyticsSystem基于構(gòu)造化數(shù)據(jù)的運營分析InformixTimeseriesTime-structuredanalyticsInfoSphereWarehouse基于構(gòu)造化數(shù)據(jù)的大容量數(shù)據(jù)分析InfoSphereStreams低延遲流數(shù)據(jù)分析Velocity,Variety&VolumeData-In-MotionMPPDataWarehouseStreamComputingInformationIntegrationHadoopInfoSphereInformationServer海量數(shù)據(jù)集成和轉(zhuǎn)化ApacheHadoop:跨效力器集群的大數(shù)據(jù)集分布式處置開放系統(tǒng)框架，采用的是一種簡單化編程模型IBMBigDataPlatform大數(shù)據(jù)平臺What:一種開源軟件，將數(shù)據(jù)計算分布到整個集群的常見商用效力器和存儲上Why:傳統(tǒng)的計算架構(gòu)是一種沿縱向擴展方式，經(jīng)過更快的SAN、大容量內(nèi)存和多級緩存將數(shù)據(jù)加載到CPU上，本錢比較高。What:Hadoop把大數(shù)據(jù)集合拆分區(qū)劃為小數(shù)據(jù)集合，再把小數(shù)據(jù)集合分發(fā)到多臺普通效力器上，是一種橫向擴展方式。Why:Scalable,Flexible,CostEffective,FaultTolerentComponents:MapReduce,HDFSWhatisHadoop?NameNode(Metadatastore)NodesHDFSClusterOperatingSystemNodesElasticStorage-SNCClusterKernelLevelIBMValueforHadoop!HDFS把數(shù)據(jù)分散存儲在多個存儲節(jié)點Node上HDFS設(shè)計時就假設(shè)存儲節(jié)點有失效的能夠，所以HDFS會把一份數(shù)據(jù)復(fù)制3份以上，分散存儲在多個節(jié)點上，從而實現(xiàn)系統(tǒng)整體上的可靠性HDFS文件系統(tǒng)是由效力器節(jié)點集群組成的，每臺效力器按照HDFS的特有block協(xié)議支持網(wǎng)絡(luò)化block數(shù)據(jù)HDFSNameNode有發(fā)生單點缺點的危險IBM在改善文件系統(tǒng)的性能同時消除了單點缺點——ElasticStorage-SNC(availableasbetacode)Hadoop闡明,MapReduce,HDFSHadoopStackWhatdoesitlooklike?典型Hadoop存儲的PainPoints在選擇HDFS的組件〔如軟件、效力器、網(wǎng)絡(luò)和存儲等〕時很難選對在從測試環(huán)境遷移到消費環(huán)境時，需求做的調(diào)優(yōu)和調(diào)整任務(wù)太繁復(fù)了長期繼續(xù)不斷的運維保證過于繁重，比如老要改換失效組件〔尤其是硬盤〕，這使得保證期望的SLA非常難CPU和存儲去耦本來用戶的CPU和內(nèi)存曾經(jīng)滿足計算需求，但為了存儲容量需求安裝更多的硬盤不得不買更多的、不用要的CPU和內(nèi)存Storageoptionsavailablehavecleargaps本地存儲的利用率低(~25%)，每次需求擴容的時候就要添加更多的效力器，而一旦硬盤失效后需求重建，效力器越多，失效的幾率越高，性能也就越差I(lǐng)BMStorageforHadoop傳統(tǒng)的Hadoop集群運用的是效力器內(nèi)置硬盤存儲。假設(shè)用作測試或科學(xué)研討還好，可作為業(yè)務(wù)運轉(zhuǎn)的存儲就要采用企業(yè)存儲Hadoop集群要擔(dān)任數(shù)據(jù)維護(hù)和復(fù)制重建〔就是copy〕失效的數(shù)據(jù)集到不同節(jié)點上——嚴(yán)重影響CPU性能，無法實現(xiàn)企業(yè)級的RASReplicatedata–問題同上擴展的時候同時添加處置器/網(wǎng)絡(luò)/存儲，無法做到物盡其用〔nowaytoseparatethese3evenifexcesscapacityexistinginone(e.g.NeededmorestoragebuthadtoaddComputeandNetwork)〕運用外部存儲可以將存儲負(fù)載和Hadoop計算節(jié)點分別，同時還獲得了企業(yè)存儲的益處。SellthevalueofXIV,V7000,SVC,etc.用戶普通會隨HadoopFileSystem部署；采用ElasticStorage可以有很多益處14數(shù)據(jù)加速ExperiencetheinstantresultsthatcomefromIBMFlashSystemDriveasmuchas45Xfasteranalyticsresultsoncertainworkloads數(shù)據(jù)負(fù)載的多樣性和靈敏性XIVdeliverspredictableperformancethatscaleslinearlywithouthotspotsdeliveringinsightsfromanalyticsfasterwithtuning-freedatadistributionScale-out,parallelprocessingofElasticStoragesoftwareandintegrationwithFlashSystemdramaticallyacceleratesperformanceofAnalyticsclustersVirtualStorageCenterwithSVCautomaticallyoptimizesdatawarehouseperformanceandcostacrossFlashandDiskMainframeDataEnvironmentsIntegrationwithDB2&specialtyanalytics“engines〞leveragingDS8870delivers4xreductioninbatchtimeswithnewHighPerformanceFlashEnclosuresHighspeedencryptiononeverydrivetypesecuresdata數(shù)據(jù)維護(hù)和保管LTFSEEw/tapeprovidesreducedTCObyupto90%overdiskforlongtermretentionofdataatrestwithalargeopenformattaperepositoryReducetheamountofdatatobestoredbyupto25timeswithProtecTIERde-duplication12x更快IBMFlashSystemincreasedSPLUNK&SASapplicationefficiencytoperformbusinessanalytics20x改善inactionablesupplychainanalytics,4xreductioninbatchtimes,virtualizationforplug&play6x時間節(jié)省“GPFSallowsustomovethemetadatafromthedisktotheFlashSystemonline.Oncewedidthat,thebackupswerereduceddowntoaboutanhour.〞2hrsbecomes2minutes失效切換時間大幅縮短MappingCharacteristicstoIBMStorageProductsStorageInfrastructure需求適用于一切的5種運用場景OptimizedMulti-TemperatureWarehouse優(yōu)化的多級存儲庫AllFlashFlashSystemHybridDS8000EasyTierXIV+SSDCachingStorwizeEasyTierFlashSystemSolution(VSC+FlashSystem)PureSystemsPureFlex(XIVorStorwizew/EasyTier)PureDataforTransactions(Storwize)PureDataforAnalytics(Netezza)Midrange&EntryTier0AccelerationSmarterStorageIntegratedSystemsEnterpriseOfferingsXIVzEnterpriseSolutionsforAnalyticswithDS8000PureDataSystemforOperationalAnalyticswithStorwizePureFlexSystemwithStorwizeDS8000SmartAnalyticsSystemswithDS3xxxOpen&ExtensibleStorwizefamilyFlashSystemfamilyIBMSmarterStorage的設(shè)計就是支持大數(shù)據(jù)分析

高效和優(yōu)化數(shù)據(jù)根底架構(gòu)IBMFlashSystem：為大數(shù)據(jù)分析運用設(shè)計的，讓運用和數(shù)據(jù)實現(xiàn)極速IBMFlashSystem的極速性能讓實時業(yè)務(wù)決策成為能夠適宜于模塊化數(shù)據(jù)存儲構(gòu)造的Hadoop系統(tǒng)。某些或一切數(shù)據(jù)可以保管到Flash閃存上，其他可以保管到XIVIBMXIV:OptimizeddataworkloaddiversityforBigData&AnalyticsIBMXIV的高性能無須人工干涉配置，且適用于各種各樣的存儲負(fù)載IBMXIV的效率高的異乎尋常，而且簡單性業(yè)內(nèi)最高，內(nèi)置友好界面IBMXIV的彈性是企業(yè)級的，完全保證了數(shù)據(jù)的可用性和業(yè)務(wù)延續(xù)性XIV:為Analytics而生無與倫比的性能可擴展的網(wǎng)格存儲架構(gòu)恣意時間支持恣意讀寫負(fù)載板上的閃存Flash

無與倫比的可靠性精致的數(shù)據(jù)分布無雙的磁盤重建時間企業(yè)級的可用性

無與倫比的簡易性簡單的規(guī)劃、供應(yīng)和靈敏性上線后零維護(hù)零調(diào)優(yōu)“XIV最吸引我們的地方就是其超強的性能…we正是由于XIV為我們的精細(xì)復(fù)雜的分析運用提供了一致的高性能，使得我們可以為我們的用戶帶來更多的價值。〞SAS和XIV網(wǎng)格架構(gòu)——完美的結(jié)合大規(guī)模并行計算堅持繼續(xù)地最正確性能BalancedPerformance性能平衡年年零調(diào)整UnprecedentedScalability史無前例的擴展性配合添加SAS節(jié)點和XIV模塊即可IBMSVC:OptimizeddataworkloadflexibilityforBigData&AnalyticsIBMSVC經(jīng)過如下功能在IBM大數(shù)據(jù)產(chǎn)品線上添加了靈敏性：完好和數(shù)據(jù)虛擬化和數(shù)據(jù)挪動性高級集群和復(fù)制多路鏡像，readpreferredoptionRealTimeCompression實時緊縮EasyTierHotExtentcachingStorwizeV7000/UIBMSVC設(shè)計原那么Real-TimeCompression實時緊縮是設(shè)計來做：作用于ActivePrimaryData公用的緊縮平臺PlatformhandlesALLheavyliftingassociatedwithcompression不會影響性能Wemodifyacompressedfilein-placeefficiently不會改動用戶運用Usersnoradminsneedtochangeanything處置流程不變緊縮是在線完成，不是事后緊縮業(yè)界規(guī)范緊縮算法所采用的緊縮算法曾經(jīng)運用了幾十年StorwizeV7000/UIBMSVC24流處置計算&IBMFlashSystemsData:是擁有還是保管?或是是分析和開場行動!DatainDataat25InfoSphereStreams:大數(shù)據(jù)流分析為分析動態(tài)數(shù)據(jù)而建多并發(fā)輸入數(shù)據(jù)流大規(guī)?？蓴U展Massivescalability分析和處置的數(shù)據(jù)多樣化Structured,unstructured,video,audioAdvancedanalyticoperators自順應(yīng)實時分析WithDataWarehousesWithHadoopSystemsCurrentfactfinding當(dāng)前數(shù)據(jù)查詢分許流動中的數(shù)據(jù)——在數(shù)據(jù)落盤前低延遲方式,pushmodel數(shù)據(jù)驅(qū)動——真正的數(shù)據(jù)分析Historicalfactfinding歷史數(shù)據(jù)查詢查找和分析存儲在磁盤上的數(shù)據(jù)信息批處置方式,pullmodel查詢驅(qū)動:submitsqueriestostaticdataTraditionalComputingStreamComputing流數(shù)據(jù)計算代表著計算方式的變化Real-timeAnalyticsRealTimeAnalytics實時分析

想象一下他如何用防火栓喝水來自多個多樣輸入源的大量數(shù)據(jù)直接處置和過濾數(shù)據(jù)，而不用存儲僅保管有價值的數(shù)據(jù)僅關(guān)聯(lián)對數(shù)據(jù)最感興趣的用戶隨著數(shù)據(jù)信息的產(chǎn)生采取行動AdaptiveAnalytics自順應(yīng)分析

DatainMotionandDataatRest的集成1.DataIngest數(shù)據(jù)集成，數(shù)據(jù)發(fā)掘，機器學(xué)習(xí)，統(tǒng)計建模實時和歷史數(shù)據(jù)洞察力的可視化3.AdaptiveAnalyticsModel數(shù)據(jù)收取，

在線分析預(yù)備，方式校驗Data2.Bootstrap/EnrichControlflowInfoSphereBigInsights,Database&WarehouseInfoSphereStreams

AdaptiveReal-TimeAnalytics自順應(yīng)實時分析來自多個多樣輸入源的大量數(shù)據(jù)過去、如今和未來全方位綜合性視圖實時分析，低延時結(jié)果Fullcontextfordeepanalysis深度分析的完好的上下文跨datainmotionanddataatrest的常用數(shù)據(jù)分析自順應(yīng)-隨機而變當(dāng)發(fā)現(xiàn)非預(yù)期行為時，自順應(yīng)當(dāng)識別出新數(shù)據(jù)意義時深度分析之開場沒有認(rèn)識到的數(shù)據(jù)意義，隨后才能夠認(rèn)識到自順應(yīng)——在開場沒有認(rèn)識到的，隨后可以找出數(shù)據(jù)方式StockmarketImpactofweatheronsecuritiespricesAnalyzemarketdataatultra-lowlatenciesMomentumCalculatorFraudpreventionDetectingmulti-partyfraudRealtimefraudpreventione-ScienceSpaceweatherpredictionDetectionoftransienteventsSynchrotronatomicresearchGenomicResearchTransportationIntelligenttrafficmanagementAutomotiveTelematicsEnergy&UtilitiesTransactivecontrolPhasorMonitoringUnitDownholesensormonitoringNaturalSystemsWildfiremanagementWatermanagementOtherManufacturingTextAnalysisERPforCommoditiesReal-timemultimodalsurveillanceSituationalawarenessCybersecuritydetectionLawEnforcement,

Defense&CyberSecurityHealth&LifeSciencesICUmonitoringEpidemicearlywarningsystemRemotehealthcaremonitoringTelephonyCDRprocessingSocialanalysisChurnpredictionGeomapping如何運用InfoSphereStreams?加快數(shù)據(jù)流入分析系統(tǒng)的速度向買賣方向加速。。。一個高效和靈敏的根底架構(gòu)顯然可以加快流速，并平衡不同數(shù)據(jù)分析的需求CoresSCMStorageNetworkCoresSCMStorageNetworkCoresSCMStorageNetworkCoresSCMStorageNetwork++預(yù)測分析

數(shù)據(jù)倉庫文本分析HadoopWorkloads優(yōu)化敏感性分析加快流速價值時間“觸發(fā)事件〞數(shù)據(jù)完備買賣Insight預(yù)見獲取數(shù)據(jù)時間分析數(shù)據(jù)時間行動時間大數(shù)據(jù)分析的新式根底架構(gòu)處理方案IBMBigData&AnalyticsInfrastructureDataZoneApplicationZoneExperiencereal-timeanalyticalinsightswithupto50xbetterperformancethanenterprisedisksystemsusingIBMFlashCore?technologyPreserveandprotectinfrastructurecontinuitywhilescalingtoover2petabyteofeffectiveall-flashcapacityunderasingleintegrateinterfaceDeliveragilityanddataeconomicswith4xgreatercapacityinlessrackspacethancompetitiveall-flashproductsSynchronizedandComplimentarytoOverarchingStorageMessaging-Acceleratetimetoinsightsthrough"datawithoutborders."IBMinnovationfreesdatawithagileandsimpletousestoragesolutionsdeliveringsuperiordataeconomicsIBMFlashSystemCoreLaunchMessagingDriveacompleteparadigmshiftinEnterpriseStoragewiththeallnewIBMFlashSystemFamilyIBMFlashSystemFamily

2021ThemeTimetoinsight.Timetovalue.Timetomarket.IBMFlashSystem,it’sabouttime.FlashRealized!IBMFlashSystemV9000

FoundationalPillarsIBMFlashCore?TechnologyistheDNAoftheFlashSystemFamilyScalablePerformanceEnduringEconomicsAgileIntegrationIntroducingtheNewIBMFlashSystemFamilyOfferingsIBMFlashSystem900ExtremePerformance:Delivers100microsecondresponsetimesMacroEfficiency:Lowestlatencyofferingwith>40%greatercapacityatalowercostpercapacityEnterpriseReliability:IBMenhancedMicronMLCflashtechnologywithFlashWearGuaranteePoweredbyIBMFlashCore?TechnologyIBMFlashSystemV9000ScalablePerformance:Growcapacityandperformancewithupto2.2PBscalingcapabilityEnduringEconomics:NextgenerationflashmediawithlowercostpercapacityAgileIntegration:FullyintegratedsystemmanagementtosimplifymanagementandimproveworkforceproductivityunderasinglenamespaceFlashSystem900IntroducingIBMFlashSystem900,thenextgenerationinourlowestlatencyofferingIBMMicroLatency?withupto1.1millionIOPS40%greatercapacityata10%lowercostpercapacityIBMFlashCore?technology,oursecretsauceTechnicalcollaborationwith