大數(shù)據(jù)技術(shù)交流37

大數(shù)據(jù)平臺技術(shù)交流吳敏達–資深技術(shù)顧問2從各種各樣類型的宏大數(shù)據(jù)中，快速獲得有價值信息的才干，就是大數(shù)據(jù)技術(shù)什么是大數(shù)據(jù)Variety:管理復(fù)雜的多角度關(guān)系和非關(guān)系類型的數(shù)據(jù)〔他能否忽略利用的非構(gòu)造化數(shù)據(jù)進展決策嗎〕Velocity:流數(shù)據(jù)或者大量數(shù)據(jù)的挪動〔他能否希望經(jīng)過實時操作提供更好的結(jié)果〕Volume:數(shù)據(jù)量從TB級到ZB級〔他能否搜集了一切數(shù)據(jù)，并在運用它嗎〕Veracity：1/3的指點在做業(yè)務(wù)決策時候不置信獲得的信息大數(shù)據(jù)參考架構(gòu)超越傳統(tǒng)的數(shù)據(jù)倉庫概念流計算Internet級別傳統(tǒng)數(shù)據(jù)倉庫In-MotionAnalyticsDataAnalytics,DataOperations&ModelBuildingResultsInternetScaleDatabase&WarehouseAt-RestDataAnalyticsResultsUltraLowLatencyResultsInfoSphereBigInsights傳統(tǒng)/關(guān)系型數(shù)據(jù)源非傳統(tǒng)/非關(guān)系型數(shù)據(jù)源傳統(tǒng)/關(guān)系型數(shù)據(jù)源非傳統(tǒng)/非關(guān)系型數(shù)據(jù)源Cloud|Mobile|SecurityIBM大數(shù)據(jù)平臺和運用框架經(jīng)過可視化的方法采集、抽取、以及探查數(shù)據(jù)運用加速器，加速運用開發(fā)，快速實現(xiàn)分析價值BI/Reporting BI/ReportingExploration/VisualizationFunctional

AppIndustry

AppPredictiveAnalyticsContentAnalyticsAnalyticApplications〔分析運用〕IBMBigDataPlatform〔大數(shù)據(jù)平臺〕SystemsManagementApplications&DevelopmentVisualization

&Discovery分析流數(shù)據(jù)，以及在大數(shù)據(jù)的是誰數(shù)據(jù)洞察數(shù)據(jù)管控〔數(shù)據(jù)質(zhì)量、生命周期、……〕低本錢地分析PB級構(gòu)造化和非構(gòu)造化數(shù)據(jù)操作型數(shù)據(jù)或者歷史數(shù)據(jù)的，基于數(shù)據(jù)倉庫內(nèi)嵌分析Accelerators〔加速器〕InformationIntegration&Governance信息整合和管控Hadoop

SystemStreamComputingDataWarehouseContextualDiscovery索引和聯(lián)邦的上下文相關(guān)分析議程IBMhadoop平臺－BigInsightsIBM流計算－StreamsIBM數(shù)據(jù)倉庫平臺－pureData基于大數(shù)據(jù)平臺的數(shù)據(jù)分析-DataExplorerIBM大數(shù)據(jù)優(yōu)勢匯總6ForresterWave關(guān)于大數(shù)據(jù)的報告BigInsights企業(yè)版銜接和集成StreamsNetezzaTextprocessingengineandlibraryJDBCFlume根底架構(gòu)JaqlHivePigHBaseMapReduceHDFSZooKeeperIndexingLuceneAdaptiveMapReduceOozieTextcompressionEnhancedsecurityFlexiblescheduler可選IBM產(chǎn)品分析和探查運用DB2BigSheetsWebCrawlerDistribfilecopyDBexportBoardreaderDBimportAdhocqueryMachinelearningDataprocessing...管理和開發(fā)工具管理控制臺Monitorclusterhealth,jobs,etc.Add/removenodesStart/stopservicesInspectjobstatusInspectworkflowstatusDeployapplicationsLaunchapps/jobsWorkwithdistribfilesystemWorkwithspreadsheetinterfaceSupportREST-basedAPI...REclipse開發(fā)工具TextanalyticsMapReduceprogrammingJaql,Hive,PigdevelopmentBigSheetsplug-indevelopmentOozieworkflowgenerationIntegratedinstallerOpenSourceIBMIBMCognosBIBigSQLAcceleratorformachinedataanalysisAcceleratorforsocialdataanalysisGuardiumDataStageDataExplorerSqoopHCatalogGPFS–FPOBigInsights優(yōu)勢列表HighPerformance&AvailabilityGPFS-FPOAtleast2XfasterthanopensourceHadoop17xthroughputspeedupfordocumentindexlookupsFaultresistanceforRealTimeDataPOSIXAdaptiveMapReduceSQLInterface(BigSQL)IntegratedInstall&MgtConsolesSecurityLDAP+HighspeedLZOCompressionDevelopmentToolingenvironment,testing,andoptimizationWarehouseRDBMS&StreamsIntegrationSystemT–TextAnalyticsBlazingFast,UsesUnstructureddata–doesnotrequirestructuring,(MapReduce)CustomizedAnnotatorsBigSheetsInsightEngineforanalyticsonMassiveamountsofdatainBigInsights.PowerofMap/ReducewithinreachoftheBusinessprofessionalwithafamiliarSpreadsheet-likeenvironment.BuiltinVisualizationsSystemML–MachineLeaning(Watson)DirectlyimplementedMLalgorithmsonMapReduceDeepStatistical/MiningembeddedintoBigInsightsPlatformBigIndexDistributedindexingandsearchParallelindexingandsearch企業(yè)級別根底設(shè)備企業(yè)級別分析才干GPFS-FPO與HDFS各項目的對比BigInsightsGPFS-FPO開源HDFS或其他方案健壯性無單點故障

99.99%NameNode存在單點故障數(shù)據(jù)一致性高數(shù)據(jù)可能會丟失可擴展性數(shù)千節(jié)點，實測4000+數(shù)千節(jié)點POSIX兼容完全兼容有限數(shù)據(jù)管理能力安全、備份、快照、緩存、復(fù)制有限傳統(tǒng)應(yīng)用性能好,兼顧讀寫性能隨機讀寫性能差安全性支持ACL,

容量限制，安全認證不支持IBMAdaptiveMapReduce提供強大的企業(yè)級管理，用于在可擴展的共享網(wǎng)格上運轉(zhuǎn)分布式運用程序和大數(shù)據(jù)分析。它可加速數(shù)十個并行運用程序，以加快實現(xiàn)成果并更好地利用一切可用資源。TeraSortThroughputSWIM10timesfewerCPUcores6timesfaster60timesfasterBerkleySWIMisaworkloadbenchmarkdevelopedatUniversityofCaliforniaatBerkley.MeasurecoreschedulingefficiencyofMapReduceworkloadsatHadoopWorld2021Multi-tenantresourcemanagement10xLesshardwareforthefastestTeraSortscore.BigSQL:讓Hadoop原生支持SQL原生SQL支持BigInsightsANSISQL92+Standardsyntaxsupport(joins,datatypes,…)

真正的JDBC/ODBCPreparedstatementsCancelsupportDatabasemetadataAPIsupportSecuresocketconnections(SSL)優(yōu)化LeveragingMapReduceparallelism

or…Directaccessforlow-latencyqueries多種數(shù)據(jù)源HBase(includingsecondaryindexes)CSV,Delimitedfiles,SequencefilesJSONHivetablesBigSQLEngineBigInsightsDataSourcesSQLHiveTablesHBasetablesCSVFilesApplicationJDBC/ODBCServerJDBC/ODBCDriver運用報表工具CognosBIserver可以下推計算到BigInsights更快呼應(yīng)時間沒有Hive的限制Application

(Map-Reduce)Storage

(HBase,HDFS)InfoSphereBigInsightsCognosBIServerExplore&AnalyzeReport&ActSQLInterface

viaJDBC可以運用已有的工具:SQuirreLSQLUsingexistingSQLtoolingagainstBigDataSupportfor“standard〞authentication!!

(notsupportedforHive,butsupportedbyBigSQL!)13可以運用已有的工具:EclipseUsingexistingSQLtoolingagainstBigDataSamesetupasforexistingSQLsources!!Supportfor“standard〞authentication!!14集成的基于Web的安裝無縫的單節(jié)點或者集群方式安裝開源組件和IBM組件的安裝驗證檢查，確保系統(tǒng)正常運轉(zhuǎn)基于Web的管理控制平臺義務(wù)和任務(wù)流管理系統(tǒng)安康監(jiān)控集群以及文件系統(tǒng)管理基于表單的分析儀表盤議程IBMhadoop平臺－BigInsightsIBM流計算－StreamsIBM數(shù)據(jù)倉庫平臺－pureData基于大數(shù)據(jù)平臺的數(shù)據(jù)分析-DataExplorerIBM大數(shù)據(jù)優(yōu)勢匯總計算方式的變革動態(tài)數(shù)據(jù)的實時分析流數(shù)據(jù)構(gòu)造或非構(gòu)造化動態(tài)數(shù)據(jù)流流計算實時對流數(shù)據(jù)進展分析計算靜態(tài)數(shù)據(jù)的歷史分析

批處置方式查詢驅(qū)動：靜態(tài)數(shù)據(jù)提交查詢依托數(shù)據(jù)庫，數(shù)據(jù)倉庫傳統(tǒng)計算方式流計算方式QueriesMemoryDiskUpdatesMemoryDiskEventDataQueriesAlertsActions20IBMInfoSphereStreamsAplatformforreal-timeanalyticsonBIGdata一個處置流數(shù)據(jù)的低延遲平臺毫秒級，甚至微妙級端到端的延遲一個可高度擴展的，用于實時分析的高性能平臺經(jīng)過橫向添加硬件獲得近線性的處置才干擴展高達125個節(jié)點擴展一個靈敏的、動態(tài)的平臺Streams運用靈敏部署支持動態(tài)部署新的分析運用

MillionsofeventspersecondMicrosecondLatencyTraditional/Non-traditionaldatasourcesRealtimedecisionsPowerfulAnalyticsAlgoTradingTelcochurnpredictSmartGridCyberSecurityGovernment/LawenforcementICUMonitoringEnvironmentMonitoring21延續(xù)注入延續(xù)分析實現(xiàn)可擴展：將運用分布到多個計算節(jié)點在流銜接的硬件節(jié)點之間分發(fā)Streams流計算平臺根底設(shè)備提供效力： 在跨硬件/軟件節(jié)點中調(diào)度分析 建立流媒體銜接 …變換過濾/采樣分類關(guān)聯(lián)注釋在適當(dāng)?shù)牡胤剑?處置單元可以是“交融〞在一同 從而消除通訊的延遲StreamsToolkit(常用)JoinFunctorAggregatePunctorSortFilterDirectory-ScanFileSourceFileSinkUDPSourceUDPSinkTCPSourceTCPSinkExportImportODBCSourceODBCEnrichsolidDBEnrichInetSourceODBCAppend高度可并行擴展才干同時利用幾十臺、上百臺進展實時數(shù)據(jù)流處置X86BoxX86BladeCellBladeBlueGeneFPGABladeX86BladeX86BladeX86BladeX86BladeX86BladeTransportStreamsDataFabric高可用、集群部署才干ProcessingElementContainerProcessingElementContainerProcessingElementContainerProcessingElementContainerProcessingElementContainer優(yōu)化調(diào)度器把操作符分配到不同的節(jié)點運轉(zhuǎn)，并繼續(xù)監(jiān)控資源的運用情況自順應(yīng)資源、任務(wù)負載、數(shù)據(jù)速率的變化在低本錢硬件上運轉(zhuǎn)－從單節(jié)點PC到刀片效力器到多陣列集群26參考性能數(shù)據(jù)－吞吐性能和延遲1,975streams2,133streams163streams24channels163DecisionEngines356BlueGeneNodes356ProcessingElements4,274streamsDataFeed每秒500萬條記錄平均延遲150微秒最小延遲50微秒65K中49條延遲>2毫秒大數(shù)據(jù)的實時分析平臺分析不同的數(shù)據(jù)源，每秒分析上百萬個事件．易用性能圖形化運用開發(fā)方式，容易管理和監(jiān)控．集成才干集成XML,MQ,DataStage,HDFS等．高級工具包和加速器事件序列和地理信息數(shù)據(jù)庫工具箱和CEP處置可定制化的電信和媒體分析加速器協(xié)助快速部署運用.InfoSphereStreamsInfoSphereStreams

議程IBMhadoop平臺－BigInsightsIBM流計算－StreamsIBM數(shù)據(jù)倉庫平臺－pureData基于大數(shù)據(jù)平臺的數(shù)據(jù)分析-DataExplorerIBM大數(shù)據(jù)優(yōu)勢匯總－專家集成數(shù)據(jù)系統(tǒng)forTransactions為滿足買賣型數(shù)據(jù)效力而優(yōu)化的系統(tǒng)forOperationalAnalytics為滿足操作型分析數(shù)據(jù)效力而優(yōu)化的系統(tǒng)forAnalytics為滿足分析型數(shù)據(jù)效力而優(yōu)化的系統(tǒng)DataPlatformDeliveringDataServices議程IBMhadoop平臺－BigInsightsIBM流計算－StreamsIBM數(shù)據(jù)倉庫平臺－pureData基于大數(shù)據(jù)平臺的數(shù)據(jù)分析-DataExplorerIBM大數(shù)據(jù)優(yōu)勢匯總NamechangeandnewreleaseIBMInfoSphereDataExplorerApplication/Users結(jié)合企業(yè)各類數(shù)據(jù)，包括大數(shù)據(jù)的分析結(jié)果FileSystemsRelationalDataContentManagementCRMSupplyChainERPRSSFeedsExternalSourcesCloudCustomSourcesVelocityPlatformIBMBigDataPlatformSystemsManagementApplicationDevelopmentVisualization

&DiscoveryAcceleratorsInformationIntegration&GovernanceHadoop

SystemStreamComputingDataWarehouseCommentingRatingSharedFoldersTaggingIDE給最終用戶提供整合的企業(yè)級的信息探查ApplicationFrameworkRealTimeAnalyticsInternetScaleAnalyticsIn-DatabaseAnalyticsFederatedDiscoveryNavigationandVisualization33EnterpriseDataConnecto