Pivotal開源大數(shù)據(jù)平臺概覽

1、A NEW PLATFORM FOR A NEW ERAO Copyright 2013 Pivotal. All rights reserved.Pivotal 介紹一家新成立的公司正式成立于 2013年4月1日繼承了 EMC 和 VMWare 兩家公司在特定產(chǎn)品和技術(shù)方向上的人員和資產(chǎn)股東: EMC, Vmware, GE公司使命: 為一個嶄新的技術(shù)時代構(gòu)建支撐平臺 Copyright 2013 Pivotal. All rights reserved. 日程為什么需要一個新的技術(shù)支撐平臺?技術(shù)支撐平臺分類與演進技術(shù)傳承，團隊組成與未來計劃特別嘉賓主題 Copyright 2013 Piv

2、otal. All rights reserved.企業(yè) IT 的帶系發(fā)展MAINFRAME實現(xiàn)財務(wù)處理自動化MainframesISAMCLIENT-SERVER& WEB實現(xiàn)大部分紙面處理流程的自動化: ERP, CRM, Email, Relational DatabasesMinis & PCsCloud-Enabled DatacenterNew Data-fabricsCONSUMER GRADE新的客戶體驗新的業(yè)務(wù)模式由服務(wù)于海量客戶的互聯(lián)網(wǎng)巨頭企業(yè)引領(lǐng)潮流 新的應(yīng)用構(gòu)建和支撐方式 Copyright 2013 Pivotal. All rights reserv

3、ed.CONSUMER GRADE先行者: 服務(wù)于海量客戶的互聯(lián)網(wǎng)巨頭企業(yè)MAINFRAMESoftware-Defined DatacenterNew Data-fabrics新的客戶體驗新的業(yè)務(wù)模式由服務(wù)于海量客戶的互聯(lián)網(wǎng)巨頭企業(yè)引領(lǐng)潮流CLIENT-SERVER & WEBSoftware-Defined DatacenterNew Data-fabrics私有的數(shù)據(jù)中心與基礎(chǔ)架構(gòu) Copyright 2013 Pivotal. All rights reserved.CONSUMER GRADE先行者: 服務(wù)于海量客戶的互聯(lián)網(wǎng)巨頭企業(yè)MAINFRAMESoftware-Defi

4、ned DatacenterNew Data-fabrics新的客戶體驗新的業(yè)務(wù)模式由服務(wù)于海量客戶的互聯(lián)網(wǎng)巨頭企業(yè)引領(lǐng)潮流CLIENT-SERVER & WEBSoftware-Defined DatacenterNew Data-fabrics私有的數(shù)據(jù)中心與基礎(chǔ)架構(gòu) Copyright 2013 Pivotal. All rights reserved.CONSUMER GRADE先行者: 服務(wù)于海量客戶的互聯(lián)網(wǎng)巨頭企業(yè)MAINFRAMESoftware-Defined DatacenterNew Data-fabrics新的客戶體驗新的業(yè)務(wù)模式由服務(wù)于海量客戶的互聯(lián)網(wǎng)巨頭企業(yè)引

5、領(lǐng)潮流CLIENT-SERVER & WEBSoftware-Defined DatacenterNew Data-fabrics私有的數(shù)據(jù)中心與基礎(chǔ)架構(gòu) Copyright 2013 Pivotal. All rights reserved.CONSUMER GRADE先行者: 服務(wù)于海量客戶的互聯(lián)網(wǎng)巨頭企業(yè)MAINFRAMESoftware-Defined DatacenterNew Data-fabrics新的客戶體驗新的業(yè)務(wù)模式由服務(wù)于海量客戶的互聯(lián)網(wǎng)巨頭企業(yè)引領(lǐng)潮流CLIENT-SERVER & WEBSoftware-Defined DatacenterNew Da

6、ta-fabrics私有的數(shù)據(jù)中心與基礎(chǔ)架構(gòu) Copyright 2013 Pivotal. All rights reserved.來自企業(yè)的獨有需求 MAINFRAMECLIENT-SERVER & WEBCONSUMER GRADE新的客戶體驗新的業(yè)務(wù)模式由服務(wù)于海量客戶的互聯(lián)網(wǎng)巨頭企業(yè)引領(lǐng)潮流大量設(shè)備大量設(shè)備, 海量日志海量日志, 多種應(yīng)多種應(yīng)用用無處不在的數(shù)據(jù)采集無處不在的數(shù)據(jù)采集Software-Defined DatacenterNew Data-fabrics Copyright 2013 Pivotal. All rights reserved.來自企業(yè)的獨有需求 M

7、AINFRAMECLIENT-SERVER & WEBCONSUMER GRADE新的客戶體驗新的業(yè)務(wù)模式由服務(wù)于海量客戶的互聯(lián)網(wǎng)巨頭企業(yè)引領(lǐng)潮流大量設(shè)備大量設(shè)備, 海量日志海量日志, 多種應(yīng)多種應(yīng)用用無處不在的數(shù)據(jù)采集無處不在的數(shù)據(jù)采集Software-Defined DatacenterNew Data-fabrics Copyright 2013 Pivotal. All rights reserved.來自企業(yè)的獨有需求 MAINFRAMECLIENT-SERVER & WEBCONSUMER GRADE新的客戶體驗新的業(yè)務(wù)模式由服務(wù)于海量客戶的互聯(lián)網(wǎng)巨頭企業(yè)引領(lǐng)潮流大

8、量設(shè)備大量設(shè)備, 海量日志海量日志, 多種應(yīng)多種應(yīng)用用無處不在的數(shù)據(jù)采集無處不在的數(shù)據(jù)采集Software-Defined DatacenterNew Data-fabrics Copyright 2013 Pivotal. All rights reserved.來自企業(yè)的獨有需求 MAINFRAMECLIENT-SERVER & WEBCONSUMER GRADE新的客戶體驗新的業(yè)務(wù)模式由服務(wù)于海量客戶的互聯(lián)網(wǎng)巨頭企業(yè)引領(lǐng)潮流大量設(shè)備大量設(shè)備, 海量日志海量日志, 多種應(yīng)多種應(yīng)用用無處不在的數(shù)據(jù)采集無處不在的數(shù)據(jù)采集Software-Defined DatacenterNew Da

9、ta-fabrics.ETC Copyright 2013 Pivotal. All rights reserved.New ExperiencesNew Biz Modelspioneered by new Consumer Internet giantsInternet-of-thingsPervasive telemetryMAINFRAMECLIENT-SERVER & WEBCONSUMER GRADESoftware-Defined DatacenterNew Data-fabrics.ETC上述因素促成了對新平臺的需求 Copyright 2013 Pivotal. Al

10、l rights reserved. 典型目標場景 需要新的消費級支撐能力 通過遙測與遙控實現(xiàn)智能工業(yè)控制與維護診斷 實時電信應(yīng)用場景 未來新零售場景 大規(guī)模智能農(nóng)業(yè)應(yīng)用 還有許多其他應(yīng)用場景A NEW PLATFORM FOR A NEW ERA Copyright 2013 Pivotal. All rights reserved.新一代企業(yè)級 PaaS 的設(shè)計準則 與當前各種 IaaS 平臺配合工作 不論是私有的還是公有的 充分關(guān)注新一代開發(fā)人員和新一代應(yīng)用的需求 為有效進行實時大容量信息處理而設(shè)計 同時兼顧與傳統(tǒng)應(yīng)用的互操作性 Copyright 2013 Pivotal. All r

11、ights reserved.ETC支持開放標準并與開源有效互動一數(shù)據(jù)為中心不與特定云平臺綁定針對開發(fā)者的需求針對企業(yè)的需求Cloud FabricData FabricApplication FabricScale-out storage: HDFS/ObjectLanguages& FrameworksIngest & Query: very high-capacity & in-memoryAnalyticsServicesCloud Abstraction (portability)Automation: App Provisioning & Life-

12、cycleService RegistryPivotal One: 重新定義企業(yè)級重新定義企業(yè)級 PaaS Copyright 2013 Pivotal. All rights reserved.Cloud FabricData FabricApplication FabricScale-out storage: HDFS/ObjectLanguages& FrameworksIngest & Query: very high-capacity & in-memoryAnalyticsServicesCloud Abstraction (portability)Auto

13、mation: App Provisioning & Life-cycleService RegistryPivotal One: 傳承傳承vFabricGemFirePivotal ConfidentialInternal Use OnlyROB MEE & TEAMAgile & RADHadoopPIVOTAL ONE HERITAGECloudJava/SpringOUTSTANDING TECHNICAL TEAM+700 ENGINEERSIMDGWeb/MobileData Science & AnalyticsMPP SQLCHRIS BEAMS

14、JUERGEN HOELLERNOELLE SIOKAUSHIK DASDAN HOLLEHUYLA EMIR-FARINASGAVIN SHERRYLUKE LONERGANEDWARD HIEATTALEXIS RICHARDSONJAMES WATTERSJAGS RAMNARAYANMILIND BHANDARKARAPURVA DESAIADRIAN COLYERSUDHIR MENONCHRISTOS TRYFONASPivotal ConfidentialInternal Use OnlyCLOUDFABRICELISABETH HENDRICKSONMATTHEW KOCHER

15、JAMES BAYERJEFF NICKJAMES WATTERSDAVID STEVENSONPivotal ConfidentialInternal Use OnlyDATA FABRICDAN HOLLELUKE LONERGANMILIND BHANDARKARAPURVA DESAIGAVIN SHERRYCHRISTOS TRYFONASJAGS RAMNARAYANSUDHIR MENONPivotal ConfidentialInternal Use OnlyAPPLICATIONFABRICCHRIS BEAMSJUERGEN HOELLERALEXIS RICHARDSON

16、ADRIAN COLYERMATTHIAS RADESTOCKSALVATORE SANFILIPPO Copyright 2013 Pivotal. All rights reserved.Cloud Abstraction (portability)Automation: App Provisioning & Life-cycleService Registry基于 Cloud Foundry開放并以開源方式實現(xiàn): C設(shè)計目標:-可移植性-可擴展性-自動化-彈性不斷成長的生態(tài)系統(tǒng)Pivotal One: 云支撐架構(gòu)云支撐架構(gòu)Cloud FabricDa

17、ta Fabric.ETCApplication FabricCloud Abstraction (portability)Automation: App Provisioning & Life-cycleService Registry Copyright 2013 Pivotal. All rights reserved.Cloud FabricData & Analytics Fabric.ETCPivotal One: 數(shù)據(jù)支撐架構(gòu)Application FabricScale-out storage: HDFS/ObjectCatalogResource MgmtDa

18、taMgmtTXNProcessHAWQMPPRealTimeDBMap ReduceHbaseK/V統(tǒng)一基于 Hadoop通用的全局分布式集群文件系統(tǒng) (HDFS)與高級數(shù)據(jù)處理服務(wù)緊密集成-MPP SQL-實時交易處理-流式數(shù)據(jù)處理原生設(shè)計 PaaS 設(shè)計并作為 Pivotal One 的有機組成部分: 支持彈性伸縮與多租戶環(huán)境Analytics Suite: Chorus/MADlibGPDBRDB Copyright 2013 Pivotal. All rights reserved.HAWQ: 性能測試結(jié)果對比4.21988.71612.04152.71,2852.81,81547X

19、19X208X476X648X4.2378.75962.0502.7552.8599X69X25X20X21XPivotal HDPivotal HD Copyright 2013 Pivotal. All rights reserved.ETC有效支撐快速應(yīng)用開發(fā)簡化在開發(fā)，測試和生產(chǎn)環(huán)境間的部署和遷移支持多種語言和開發(fā)框架下的自動部署和自動擴展業(yè)界領(lǐng)先的 Java/Spring 運行時支撐松耦合的即時服務(wù)綁定缺省提供運行時數(shù)據(jù)的分析和可視化監(jiān)控功能Pivotal One: 應(yīng)用支撐架構(gòu)Cloud FabricData FabricApplication FabricLanguages&am

20、p; FrameworksAnalyticsServices Copyright 2013 Pivotal. All rights reserved.Cloud FabricData FabricApplication FabricData & AnalyticsApp DevCloud Automation Pivotal HD GemFire GPDB Spring vFabric Cloud FoundryServices Pivotal Labs & Data Science LabsPivotal One: 切入點切入點PIVOTAL ONEPLATFORM Copy

21、right 2013 Pivotal. All rights reserved.Pivotal 專業(yè)服務(wù): 迅速實現(xiàn)價值 Pivotal Labs: 快速開發(fā)和部署新應(yīng)用 經(jīng)過廣泛證明的方法論降低風險同時加速開發(fā)過程Pivotal Data Science Labs: 經(jīng)過檢驗的的數(shù)據(jù)科學(xué)實踐加速分析項目實施 用數(shù)據(jù)分析來驅(qū)動業(yè)務(wù)價值實現(xiàn)Open Source Support: 協(xié)作式客戶驅(qū)動開源支持，服務(wù)和共同研發(fā)Pivotal Data Science Labs Copyright 2013 Pivotal. All rights reserved.關(guān)鍵時間點 2012 戰(zhàn)略規(guī)劃階段:-四月: 形成正式構(gòu)想-十二月: 構(gòu)想在內(nèi)部發(fā)布 2013 正式發(fā)布與推進:-二月: Pivotal HD 正式發(fā)布-四月: 公司完成正式注冊-第二季度: Cloud Foundry on VMware Cloud & AWS-第四季度: Pivotal One GAA NEW PL