湖倉(cāng)一體全面開(kāi)啟實(shí)時(shí)化時(shí)代

DatawarehousesApplicationsDatabaseETLDatawarehousesApplicationsDatabaseETLDataExploreETLDatawarehousesstructured,semi-structuredandunstructuredDataRealtimeAnalyticsMachineDatasciencestructuredstructured,semi-structuredandunstructuredDataAnalyticsMachineLearningDatascienceAlakeformatthatenablesbuildingaRealtimeLakehouseArchitecturewithFlinkandSparkforbothstreamingandbatchoperationsStreamingIngestionstreamingstreamingstreaming001011101010101010101010101001001回阿里巴巴集團(tuán)數(shù)據(jù)業(yè)務(wù)回阿里巴巴集團(tuán)數(shù)據(jù)業(yè)務(wù)RealtimeComputeMaxComputeHologresE-MapReduce001011011010101010101010101010010ThanksApachePaimon統(tǒng)??數(shù)據(jù)湖存儲(chǔ)底座李勁松/ApachePaimonPMCChair010102020303040401 Warehouse:天級(jí) StreamingLakehouse:分鐘級(jí) Warehouse:天級(jí) StreamingLakehouse:分鐘級(jí) LakehouseLakehouse:天級(jí)/小時(shí)級(jí)實(shí)時(shí)數(shù)據(jù)湖02流式數(shù)據(jù)湖03國(guó)人主導(dǎo)社區(qū)03000250020005000250020005000構(gòu)建流式鏈路構(gòu)建流式鏈路02一套架構(gòu)，三種計(jì)算StreamingIngestionstreamingstreamingstreamingBatchBatchBatch001011101010101010101010101001001流流批批03擁有全部能力的是Flink，是star04454545進(jìn)展進(jìn)展ServerlessServerless·所有計(jì)算引擎打通paimon達(dá)成state-of-Art的性能·結(jié)合Flink構(gòu)建流批一體的streamingLakehouse·統(tǒng)一存儲(chǔ)，按照不同需求使用不同的計(jì)算引擎MaxComputeSparkServerlessMaxComputeSparkServerlessFlinkServerlessStarRocks010101010101011010101010010101001011101·查詢腳本：開(kāi)發(fā)過(guò)程查詢數(shù)據(jù)及系到最小5秒內(nèi)返回·作業(yè)運(yùn)維：支持啟動(dòng)點(diǎn)位的設(shè)置；面向stopwithsavepoint批調(diào)度：支持基本的批工作流與調(diào)度，以支撐簡(jiǎn)單的數(shù)據(jù)回刷查詢查詢...);dw.order_dw.dwd_ordersSELECT……查詢性能：內(nèi)置Jar&向量化讀取寫(xiě)入能力：完整支持寫(xiě)入及覆蓋寫(xiě)入Thanks010102020303TransactionsAlgorithmdata…DatabaseFileSystem…OfflineOfflineSystemStreamProcessingBatchProcessingFileSystem…SearchEngineAdvertisingEngineRecommendationEngineSampleEngine…應(yīng)用規(guī)模增量延遲應(yīng)用規(guī)模增量延遲支持雙11作業(yè)規(guī)模 ……解決Lambda架構(gòu)缺點(diǎn) —//--………PaimonTableTable1Table2Cache消息隊(duì)列FlinkBatchJobHTTP…PaimonTableTable1Table2Cache消息隊(duì)列FlinkBatchJobHTTP 作為Cache使用TableTablen…所以分了十多張KV表存圖片Cache，用戶使用不便。二、新鏈路預(yù)期收益所以不存在Build鏈路不穩(wěn)定的情況?！瑼ppnApp1…AppAppnApp1…AppnApp2…AppApp2PaimonTablePaimonTableScanSnapshot-1ScanSnapshot-1Snapshot-3Snapshot-3Snapshot-2AppApp11、并發(fā)有上限限制，吞吐受限，而盲目加并發(fā)有拉掛庫(kù)的風(fēng)險(xiǎn)。2、核心庫(kù)拉取時(shí)間只能晚上。二、新鏈路預(yù)期收益ScanSnapshot-nScanSnapshot-1…ScanSnapshot-nScanSnapshot-1……/confluence/display/PAIMO……………FilesFilesFilesTableschainFilesrebalancePickCloneTables…FilesFilesFilesTableschainFilesrebalancePickCloneTables（database.table）Database.Table…Recreate…RecreateSnapshotRecreateRecreateSnapshotRecreateRecreateSnapshotparallelism=nThanks010102020303…01螞蟻Paimon應(yīng)用場(chǎng)景u簡(jiǎn)化研發(fā)模式,降低研發(fā)成本u節(jié)約了80%的存儲(chǔ)資源以及70%的計(jì)算資源u在最后?批發(fā)獎(jiǎng)的過(guò)程中，節(jié)約了1.2?時(shí)的核對(duì)時(shí)間，提?了