邵賽賽周康：AGI時(shí)代統(tǒng)一數(shù)據(jù)目錄的設(shè)計(jì)與實(shí)踐

演講人：邵賽賽演講人：周康A(chǔ)pacheGravitino創(chuàng)始人 實(shí)驗(yàn)的跟蹤和可視化打包、運(yùn)行ML流程作業(yè)超參優(yōu)化打包、運(yùn)行ML流程作業(yè)超參優(yōu)化評(píng)估GenAI應(yīng)用訓(xùn)練，調(diào)優(yōu)，及生產(chǎn)化監(jiān)控、調(diào)試LLM評(píng)估GenAI應(yīng)用訓(xùn)練，調(diào)優(yōu)，及生產(chǎn)化開發(fā)GenAI應(yīng)用AutomationAutomation自動(dòng)觸發(fā)流程發(fā)布和分享模型及數(shù)據(jù)集自動(dòng)觸發(fā)流程ReportsArtifactsDataReportsQ不同組織之間分享模型、數(shù)據(jù)集數(shù)據(jù)集、模型生不同組織之間分享模型、數(shù)據(jù)集數(shù)據(jù)集、模型生命周期管理數(shù)據(jù)集、模型治理及權(quán)限控制https://www.llamaindex.ai/blog/introducing-llamacloud-and-llamaparse-af8cedf9006b下------------、karkakarka數(shù)據(jù)連接ConnecttotheDataThatMatters數(shù)據(jù)發(fā)現(xiàn)Automaticallyfind,classify,數(shù)據(jù)連接ConnecttotheDataThatMatters數(shù)據(jù)發(fā)現(xiàn)Automaticallyfind,classify,andmapallofyourdata-everywhere.數(shù)據(jù)主權(quán)Automateend-to-enddatarightsrequestsandreporting.3rdParty數(shù)據(jù)主權(quán)Automateend-to-enddatarightsrequestsandreporting.數(shù)據(jù)分類AutomaticallyAutomaticallyclassifymoretypesofdatainmoreplaces.元數(shù)據(jù)增強(qiáng)Enrichtechnicalmetadatawith元數(shù)據(jù)增強(qiáng)Enrichtechnicalmetadatawithbusinessandoperationalmetadataforfullvisibility.數(shù)據(jù)生命周期管理Simplifyandautomatedatalifecyclemanagementfromcollectiontodestruction.–HiveHiveMetastoreBuilt-inBuilt-inCatalogSchemaSchemaRegistryRegistryRegistry消息隊(duì)列數(shù)據(jù)倉庫數(shù)據(jù)湖機(jī)器學(xué)習(xí)消息隊(duì)列數(shù)據(jù)倉庫數(shù)據(jù)湖機(jī)器學(xué)習(xí)●數(shù)據(jù)統(tǒng)一視圖●從元數(shù)據(jù)層面上達(dá)到SSOT●統(tǒng)一訪問和治理Next-GenDataCatalogistheCoreinNewOpenUnifiedAccessingUnifiedManagingGravitinoUnifiedAccessingUnifiedManagingGravitinoWarehouseGravitinoGravitinoGravitinoGravitinoFunctionalitylayerInterfacelayerCorewithobjectmodelConnectionLayerMetadataMetadataStorageUnifiedUnifiedProcessingUnifiedGoverningUnifiedUnifiedRESTAPIsIcebergIcebergRESTAPIsCatalogCatalogCatalogSchemaSchemaSchemaTableModelFilesetMetalakeCatalogCatalogCatalogSchemaSchemaSchemaTableModelFilesetMetalakeCatalogCatalogSchemaSchemaTopicTopicAlterAlter…}….}ArrowFileSystemArrowFileSystemAlterADLSADLS…}storageLocation:str….}Spark/Spark/Flink/Presto對(duì)接統(tǒng)一元數(shù)據(jù)完善用戶審計(jì)轉(zhuǎn)換底層權(quán)限隱藏認(rèn)證憑據(jù)在線數(shù)據(jù)庫數(shù)據(jù)湖表格OLAP系統(tǒng)對(duì)接機(jī)器學(xué)習(xí)框架傳統(tǒng)機(jī)器學(xué)習(xí)對(duì)接機(jī)器學(xué)習(xí)框架傳統(tǒng)機(jī)器學(xué)習(xí)大模型預(yù)訓(xùn)練/微調(diào)流程打通、產(chǎn)品打通DataOps/MLOps/LLMOps非結(jié)構(gòu)化數(shù)據(jù)資產(chǎn)地圖與資產(chǎn)治理存量數(shù)據(jù)納管支持設(shè)置TTL與TTV數(shù)據(jù)處理流程AI流程數(shù)據(jù)加工spark/Flink數(shù)據(jù)分析OLAP數(shù)據(jù)探查數(shù)據(jù)加工spark/Flink數(shù)據(jù)分析OLAP數(shù)據(jù)探查Notebook模型訓(xùn)練LLM模型部署數(shù)據(jù)集成集成引擎根據(jù)血緣推薦TTL、TTV管理前降本40%管理后特征分析特征分析pytorch模型訓(xùn)練pandaspytorch模型訓(xùn)練TalosModel模型部署數(shù)據(jù)落倉提取特征模型部署數(shù)據(jù)落倉lcebergFileset模型校驗(yàn)lcebergFilesetModelModel數(shù)據(jù)分析