并行數(shù)據(jù)庫技術(shù)分析及未來展望

并行數(shù)據(jù)庫技術(shù)分析及未來展望Agenda并行數(shù)據(jù)庫技術(shù)要點分析未來展望Paralleldatabasesystemseekstoimproveperformancethroughparallelizationofvariousoperationssuchasloadingdata,buildingindexes,andevaluatingqueriesbyusingmultipleCPUsandDisksinParallel.[1]MotivationParallelmachinesarebecomingquitecommonandaffordablePricesofmicroprocessors,memoryanddiskshavedroppedsharplyDatabasesaregrowingincreasinglylargelargevolumesoftransactiondataarecollectedandstoredforlateranalysis.multimediaobjectslikeimagesareincreasinglystoredindatabases并行數(shù)據(jù)庫

ImprovesResponseTime.

INTERQUERYPARALLELISM

Itispossibletoprocessanumberoftransactionsinparallelwitheachother.

ImprovesThroughput.

INTRAQUERYPARALLELISMItispossibletoprocess‘sub-tasks’ofatransaction

inparallelwitheachother.并行數(shù)據(jù)庫–目標(1)

Speed-UpAddingmoreresourcesresultsinproportionallylessrunningtimeforafixedamountofdata.10secondstoscanaDBof10,000recordsusing1CPU1secondtoscanaDBof10,000recordsusing10CPUs

Scale-Up(transaction/data)Ifresourcesareincreasedinproportiontoanincreaseindata/problemsize,theoveralltimeshouldremainconstant1secondtoscanaDBof1,000recordsusing1CPU

1secondtoscanaDBof10,000recordsusing10CPUs并行數(shù)據(jù)庫–目標(2)Shared-MemoryandShared-DiskArchitecturesShared-Memory:allprocessorsshareacommonmainmemoryandsecondarymemoryLoadbalancingisrelativelyeasytoachieve,butsufferfrommemoryandbuscontentionShared-Disk:allprocessors,eachofwhichhasitsownlocalmainmemory,sharethedisks并行數(shù)據(jù)庫-體系結(jié)構(gòu)(1)Shared-NothingArchitectureEachprocessorhasitsownlocalmainmemoryanddisksLoadbalancingbecomesdifficult并行數(shù)據(jù)庫-體系結(jié)構(gòu)(2)Shared-SomethingArchitectureAmixtureofshared-memoryandshared-nothingarchitecturesEachnodeisashared-memoryarchitectureconnectedtoaninterconnectionnetworkalashared-nothingarchitecture并行數(shù)據(jù)庫-體系結(jié)構(gòu)(3)元數(shù)據(jù)管理查詢分析和執(zhí)行器分布式事務(wù)分布式鎖并發(fā)控制副本數(shù)據(jù)同步并行數(shù)據(jù)庫–功能模塊Agenda架構(gòu)特點技術(shù)要點分析未來展望2/5/2023111.ParallelDB/D.S.Jagli111.ParallelDB/D.S.Jagli技術(shù)要點–層次結(jié)構(gòu)MasterServer:外部App訪問入口集群元數(shù)據(jù)管理SQLParser生成執(zhí)行計劃和任務(wù)調(diào)度2/5/2023121.ParallelDB/D.S.Jagli12Range

HashRoundRobinA...EF...JK...NO...ST...ZA...EF...JK...NO...ST...ZA...EF...JK...NO...ST...ZGoodforequijoins,exact-matchqueries,andrangequeriesGoodforequijoins,exactmatchqueriesGoodtospreadload1.ParallelDB/D.S.Jagli技術(shù)要點–數(shù)據(jù)分區(qū)2/5/2023131.ParallelDB/D.S.Jagli131.ParallelDB/D.S.Jagli技術(shù)要點–存儲和索引(1)列存儲(columnstore)數(shù)據(jù)按列存儲，每一列單獨存放只訪問查詢涉及的列，大量降低I/O數(shù)據(jù)類型一致，方便壓縮數(shù)據(jù)包建索引，數(shù)據(jù)即索引行存儲(rowstore)數(shù)據(jù)是按行存儲的 沒有索引的查詢使用大量I/O建立索引和物化視圖需要花費大量時間和資源面對查詢的需求，數(shù)據(jù)庫必須被大量膨脹才能滿足性能要求。技術(shù)要點–存儲和索引(2)混合存儲(hybridstore)在大尺度上數(shù)據(jù)是按行存儲，在小尺度上數(shù)據(jù)是按列存儲在大尺度上以行作為任務(wù)切分，在小尺度上以列進行I/O優(yōu)化技術(shù)要點–存儲和索引(3)模糊索引精確索引

B+樹

LSM

COLA3.列存索引(Projection）

技術(shù)要點–查詢引擎技術(shù)要點–分布式事務(wù)(1)技術(shù)要點–分布式事務(wù)(2)技術(shù)要點–任務(wù)調(diào)度與資源管理技術(shù)要點–高可用技術(shù)要點–集群擴展數(shù)據(jù)庫對一致性的天然要求數(shù)據(jù)庫元數(shù)據(jù)管理數(shù)據(jù)分布的再平衡技術(shù)要點–備份和災(zāi)難恢復(fù)在線增量備份。支持HDFS作為備份存儲介質(zhì)，HDFS上的備份數(shù)據(jù)可通過Hadoop平臺直接進行分析。以異地災(zāi)備替代常規(guī)備份操作。異地數(shù)據(jù)中心之間實現(xiàn)同步/異步操作。備份和恢復(fù)兩個過程，以及異地災(zāi)備中的主從集群可實現(xiàn)異構(gòu)恢復(fù)。技術(shù)要點–SSD應(yīng)用場景數(shù)據(jù)量大，熱點數(shù)據(jù)無法全部在緩存中。用戶訪問I/O量大，造成系統(tǒng)響應(yīng)慢系統(tǒng)整體壓力低，QPS值無法提高系統(tǒng)切換后，預(yù)熱時間長等于故障Agenda并行數(shù)據(jù)庫技術(shù)要點分析未來展望未來展望–并行數(shù)據(jù)庫在大數(shù)據(jù)基礎(chǔ)設(shè)置中的定位(1)適應(yīng)已有的數(shù)據(jù)庫生態(tài)，平滑替代無法跟上快速增長的數(shù)據(jù)量的傳統(tǒng)EDW，連接BI工具。EDW工具的復(fù)雜性和對復(fù)雜SQL的支持程度，超過了目前Hadoop的能力。IngestLandingandAnalyticsSandboxZoneIndexes,facetsHive/HBaseColStoresDocumentsInVarietyofFormatsAnalyticsMapReduceIngestionandReal-timeAnalyticZoneDataSinksFilter,TransformIngestCorrelate,ClassifyExtract,AnnotateWarehousingZoneEnterpriseWarehouseDataMartsQueryEnginesCubesDescriptive,PredictiveModelsModelsWidgetsDiscovery,VisualizerSearchAnalyticsandReportingZoneMetadataandGovernanceZoneConnectors未來展望–并行數(shù)據(jù)庫與Hadoop未來展望–并行數(shù)據(jù)庫與Hadoop集成(SQL-on-HadoopGoogle,Mega，Spanner和F1向我們展現(xiàn)了跨數(shù)據(jù)中心數(shù)據(jù)庫的可能。Cloudera開源了大數(shù)據(jù)查詢分析引擎。ImpalaHortonworks開源了StingerFackbook開源了Presto類似Pregel，UCBerkeleyAMPLAB實驗室開發(fā)了Spark圖計算框架，并以Spark為核心開源了大數(shù)據(jù)查詢分析引擎Shark。EMC的PivotalHAWQHadapt的HybridStorageEngine未來展望–并行數(shù)據(jù)庫與Hadoop集成(Connector)未來展望–并行數(shù)據(jù)庫與Hadoop集成(Hybrid)未來展望–IT基礎(chǔ)設(shè)施的變化新的IT基礎(chǔ)設(shè)置，如企業(yè)云環(huán)境，一體機，存儲平臺SDS(SoftwareDefineStorage)都解決了傳統(tǒng)MPP數(shù)據(jù)庫中的存儲高可用問題。新一代的MPP數(shù)據(jù)庫，將不再需要在MPP內(nèi)部進行存儲層副本的管理和高可用機制，這將極大簡化MPP實現(xiàn)的復(fù)雜度。MPP數(shù)據(jù)庫將更傾向于具有事務(wù)能力的SQLSuperComputer，以及通過對存儲層的適配，達到最高性能。

Figure5.Tenant-relatedfactorsandhowtheyaffect“

isolatedversusshared"dataarchitecturedecisions未來展望–IT基礎(chǔ)設(shè)施的變化（多租戶）未來展望–并行數(shù)據(jù)庫技術(shù)的發(fā)展實現(xiàn)跨不同地理位置數(shù)據(jù)中心數(shù)據(jù)庫。例如,Google的Mega加強數(shù)據(jù)庫內(nèi)部的資源管理機制和任務(wù)的優(yōu)先級調(diào)用實現(xiàn)完整的事務(wù)和并發(fā)控制能力。擴展支持應(yīng)用類型的范圍，逐漸模糊NOSQL與SQL的不同。例如，Vertica中的FlexEngine支持類似MongoDB可動態(tài)調(diào)整的基于非循環(huán)有向圖的具有容錯