第五屆云計(jì)算大會(huì)-應(yīng)用驅(qū)動(dòng)的數(shù)據(jù)中心計(jì)算

應(yīng)用驅(qū)動(dòng)的數(shù)據(jù)中心計(jì)算網(wǎng)頁庫倒排表WWeebbPagesDataMiningQuery

ServingCrawlingIndexBuilding從搜索引擎講起…StreamBlock0Block

1…Block

XIn-Memory

Records{<key,

data>}Log

Block-NLog

Block-1New

streamBlock

0Block

1…Block

YMemoryDiskUpdateQueryDumpCommit…Log-based

結(jié)構(gòu)Block

I/O批量Commit流式讀寫高吞吐的存儲(chǔ)模型BlockBlock@disk0Block@disk1Block@diskN<key,data>MemoryDiskdump…Block…通過Blocks形成一個(gè)大的“文件”最大化并行度去RAID，暴露裸盤Direct

I/O無文件系統(tǒng)高吞吐的存儲(chǔ)結(jié)構(gòu)實(shí)現(xiàn)Data1Data2…DataNLogRecord…LogXLog1DatablocksData2’ …New

datablocksMerge-sortDump通過Merge-Sort生成Sorted

StreamMemoryDiskData1’DataM’Block-N…Block-2Block-1Indexstreamkey->addrMod

stream<key,

mod>Patch

stream<addr,mod>appendmergemergegenerateBasestream通過幾個(gè)Sorted

Stream組成一個(gè)TableBlockBase

StreamModStreamTableBlockBlockBlockBlockBlock…Index

StreamPatch

Stream一個(gè)典型存儲(chǔ)系統(tǒng)的三層結(jié)構(gòu)在哪個(gè)層次做Partition？如何選擇Replication粒度？復(fù)雜度

(管理

實(shí)現(xiàn))數(shù)據(jù)交換與重組的開銷可靠性

(Reliability)變成分布式系統(tǒng)壞處：3倍Commit開銷好處：本地I/OBlockBaseStreamModStreamTable… …Replica

1BlockIndexStreamPatchStreamBlockBaseStreamModStreamTable… …Replica

2BlockIndexStreamPatchStreamBlockBaseStreamModStreamTable… …Replica

3BlockIndexStreamPatchStream方案1Replica

3 Replica

1好處：1倍Commit開銷壞處：網(wǎng)絡(luò)I/OBase

StreamBlock…Replica

1Block…Replica

2Block…Replica

3Mod

StreamBlock…Replica

1Block…Replica

2Block…Replica

3IndexStreamBlock…Replica

1Block…Replica

2Block…PatchStreamBlock…Block…Replica

2Block…Replica

3方案2還有一些結(jié)構(gòu)需要考慮Data1Data2…DataNLogRecordDumpMemoryDiskLogX…Log1DatablocksMerge-sortData2’ …New

datablocksData1’DataM’統(tǒng)一還是分開？存儲(chǔ)方式？Logical

Layer

TableStream

BlockPhysical

BoundaryDatacenterClusterRackNodePhysical

Layer

Memory

FlashDisk從邏輯結(jié)構(gòu)到物理結(jié)構(gòu)的映射衍化出多種系統(tǒng)架構(gòu)Software-Defined

Hardware去RAID，暴露裸盤新SSDRack-Scale

Architecture新SSDRack-Scale

Architecture物理集成共享電源風(fēng)扇高速全互聯(lián)

共享總線可重組部件整個(gè)數(shù)據(jù)中心是一臺(tái)計(jì)算機(jī)應(yīng)用驅(qū)動(dòng)，軟硬件協(xié)同創(chuàng)新30年P(guān)C時(shí)代終結(jié)單用戶多任務(wù)

多用戶單任務(wù)規(guī)模、成本交付時(shí)間從單機(jī)到數(shù)據(jù)中心:環(huán)境與設(shè)計(jì)目標(biāo)的變革透明性不再成為美德層次化向豎井式演進(jìn)Application-Driven,

Software-Defined從單機(jī)到數(shù)據(jù)中心:設(shè)計(jì)理念的變革ArchitectureBIOSKernelApplications0

bug

CorrectnessFabricHost

OSInfrastructureServices/