分布式鎖服務(wù)-debby的設(shè)計與實(shí)現(xiàn)

1、分布式鎖服務(wù)-debby的設(shè)計與實(shí)現(xiàn)Ifox小組：單棟棟，趙東升， 樊 楷，蘇 飛主要內(nèi)容 Debby系統(tǒng)整體設(shè)計 服務(wù)器端設(shè)計與實(shí)現(xiàn) 數(shù)據(jù)存儲的設(shè)計與實(shí)現(xiàn) 客戶端設(shè)計與實(shí)現(xiàn) 容錯日志(paxos)的設(shè)計與實(shí)現(xiàn)系統(tǒng)的整體結(jié)構(gòu)Debby server實(shí)現(xiàn) 服務(wù)器和客戶端的通信 一致性的保證 文件、目錄的實(shí)現(xiàn) Session的實(shí)現(xiàn) 事件（Event）管理的實(shí)現(xiàn) SnapShot服務(wù)器和客戶端的通信 用戶調(diào)用客戶端庫于服務(wù)器通信 通過ICE遠(yuǎn)程過程調(diào)用實(shí)現(xiàn) 提供的接口connect, close, keepAlive,addEventgetData, setData, create, mkdir,

2、remove, isDir, exists .服務(wù)器一致性的保證 調(diào)用底層Paxos協(xié)議 對文件的操作時，把操作提交給Paxos Paxos保證在3臺服務(wù)器上操作的一致性 Paxos提供的接口sendProposal() Session 的實(shí)現(xiàn) 服務(wù)器維護(hù)一個Debby管理器 Session通過KeepAlive來保證 每個KeepAlive會捎帶事件信息 KeepAlive:客戶端等待，服務(wù)器受到請求立即返回文件、目錄的實(shí)現(xiàn) 文件、目錄放在內(nèi)存 常規(guī)文件系統(tǒng)和臨時文件系統(tǒng) 常規(guī)文件系統(tǒng)map MemDir 用tree.hh實(shí)現(xiàn) 臨時文件系統(tǒng)map 事件管理的實(shí)現(xiàn) Debby維護(hù)了一個事件管理

3、器 已注冊的事件和已發(fā)生的事件 對于已注冊的事件，系統(tǒng)維護(hù)一個事件到handle列表的map 當(dāng)心跳發(fā)生時，將發(fā)生的事件返回給訂閱的客戶SnapShot 只用log恢復(fù)服務(wù)器帶來的問題：日志將會越來越多恢復(fù)時間越來越長 本系統(tǒng)采用snapshot(快照)機(jī)制解決此問題SnapShot 將內(nèi)存中的文件系統(tǒng)數(shù)據(jù)結(jié)構(gòu)直接序列化到磁盤上 Snapshot過程執(zhí)行成功后，比snapshot備份時間早的log信息不再需要，可通知paxos將log刪除。SnapShot SnapShot方法增加了額外的復(fù)雜性實(shí)現(xiàn)SnapShot之前，crush掉的服務(wù)器只需從其他機(jī)器獲得最近的log即可進(jìn)行恢復(fù)。 實(shí)現(xiàn)Sn

4、apShot之后，需同時考慮log和snapshot信息。SnapShot class SnapShot private static string DIR_PATH; public static void serialize(MemDir& md); public MemDir& void Unserialize(); Debby Client00448161APIvoid create(const string &path, bool ephemeral)void mkdir(const string &path)void remove(const stri

5、ng &path)bool exists(const string &path)bool isdir(const string &path)vector list(const string &path)bool lock(const string &path, bool share)void release(const string &path)string read(const string &path)void write(const string &path, const string &content)void r

6、egcb(const string &path, EventType e, shared_ptr cb)void clearcb(const string &path)Lock Server dont support lock directly, client use ephemeral file to implement lock service. When client obtain a lock on file, create filename.lck ephemeral file. If file already exists, server would throw a

7、 exception, and client returns failure. When client release the lock, simply delete the file.Lock(2) When client lose connection with server, ephemeral file is deleted, including those indicate locks, thus locks is released. To prevent ambiguity, user file is not allowed to end with “”, so they are

8、easy to be differentiated from files used to implement locks.Events EventType EventCreated EventRemoved EventChanged EventLockChanged EventArbitrary All event would apply on both directories and filesEvents(2) All callbacks are managed by client, when a callback is first registered on a event, the c

9、lient registers the event on master. When a client receives a event, it invoke all callbacks registered on the event. User could cancel all callbacks on a certain path, and client would unregister events on server.Event(3) Client supply a Callback class to implement use callback, it contains a pure

10、virtual function run(). User implement their own class based on Callback, and implement the run() function. User could save any necessary information in the class. When client invoke a Callback, it create a thread to invoke the run() function.Choose Server There a 5 server in a debby cell, while onl

11、y one of them is the master. Client use ICE multi-endpoints mechanism to find the only master. Client register the address of all 5 servers to ICE, and ICE will try all 5 addresses to automatically find the right server, as long as there is only one master at one time.Grace Period When the master el

12、ection is going on, no service is available, and client must wait for new master to be elected. Use ICE retry mechanism to enable this function, use indicate a retry time series in which ICE will retry connection, for example, 3, 5, 10. We retry connection in 10, 30, 60Paxos Framework Implementfor f

13、ault-tolerance log單棟棟 10748200網(wǎng)絡(luò)實(shí)驗(yàn)室系統(tǒng)結(jié)構(gòu) Api for fault-tolerant logFrom Paxos made livePaxos normal-case operationclientrequetproposalacceptreply012相當(dāng)于leader客戶的兩種提交方式： 1.只能由leader接受請求并提交(我們的做法,chubby) 2.所有服務(wù)器都可以接受請求，并把這些請求轉(zhuǎn)給leader，由leader提交。(zookeeper)Paxos消息類型 ViewChangeMessage(選leader) HeartBeatMes

14、sage(leader租約) PrepareMessage(成leader前的內(nèi)容同步) PrepareOKMessage ProposalMessage(提議) AcceptMessageLeaderElection 何時選leader 系統(tǒng)啟動時 檢測到當(dāng)前l(fā)eader宕機(jī)，并已過leader的租約 每次選leader都要提交一個全序的view號 View號的產(chǎn)生 兩種選leader的方法 每臺服務(wù)器只提自己當(dāng)leader 可以提別的服務(wù)器為leader(我們的實(shí)現(xiàn))PreparePhare Leader被選出后，由leader執(zhí)行，并只執(zhí)行一次 保證系統(tǒng)安全的過渡 Leader catc

15、h upProposalPhare 由leader發(fā)起Proposal 兩種proposal 同步proposal，客戶提交決議后一直等待，直到?jīng)Q定被完成(有可以失敗) 異步proposal，客戶提交決議后馬上返回，paxos執(zhí)行完決議后再通知客戶 Proposal的限時,重發(fā)消息的發(fā)送與接收 使用boost的asio庫進(jìn)行網(wǎng)絡(luò)通訊 采用多播的方式 點(diǎn)對點(diǎn)的catch up使用TCP連接 接受消息使用異步socket 采用多線程 多類型的消息傳輸實(shí)驗(yàn) 向paxos不停的提交proposal，讓paxos到保證每臺服務(wù)器數(shù)據(jù)的一致性(log) 每臺服務(wù)器都記log，并且同步寫磁盤 對于一個proposal有1秒來還沒答成一致，作提交失敗處理 總共4組實(shí)驗(yàn)，每組各進(jìn)行3次，每次5000個proposal(在單臺機(jī)器上模擬) 3臺 (運(yùn)行2臺，運(yùn)行3臺) 5臺 (運(yùn)行3臺，運(yùn)行5臺)結(jié)果1-3臺機(jī)器機(jī)器數(shù)(3臺)運(yùn)行時間提交失敗數(shù)(5000次提交)平均每秒提交數(shù)每100失敗率只運(yùn)行2臺 99.133050.7次/秒0.02795.1260101.8564運(yùn)行3臺126.893038.95次