2015pg大會(huì)內(nèi)容源代碼主題03postgresql checkpoint源碼分析

PostgreSQLcheckpoint2015-05-0615:19:43|作者：|分類：PgSQL2015年度PG大象會(huì)報(bào)名地址：PostgreSQL中國(guó)社區(qū)： Postgres PostgreSQL用戶pg_clog從數(shù)據(jù)恢復(fù)（XLOG）XLOG文件中分為兩個(gè)位置，一 務(wù)狀態(tài)和臟數(shù)據(jù)都寫(xiě)入持久化，這個(gè)需要一個(gè)過(guò)程，這些刷臟頁(yè)面和CLOG的動(dòng)作同樣會(huì)產(chǎn)生XLOG，所以這一系列動(dòng)作完成后，就是檢查點(diǎn)結(jié)束從邏輯角度來(lái)看，這兩個(gè)XLOG位置實(shí)際是同一個(gè)位置，所以在做數(shù)據(jù)恢復(fù)時(shí)，先找到檢查點(diǎn)的XLOG物理位置，然后根據(jù)這里的結(jié)束檢查點(diǎn)時(shí)寫(xiě)入的XLOG信息找到邏輯位置，從邏輯位置開(kāi)始，XLOG并實(shí)施xlogrey恢復(fù)，至少要恢復(fù)到XLOG物理位置才能確保數(shù)據(jù)庫(kù)的一致性和完整性。checkpotrproesseckpoteMa@rc/backnd/posmastr/heckpotr.c接收檢查點(diǎn)請(qǐng)求：*Processanyrequestsorsignalsreceivedif{checkpoint_requested=do_checkpoint=}超時(shí)(checkpoint_timeout參數(shù))*acheckpointiftoomuchtimehaselapsedsincelast*bitevenifthereisalsoanexternalnow=(pg_time_t)elapsed_secs=now-if(elapsed_secs>={ifdo_checkpoint=flags|=}*Notethatwecounta*Notethatwecountatimedcheckpointinstatsonlywhen*occurswithoutanexternalrequest,butwesettherestartpoint.As*side-effect,RecoveryInProgress()TimeLineID*Doacheckpointifif{ ckpt_performed= /*usevolatilepointertopreventcoderearrangementvolatileCheckpointerShmemStruct*cps=*Checkifweshouldperformacheckpointor*it'snotsetdo_restartpoint=*Atomicallyfetchtherequestflagstofigureoutkindof*checkpointweshouldperform,andincreasethe*toacknowledgethatwe'vestartedanewflags|=cps-cps->ckpt_flags= *Theend-of-recoverycheckpointisareal*performedwhilewe'restillinif(flags&do_restartpoint= *Wewillwarnif(a)toosoonsincelast*causedit)and(b)somebodysetCHECKPOINT_CAUSE_XLOG*sincethelastcheckpoint Noteinthat*implementationwillnotgeneratewarningscaused*CheckPointTimeout<if(!do_restartpoint(flags&CHECKPOINT_CAUSE_XLOG)elapsed_secs<(errmsg_plural("checkpointsaretoofrequently(%dsecond"checkpointsareoccurringtoofrequently(%derrhint("Considerincreasingconfigurationparameter *Initializecheckpointer-privatevariablesused*ckpt_active=ifckpt_start_recptr ckpt_start_time=ckpt_cached_elapsed=*Dotheif{ckpt_performed=} ckpt_performed= *Afteranycheckpoint,closeallsmgr Thisso *won'thangontosmgrreferencestodeletedfiles *Indicatecheckpointcompletiontoanywaitingcps->ckpt_done=cps- if *Notewerecordthecheckpointstarttimenottime * Thisissothat *checkpointshappenatapredictable last_checkpoint_time= {*WewerenotabletoperformthehavenotreceivedanynewcheckpointWALlastrestartpoint.Tryagainin15last_checkpoint_time=now-CheckPointTimeout+}*throwanERROR*throwanERRORincaseof Mostbecauseckpt_active=}GetInsertRecPtr--Returnsthecurrentinsert*NOTE:Thevalue*actually*returnedisthepositionofthelastxlogpage.Itlagsbehindtherealinsertpositionbyatmost1Forthat,wedon'tneedtoscanthroughWALinsertionlocks,andapproximationisenoughforthecurrentusageofthis{/*usevolatilepointertopreventcoderearrangement*/volatileXLogCtlData*xlogctl=XLogCtl;recptr=xlogctl- return}*IsCheckpointOnSchedule--areweonscheduletofinishthis*in*Comparesthecurrentprogressagainstthetime/segmentselapsedsincecheckpoint,andreturnstrueiftheprogresswe'vemadethisfaristhantheelapsedstaticboolIsCheckpointOnSchedule(double{structtimeval /*Scaleprogressaccordingpletion_gresspletion_targetCheckagainstthecachedvaluefirst.Onlydothemorecalculationsoncewereachthetargetpreviouslycalculated.neithertimeorWALinsertpointermovesbackwards,acalculatedvaluecanonlybegreaterthanorequaltotheif(progress<returnfalse;false，checkpointerCheckprogressagainstWALsegmentswrittenWecomparethecurrentWALinsertlocationagainstthecomputedbeforecallingCreateCheckPoint.Thecodeinactuallytriggersacheckpointwhencheckpoint_segmentsiscomparesagainstRedoRecptr,sothisisnotcompleyHowever,it'sgoodenoughforourpurposes,we'reonlyestimateif{recptr=elapsed_xlogs=(((double)(recptr-ckpt_start_recptr))XLogSegSize)/ pletion_target0-1 假 pletion_target=0.1,progress傳參數(shù)=1, checkpoint_segments=100,0.1×100=10文件后, if(progresselapsed_xlogs){ckpt_cached_elapsed=returnfalse;false，checkpointer息}}Checkprogressagainsttimeelapsedandgettimeofday(&now,elapsed_time=((double)((pg_time_t)now.tv_sec-ckpt_start_time)now.tv_usec.0)CheckPointTimeout;//if(progress<{ckpt_cached_elapsed=return}/*Itlookslikewe'reonschedule.return}100CheckpointWriteDelay--controlrateof*ThisfunctioniscalledaftereachpagewriteperformedbyItisresponsibleforthrottlingBufferSync()'swriterateto**Thecheckpointrequestflagsshouldbepassedin;currentlytheonlyexaminedisCHECKPOINT_IMMEDIATE,whichdisablesdelaysbetween'progress'isanestimateofhowmuchoftheworkhasbeendone,asfractionbetween0.0meaningnone,and1.0meaningallCheckpointWriteDelay(intflags,double{staticabsorb_counter=/*Donothingifcheckpointisbeingexecutedbynon-processif*Performtheusualdutiesandtakeanap,unlesswe're*inwhichcasewejusttrytocatchupasquicklyasif(!(flags&CHECKPOINT_IMMEDIATE)!shutdown_requested!ImmediateCheckpointRequested() edule{if{got_SIGHUP=/*updateshmemcopiesofconfigvariables}absorb_counter=*ReportinterimactivitystatisticstotheThissleepusedtobeconnectedtotypicallyThatresultedinmorefrequentwakeupsifnotmuchtoCheckpointerandbgwriterarenolongerrelatedsothe 100000100checkpointerbgwriterbgwriter_delay}}elseif(--absorb_counter<={AbsorbpendingfsyncrequestsafterWRITES_PER_ABSORBoperationsevenwhenwedon'tsleep,tooverflowoffsyncrequestabsorb_counter=}}0.5。檢查點(diǎn)的任務(wù)之一是刷臟塊，例1000個(gè)臟塊需要刷新，那么當(dāng)刷到如果這個(gè)時(shí)候，XLOG10個(gè)文件，checkpoint_segments100，也就0.10.05<0.1,返回false,xlog經(jīng)歷個(gè)數(shù)比值小于0.05時(shí)才能休息，也就是發(fā)生在XLOG5個(gè)或以內(nèi)時(shí)。1，那么progress=(100/1000)*1=0.1xlog經(jīng)歷個(gè)數(shù)0.1XLOG10個(gè)或以內(nèi)時(shí)。調(diào)度系數(shù)越大，checkpointer休息區(qū)間越大，checkpointer可以經(jīng)常休息，慢悠悠的fsync；調(diào)度系數(shù)越小，checkpointer休息區(qū)間越小，checkpointer只能在最初的Performacheckpointeitherduringshutdown,oron-the-*flagsisabitwiseORoftheCHECKPOINT_IS_SHUTDOWN:checkpointisfordatabaseCHECKPOINT_END_OF_RECOVERY:checkpointisforendofWALCHECKPOINT_IMMEDIATE:finishthecheckpoint pletion_target acheckpointevenifnoXLOGactivitysincethelastone(impliedbyCHECKPOINT_IS_SHUTDOWNNote:flagscontainsotherbits,ofinteresthereonlyforloggingInparticularnotethatthisroutineissynchronousanddoesnotattentiontoIf!shutdownthenwearewritinganonlinecheckpoint.ThisisaverykindofoperationandWALrecordbecausethecheckpointactionoccursaperiodoftimeyetlogicallyoccursatjustasingleLSN.The positionoftheWALrecord(redoptr)isthesameorearlierthanphysicalposition.Whenwe yWALwelocatethecheckpointviaphysicalpositionthenreadtheredoptrandactuallystart yatearlierlogicalposition.Notethatwedon'twrite*anything*toWAL thelogicalposition,sothatlocationcouldbeanyotherkindofWALAllofthismechanismallowsustocontinueworkingwhileweAsaresult,timingofactionsiscriticalhereandbecarefultonotethisfunctionwilllikelytakeminutestoexecuteonabusyCreateCheckPoint(intCreateCheckPoint(int{/*usevolatilepointertopreventcoderearrangement*/volatileXLogCtlData*xlogctl=XLogCtl;XLogCtlInsert*Insert=&XLogCtl-XLogRecDataVirtualTransactionIdAcquireCheckpointLocktoensureonlyonecheckpointhappensat(Thisisjustproforma,sinceinthepresentsystemthereonlyoneprocessthatisallowedtoissuecheckpointsatanyLWLockAcquire(CheckpointLock,ifif{LWLockAcquire(ControlFileLock,ControlFile->state=DB_SHUTDOWNING;ControlFile->time=(pg_time_t)time(NULL);}獲取XLOGcheckpoint的邏輯XLOGWemustblockconcurrentinsertionswhileexamininginsertstatedeterminethecheckpointREDOcurInsert=XLogBytePosToRecPtr(Insert-checkpoint的邏輯XLOG位置，即開(kāi)始位置，檢查點(diǎn)執(zhí)行fsync時(shí)依賴這ComputenewREDOrecordComputenewREDOrecordptr=locationofnextXLOG*NB:thisisNOTnecessarilywherethecheckpointrecorditselfsinceotherbackendsmayinsertmoreXLOGrecordswhilewe'rethebufferflush ThoseXLOGrecordsarelogicallyaftercheckpoint,eventhoughphysicallybefore Gotspace=INSERT_ if( space==0){if(curInsert%XLogSegSize==curInsert+=curInsertcurInsert+=}checkPoint.redo=*HereweupdatethesharedRedoRecPtrforfutureXLogInsertmustbedonewhileholdingalltheinsertion*Note:ifwefailtocompletethecheckpoint,RedoRecPtrwill*pointingpastwhereitreallyneedsto Thisisokay;consequenceisthatXLogInsertmightbackupwholebuffersthatdidn'treallyneed Wecan'tpostponeadvancing*XLogInsertsthathappenwhilewearebuffersmust*theirbufferchangesarenotincludedintheRedoRecPtr=xlogctl->Insert.RedoRecPtr=NowwecanreleasetheWALinsertionlocks,allowingotherxactsproceedwhileweareflushingdisk釋放XLOG獲得檢查點(diǎn)的其他數(shù)據(jù)，例如XID,OID,MXID*GettheotherinfoweneedforthecheckpointLWLockAcquire(XidGenLock,checkPoint.nextXid=ShmemVariableCache->nextXid;checkPoint.oldestXid=ShmemVariableCache->oldestXid;checkPoint.oldestXidDBcheckPoint.nextXid=ShmemVariableCache->nextXid;checkPoint.oldestXid=ShmemVariableCache->oldestXid;checkPoint.oldestXidDB=ShmemVariableCache->oldestXidDB;/*IncreaseXIDepochifwe'vewrappedaroundsincelastcheckpoint*/checkPoint.nextXidEpoch=ControlFile->checkPointCopy.nextXidEpoch;if(checkPoint.nextXid<ControlFile->checkPointCopy.nextXid)LWLockAcquire(OidGenLock,LW_SHARED);checkPoint.nextOid=ShmemVariableCache->nextOid;if(!shutdown)checkPoint.nextOid+=ShmemVariableCache->oidCount;checkpoint開(kāi)始FsyncclogclogXLOGInsomecasestherearegroupsofactionsthatmustalloccursideortheotherofacheckpointrecord.Beforeflushingcheckpointrecordwemustexplicitlywaitforanybackendperformingthosegroupsof*Oneexampleisendoftransaction,sowemustwaitforthatarecurrentlyincommitcritical IfanitscommitrecordintoXLOGjustbeforetheREDOpoint,thenrestartfromtheREDOpointwouldnotreythatrecord,thatourflushinghadbetterincludethexact'supdate wewaittillhe'soutofhiscommitcriticalsectionSeenotesin Becausewe'vealreadyreleasedtheinsertionlocks,thistestisfuzzy:itispossiblethatwewillwaitforxactswedidn'ttowait Butthedelayshouldbeshortanditseemsbettercheckpointtakeabitlongerthantoholdoffinsertionsnecessary.(Infact,thewholereasonwehavethisissueis 根源在這里，因?yàn)樘峤粚?xiě)clogXLOGCLOG //為什么呢？因?yàn)镽ECOVERY時(shí)檢查點(diǎn)之前的XLOG是不會(huì)去rey的，如果clog的xlog在doescommitrecordXLOGinsertionandclogupdateastwo**protectedbydifferentlocks,butagainthatseemsbestonminimizinglock*AtransactionthathasnotyetsetdelayChkptwhenwelookberisk,sincehe'snotinsertedhiscommitrecordyet;andonealreadycleareditisnotatriskeither,sincehe'sdone*andwewillcorrectlyflushtheupdate Sowecannot*xactsweneedtowaitvxids=GetVirtualXIDsDelayingChkpt(&nvxids);if(nvxids>0){{/*waitfor10}while(HaveVirtualXIDsDelayingChkpt(vxids,} Fsync完成后，寫(xiě)入一段XLOG*Nowinsertthecheckpointrecordintordata.data=(char*)rdata.lenrdata.len=sizeof(checkPoint);rdata.buffer=InvalidBuffer;rdata.next=NULL;recptr=shutdownXLOG_CHECKPOINT_SHUTDOWN更新控制文件，控制文件中寫(xiě)入檢查點(diǎn)的XLOGSelectpointatwhichwecantruncatethelog,whichwebaseonpriorcheckpoint'searliest*UpdatethecontrolLWLockAcquire(ControlFileLock,LW_EXCLUSIVE);if(shutdown)ControlFile->state=ControlFile->prevCheckPoint=ControlFile-ControlFile->checkPoint= ControlFile->checkPointCopy=xlog輯位置 ControlFile->time=(pg_time_t)/*crashrecoveryshouldalwaysrecovertotheendofWAL*/ControlFile->minRecoveryPoint=InvalidXLogRecPtr;ControlFile->minRecoveryPointTLIControlFile->minRecoveryPointTLI=PersistunloggedLSNvalue.It'sresetoncrashrecovery,sounusedonnon-shutdowncheckpoints,butseemsusefultostorefordebuggingControlFile->unloggedLSN=XLogCtl->unloggedLSN;FsyncCheckPointGutsFlushalldatainsharedmemorytodisk,and*ThisisthecommoncodesharedbetweenregularcheckpointsrecoverystaticvoidCheckPointGuts(XLogRecPtrcheckPointRedo,int{ //src/backend/access/transam/clog.c //src/backend/access/transam/subtrans.c //src/backend/access/transam/multixact.c // // //src/backend/replication/logical/snapbuild.c ///*p