計算機外文翻譯-基于網(wǎng)絡(luò)爬蟲的有效URL緩存

iZBroderIBMTJWatsonResearchCenter19SkylineDrHawthorne,NY10532roderusibmcomMicrosoftResearch1065LaAvenidaMountainView,CA94043osoftcomJanetL.WienerHewlettPackardLabs1501PageMillRoadPaloAlto,CA94304janet.wiener@(dynamic)subsetofthe―seen‖URLs.Themaingoalofthispaperistocarefully ndmamassivedaywebcrawlthatissuedoveronebillionHTTP1.INTRODUCTIONArecentPewFoundationstudy[31]statesthat―SearchengineshavebecomeanindispensableutilityforInternetusers‖andestimatesthatasofmid-2002,slightlysds ddn 2.CRAWLINGgeURLstocrawlerprocessesisbasedonahashoftheURL’shostcomponent.ACPChoandGarcia-Molina’scrawler[13]issimilartoMercator.Thesystemis URL’shostpart),andhierarchical(assigninganURLtoaC-procbasedonsomecommunicatingcrawlingprocesses(the―ants‖).AnantthatdiscoversanURLforwhichitisnotresponsible,sendsthisURLtoadedicatedprocess(the―controller‖),ifferentcrawlercomponentsareimplementedasdifferentprocesses.A―crawlingapplication‖sthemtoapoolof―downloader‖processes.ThedownloaderprocessesfetchthepagesWebFountain[16],UbiCrawler[4],andChoandMolina’scrawler[13],are 3.CACHINGystheideaofstoringfrequentlyuseditemsfromaslowerlInourcontext,forinstance,isamuchmorecommonethanalinktotheauthorshomepages eefilllikelynotbeneededforalongtimeinthefutureand nmf ThesecondissueaboveistheveryreasonwedecidedtotestSTATIC:if4.EXPERIMENTALSETUPaTheURLstreamthenflowsintotheHostSplitter,whichassignsURLstocrawlingprocessesusingahashoftheURL’shostname.Sincemostlinksarere today’swebcontainsseveralbillionvalidURLs,thememoryrequirementstoegbyreducingthepenaltyimposedbydiskheascenarioeachcrawlerwouldberesponsibleforwebservers―closetoit‖.importantly,theschedulingofdownloadsismoderatedbyMercator’sserver.Mercator’spolitenesspolicyguaranteesthatnoservereverreceivesmultipleserverwillonlybeissuedafteramultiple(typically10_)ofthetimeittookto InternetExchange,quiteclosetotheInternet’sbackbone.hinesperformedbilliondownloadeuptoMBinsizeasettingthatissignificantlyhigherthanusual),makingitThelinksextractedfromtheseHTMLpages,plusabout38millionHTTPdMercator’sHostSplittercomponenttologallincomingURLstodisk.TheHost thisnumbertothenumberof5.SIMULATIONRESULTSfthecachingalgorithmslist.ForCLOCK,itisalistandaclockhandle,andtheitemsalsocontain―mark‖ WethereforedescribeMINinmoredetail.LetAbethetraceorsequenceofthetimewhenAtnextappearsinAIfthereisnofurtherrequestisonlyslightlyinferiortoCLOCKandLRU,whileSTATICisgenerallymuchomissratewouldbeofcourseworsethanMIN’s.6.CONCLUSIONSANDFUTUREDIRECTIONS performroughlythesamewerecommendusingeitherCLOCKorRANDOM,tntostareprobablycloserto 要在網(wǎng)絡(luò)上爬行非常簡單：基本的算法是：(a)取得一個網(wǎng)頁(b)解析它提取所有的鏈接URLs(c)對于所有沒有見過的URLs重復(fù)執(zhí)行(a)-(c)。但是，系統(tǒng)設(shè)計步驟(a)必須每秒鐘執(zhí)行大約1000次，因此，成員檢測(c)必須每秒鐘執(zhí)行URLs存入主內(nèi)存中的一個(動態(tài))子集中。這個論文最主要的成果就是仔細的一個搜索引擎的文集。 絡(luò)資源無法被訪問到。(估計有超過20%)蟲一般使用一個分布式的構(gòu)造來平行地爬行更多的網(wǎng)頁，這使成員檢測更為復(fù)回答了一個同行節(jié)點，而不是當(dāng)?shù)?。一個非常重要的方法加速這個檢測就是用cache(高速緩存)，這個是把見過的URLs存入主內(nèi)存中的一個(動態(tài))子集中。這個論文最主要的成果就是仔細的億次的 部分描述和討論模擬的結(jié)果。第6部分是我們推薦的實際算法和數(shù)據(jù)結(jié)構(gòu)關(guān)于2.CRAWLING網(wǎng)絡(luò)爬蟲用網(wǎng)絡(luò)存檔雇員多個爬行進程，每個一次性完成一個徹底的爬行對于鍵字和超鏈接；URL解決進程將相對路徑轉(zhuǎn)換給絕對路徑。這些不同的進程通3.CACHING 下一個請求最遠的項目。這個理論的算法叫MIN，因為他達到了最小的數(shù)量關(guān)3.3最近被用到(LRU)3.4CLOCKK 。舉一個例子，每個工作現(xiàn)場在一個系統(tǒng)里用4個爬行進程。URL的主機名。直到大多數(shù)的超鏈接被關(guān)聯(lián)，大部分的URL(在我們的實驗中 爬蟲進程維護URLs的子集時分配給它)；但是DUE執(zhí)行(這個強制URLs成連接。每個機器定位于PaloAltoInternetExchange,十分接近于Internet的 供了3個解釋關(guān)于為什么我們每個頁面找到了更過的超鏈接。首先，我們認為顯著地大于平常)，讓它可能遇到上萬個的超鏈接頁面.第三，大部分的論文報e也獲得了一