《數(shù)據(jù)采集技術(shù)》課件-Middlewares中間件

Middlewares——中間件Middlewares—中間件scrapy中間件Middlewares—中間件根據(jù)其程序執(zhí)行流程位置不同分為兩類：(介于引擎與下載器之間)下載中間件(介于引擎與爬蟲組件之間)爬蟲中間件Middlewares—中間件中間件位于同一個middlewares.py文件之中：下載中間件下載中間件作用：對request對象進(jìn)行設(shè)置隨機請求頭、IP代理、Cookie等1判斷response對象的狀態(tài)碼，來決定是否將response提交給引擎2下載中間件process_request方法用來處理正常的請求對象的數(shù)據(jù)，每個request通過下載中間件時被調(diào)用process_request(self,request,spider)request要處理的Request對象spider該request對應(yīng)的Spider對象參數(shù)如下：下載中間件process_response方法用于對response進(jìn)行處理，下載器完成http請求，傳遞響應(yīng)給引擎的時候調(diào)用process_response(self,request,response,spider)request表示response所對應(yīng)的requestresponse表示被處理的response參數(shù)如下：spider表示response所對應(yīng)的spider下載中間件process_exception方法用來處理拋異常的請求對象的數(shù)據(jù)process_exception(self,request,exception,spider)request表示response所對應(yīng)的requestexception表示拋出的異常參數(shù)如下：spider表示response所對應(yīng)的spider下載中間件案例：middlewares.py，代碼如下：classRandomUserAgent(object):defprocess_request(self,request,spider):useragent=random.choice(USER_AGENTS)request.headers.setdefault("User-Agent",useragent)下載中間件案例：settings.py，代碼如下：USER_AGENTS=["Mozilla/4.0(compatible;MSIE6.0;WindowsNT5.1;SV1;AcooBrowser;.NETCLR1.1.4322;.NETCLR2.0.50727)","Mozilla/4.0(compatible;MSIE7.0;WindowsNT6.0;AcooBrowser;SLCC1;.NETCLR2.0.50727;MediaCenterPC5.0;.NETCLR3.0.04506)","Mozilla/4.0(compatible;MSIE7.0;AOL9.5;AOLBuild4337.35;WindowsNT5.1;.NETCLR1.1.4322;.NETCLR2.0.50727)","Mozilla/5.0(Windows;U;MSIE9.0;WindowsNT9.0;en-US)","Mozilla/5.0(compatible;MSIE9.0;WindowsNT6.1;Win64;x64;Trident/5.0;.NETCLR3.5.30729;.NETCLR3.0.30729;.NETCLR2.0.50727;MediaCenterPC6.0)","Mozilla/5.0(compatible;MSIE8.0;WindowsNT6.0;Trident/4.0;WOW64;Trident/4.0;SLCC2;.NETCLR2.0.50727;.NETCLR3.5.30729;.NETCLR3.0.30729;.NETCLR1.0.3705;.NETCLR1.1.4322)","Mozilla/4.0(compatible;MSIE7.0b;WindowsNT5.2;.NETCLR1.1.4322;.NETCLR2.0.50727;InfoPath.2;.NETCLR3.0.04506.30)","Mozilla/5.0(Windows;U;WindowsNT5.1;zh-CN)AppleWebKit/523.15(KHTML,likeGecko,Safari/419.3)Arora/0.3(Change:287c9dfb30)","Mozilla/5.0(X11;U;Linux;en-US)AppleWebKit/527+(KHTML,likeGecko,Safari/419.3)Arora/0.6“]下載中間件設(shè)置settings.py中的DOWNLOADER_MIDDLEWARES，取消注釋：DOWNLOADER_MIDDLEWARES={'mySpider.middlewares.MyspiderDownloaderMiddleware':543,}COOKIES_ENABLED=False案例：禁用cookies以防止某些網(wǎng)站根據(jù)Cookie來封鎖爬蟲下載中間件設(shè)置成功后，運行爬蟲就會發(fā)現(xiàn)請求頭設(shè)置成功請求發(fā)送給服務(wù)器之前下載中間件隨機選擇一個請求頭，就可以避免總使用一個請求頭防止爬蟲被服務(wù)器發(fā)現(xiàn)爬蟲中間件爬蟲中間件爬蟲中間件作用：1在Response發(fā)送給Spider之前，對Response進(jìn)行處理2在Request發(fā)送給Scheduler之前，對Request進(jìn)行處理3在Item發(fā)送給ItemPipeline之前，對Item進(jìn)行處理爬蟲中間件自帶了許多SpiderMiddleware爬蟲中間件SPIDER_MIDDLEWARES_BASE變量的內(nèi)容如下：'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware':50,'scrapy.spidermiddlewares.offsite.OffsiteMiddleware':500,'scrapy.spidermiddlewares.referer.RefererMiddleware':700,'scrapy.spidermiddlewares.urllength.UrllengthMiddleware':800,'scrapy.spidermiddlewares.depth.DepthMiddleware':900,SPIDER_MIDDLEWARES={'mySpider.middlewares.MyspiderSpiderMiddleware':543,}爬蟲中間件自帶了許多SpiderMiddleware爬蟲中間件process_spider_input方法當(dāng)Response被爬蟲中間件處理時被調(diào)用。返回結(jié)果是：None或者拋出一個異常process_spider_input(response,spider)response被處理的Response對象spider該Response對應(yīng)的Spider參數(shù)如下：爬蟲中間件process_spider_output方法當(dāng)Spider處理Response返回結(jié)果時，該方法被調(diào)用process_spider_output(response,result,spider)response生成該輸出的ResponseresultSpider返回的結(jié)果參數(shù)如下：spider其結(jié)果對應(yīng)的Spider爬蟲中間件process_spider_exception方法當(dāng)process_spider_input方法拋出異常時被調(diào)用process_spider_exception(response,exception,spider)爬蟲中間件process_start_r