Python爬蟲程序設(shè)計(jì)KC26課件

2.6.1項(xiàng)目簡介2.6.1項(xiàng)目簡介2.6.1項(xiàng)目簡介在中國天氣網(wǎng)()中輸入一個(gè)城市的名稱，例如輸入深圳，那么會轉(zhuǎn)到地址/weather1d/101280601.shtml的網(wǎng)頁顯示深圳的天氣預(yù)報(bào)，其中101280601是深圳的代碼，每個(gè)城市或者地區(qū)都有一個(gè)代碼。如圖2-6-1、2-6-2所示。2.6.1項(xiàng)目簡介Python爬蟲程序設(shè)計(jì)KC26課件Python爬蟲程序設(shè)計(jì)KC26課件2.6.2HTML代碼分析2.6.2HTML代碼分析2.6.2HTML代碼分析用Chrome瀏覽器瀏覽網(wǎng)站，鼠標(biāo)指向7天天氣預(yù)報(bào)的今天位置，點(diǎn)擊右鍵彈出菜單，選擇“檢查”就可以打開這個(gè)位置對應(yīng)的HTML代碼，如圖2-6-3所示。2.6.2HTML代碼分析選擇<ulclass="tclearfix">元素，點(diǎn)擊右鍵彈出菜單選擇"EditasHTML"，就可以進(jìn)入編輯狀態(tài)，復(fù)制整個(gè)HTML，結(jié)果如下：<ulclass="tclearfix"><liclass="on"><h1>5日（今天）</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>32</span>/<i>28℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em>選擇<ulclass="tclearfix">元素，點(diǎn)擊<i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>6日（明天）</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>32</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><i>微風(fēng)</i><h1>7日（后天）</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>32</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>8日（周四）</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>32</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>7日（后天）</h1><h1>9日（周五）</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>33</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>9日（周五）</h1><h1>10日（周六）</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>33</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>10日（周六）</h1><h1>11日（周日）</h1><bigclass="png40d01"></big><bigclass="png40n07"></big><ptitle="多云轉(zhuǎn)小雨"class="wea">多云轉(zhuǎn)小雨</p><pclass="tem"><span>33</span>/<i>26℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li></ul><h1>11日（周日）</h1>2.6.3爬取天氣預(yù)報(bào)數(shù)據(jù)2.6.3爬取天氣預(yù)報(bào)數(shù)據(jù)2.6.3爬取天氣預(yù)報(bào)數(shù)據(jù)通過分析HTML代碼，我們可以編寫爬取的程序爬取深圳7天的天氣預(yù)報(bào)數(shù)據(jù)：

frombs4importBeautifulSoupfrombs4importUnicodeDammitimporturllib.request

url="/weather/101280601.shtml"try:headers={"User-Agent":"Mozilla/5.0(Windows;U;WindowsNT6.0x64;en-US;rv:1.9pre)Gecko/2008072421Minefield/3.0.2pre"}req=urllib.request.Request(url,headers=headers)data=urllib.request.urlopen(req)data=data.read()dammit=UnicodeDammit(data,["utf-8","gbk"])data=dammit.unicode_markupsoup=BeautifulSoup(data,"lxml")lis=soup.select("ul[class='tclearfix']li")forliinlis:try:date=li.select('h1')[0].textweather=li.select('p[class="wea"]')[0].texttemp=li.select('p[class="tem"]span')[0].text+"/"+li.select('p[class="tem"]i')[0].textprint(date,weather,temp)exceptExceptionaserr:print(err)exceptExceptionaserr:print(err)2.6.3爬取天氣預(yù)報(bào)數(shù)據(jù)程序爬取結(jié)果：5日（今天）多云32/28℃6日（明天）多云32/27℃7日（后天）多云32/27℃8日（周四）多云32/27℃9日（周五）多云33/27℃10日（周六）多云33/27℃11日（周日）多云轉(zhuǎn)小雨33/26℃由此可見爬取的數(shù)據(jù)與我們直接從網(wǎng)站看到的是一樣的。

程序爬取結(jié)果：2.6.4爬取與存儲天氣預(yù)報(bào)數(shù)據(jù)2.6.4爬取與存儲天氣預(yù)報(bào)數(shù)據(jù)2.6.4爬取與存儲天氣預(yù)報(bào)數(shù)據(jù)我們可以獲取北京、上海、廣州、深圳等城市的代碼，爬取這些城市的天氣預(yù)報(bào)數(shù)據(jù)，并存儲到sqllite數(shù)據(jù)庫weathers.db中，存儲的數(shù)據(jù)表weathers是：createtableweathers(wCityvarchar(16),wDatevarchar(16),wWeathervarchar(64),wTempvarchar(32),constraintpk_weatherprimarykey(wCity,wDate))"編寫程序依次爬取各個(gè)城市的天氣預(yù)報(bào)數(shù)據(jù)存儲在數(shù)據(jù)庫中，程序如下：2.6.4爬取與存儲天氣預(yù)報(bào)數(shù)據(jù)frombs4importBeautifulSoupfrombs4importUnicodeDammitimporturllib.requestimportsqlite3

classWeatherDB:defopenDB(self):self.con=sqlite3.connect("weathers.db")self.cursor=self.con.cursor()try:self.cursor.execute("createtableweathers(wCityvarchar(16),wDatevarchar(16),wWeathervarchar(64),wTempvarchar(32),constraintpk_weatherprimarykey(wCity,wDate))")except:self.cursor.execute("deletefromweathers")

defcloseDB(self):mit()self.con.close()frombs4importBeautifulSoupdefinsert(self,city,date,weather,temp):try:self.cursor.execute("insertintoweathers(wCity,wDate,wWeather,wTemp)values(?,?,?,?)",(city,date,weather,temp))exceptExceptionaserr:print(err)

defshow(self):self.cursor.execute("select*fromweathers")rows=self.cursor.fetchall()print("%-16s%-16s%-32s%-16s"%("city","date","weather","temp"))forrowinrows:print("%-16s%-16s%-32s%-16s"%(row[0],row[1],row[2],row[3]))

classWeatherForecast:def__init__(self):self.headers={"User-Agent":"Mozilla/5.0(Windows;U;WindowsNT6.0x64;en-US;rv:1.9pre)Gecko/2008072421Minefield/3.0.2pre"}self.cityCode={"北京":"101010100","上海":"101020100","廣州":"101280101","深圳":"101280601"}definsert(self,city,date,weatdefforecastCity(self,city):ifcitynotinself.cityCode.keys():print(city+"codecannotbefound")return

url="/weather/"+self.cityCode[city]+".shtml"try:req=urllib.request.Request(url,headers=self.headers)data=urllib.request.urlopen(req)data=data.read()dammit=UnicodeDammit(data,["utf-8","gbk"])data=dammit.unicode_markupsoup=BeautifulSoup(data,"lxml")lis=soup.select("ul[class='tclearfix']li")defforecastCity(self,city):forliinlis:try:date=li.select('h1')[0].textweather=li.select('p[class="wea"]')[0].texttemp=li.select('p[class="tem"]span')[0].text+"/"+li.select('p[class="tem"]i')[0].textprint(city,date,weather,temp)self.db.insert(city,date,weather,temp)exceptExceptionaserr:print(err)exceptExceptionaserr:print(err)forliinlis:defprocess(self,cities):self.db=WeatherDB()self.db.openDB()

forcityincities:self.forecastCity(city)

#self.db.show()self.db.closeDB()

ws=WeatherForecast()cess(["北京","上海","廣州","深圳"])print("completed")defprocess(self,cities):北京7日（今天）晴間多云，北部山區(qū)有陣雨或雷陣雨轉(zhuǎn)晴轉(zhuǎn)多云31℃/17℃北京8日（明天）多云轉(zhuǎn)晴，北部地區(qū)有分散陣雨或雷陣雨轉(zhuǎn)晴34℃/20℃北京9日（后天）晴轉(zhuǎn)多云36℃/22℃北京10日（周六）陰轉(zhuǎn)陣雨30℃/19℃北京11日（周日）陣雨27℃/18℃北京12日（周一）陰轉(zhuǎn)晴28℃/20℃北京13日（周二）晴32℃/21℃上海7日（今天）多云30/21℃上海8日（明天）多云轉(zhuǎn)陰32/23℃上海9日（后天）陣雨32/24℃上海10日（周六）中雨27/22℃上海11日（周日）小雨轉(zhuǎn)多云29/22℃上海12日（周一）多云30/22℃上海13日（周二）多云轉(zhuǎn)陰30/21℃廣州7日（今天）多云35/27℃廣州8日（明天）多云35/28℃廣州9日（后天）多云35/28℃廣州10日（周六）多云35/28℃廣州11日（周日）多云35/28℃廣州12日（周一）雷陣雨35/27℃廣州13日（周二）雷陣雨轉(zhuǎn)大雨33/24℃深圳7日（今天）陣雨轉(zhuǎn)多云34/28℃深圳8日（明天）晴34/28℃深圳9日（后天）晴34/28℃深圳10日（周六）晴轉(zhuǎn)陣雨34/28℃深圳11日（周日）陣雨33/27℃深圳12日（周一）陣雨32/27℃深圳13日（周二）陣雨轉(zhuǎn)中雨32/25℃北京7日（今天）晴間多云，北部山區(qū)有陣雨或雷陣雨轉(zhuǎn)晴轉(zhuǎn)多2.6.1項(xiàng)目簡介2.6.1項(xiàng)目簡介2.6.1項(xiàng)目簡介在中國天氣網(wǎng)()中輸入一個(gè)城市的名稱，例如輸入深圳，那么會轉(zhuǎn)到地址/weather1d/101280601.shtml的網(wǎng)頁顯示深圳的天氣預(yù)報(bào)，其中101280601是深圳的代碼，每個(gè)城市或者地區(qū)都有一個(gè)代碼。如圖2-6-1、2-6-2所示。2.6.1項(xiàng)目簡介Python爬蟲程序設(shè)計(jì)KC26課件Python爬蟲程序設(shè)計(jì)KC26課件2.6.2HTML代碼分析2.6.2HTML代碼分析2.6.2HTML代碼分析用Chrome瀏覽器瀏覽網(wǎng)站，鼠標(biāo)指向7天天氣預(yù)報(bào)的今天位置，點(diǎn)擊右鍵彈出菜單，選擇“檢查”就可以打開這個(gè)位置對應(yīng)的HTML代碼，如圖2-6-3所示。2.6.2HTML代碼分析選擇<ulclass="tclearfix">元素，點(diǎn)擊右鍵彈出菜單選擇"EditasHTML"，就可以進(jìn)入編輯狀態(tài)，復(fù)制整個(gè)HTML，結(jié)果如下：<ulclass="tclearfix"><liclass="on"><h1>5日（今天）</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>32</span>/<i>28℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em>選擇<ulclass="tclearfix">元素，點(diǎn)擊<i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>6日（明天）</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>32</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><i>微風(fēng)</i><h1>7日（后天）</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>32</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>8日（周四）</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>32</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>7日（后天）</h1><h1>9日（周五）</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>33</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>9日（周五）</h1><h1>10日（周六）</h1><bigclass="png40d01"></big><bigclass="png40n01"></big><ptitle="多云"class="wea">多云</p><pclass="tem"><span>33</span>/<i>27℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li><li><h1>10日（周六）</h1><h1>11日（周日）</h1><bigclass="png40d01"></big><bigclass="png40n07"></big><ptitle="多云轉(zhuǎn)小雨"class="wea">多云轉(zhuǎn)小雨</p><pclass="tem"><span>33</span>/<i>26℃</i></p><pclass="win"><em><spantitle="無持續(xù)風(fēng)向"class=""></span><spantitle="無持續(xù)風(fēng)向"class=""></span></em><i>微風(fēng)</i></p><divclass="slid"></div></li></ul><h1>11日（周日）</h1>2.6.3爬取天氣預(yù)報(bào)數(shù)據(jù)2.6.3爬取天氣預(yù)報(bào)數(shù)據(jù)2.6.3爬取天氣預(yù)報(bào)數(shù)據(jù)通過分析HTML代碼，我們可以編寫爬取的程序爬取深圳7天的天氣預(yù)報(bào)數(shù)據(jù)：

frombs4importBeautifulSoupfrombs4importUnicodeDammitimporturllib.request

url="/weather/101280601.shtml"try:headers={"User-Agent":"Mozilla/5.0(Windows;U;WindowsNT6.0x64;en-US;rv:1.9pre)Gecko/2008072421Minefield/3.0.2pre"}req=urllib.request.Request(url,headers=headers)data=urllib.request.urlopen(req)data=data.read()dammit=UnicodeDammit(data,["utf-8","gbk"])data=dammit.unicode_markupsoup=BeautifulSoup(data,"lxml")lis=soup.select("ul[class='tclearfix']li")forliinlis:try:date=li.select('h1')[0].textweather=li.select('p[class="wea"]')[0].texttemp=li.select('p[class="tem"]span')[0].text+"/"+li.select('p[class="tem"]i')[0].textprint(date,weather,temp)exceptExceptionaserr:print(err)exceptExceptionaserr:print(err)2.6.3爬取天氣預(yù)報(bào)數(shù)據(jù)程序爬取結(jié)果：5日（今天）多云32/28℃6日（明天）多云32/27℃7日（后天）多云32/27℃8日（周四）多云32/27℃9日（周五）多云33/27℃10日（周六）多云33/27℃11日（周日）多云轉(zhuǎn)小雨33/26℃由此可見爬取的數(shù)據(jù)與我們直接從網(wǎng)站看到的是一樣的。

程序爬取結(jié)果：2.6.4爬取與存儲天氣預(yù)報(bào)數(shù)據(jù)2.6.4爬取與存儲天氣預(yù)報(bào)數(shù)據(jù)2.6.4爬取與存儲天氣預(yù)報(bào)數(shù)據(jù)我們可以獲取北京、上海、廣州、深圳等城市的代碼，爬取這些城市的天氣預(yù)報(bào)數(shù)據(jù)，并存儲到sqllite數(shù)據(jù)庫weathers.db中，存儲的數(shù)據(jù)表weathers是：createtableweathers(wCityvarchar(16),wDatevarchar(16),wWeathervarchar(64),wTempvarchar(32),constraintpk_weatherprimarykey(wCity,wDate))"編寫程序依次爬取各個(gè)城市的天氣預(yù)報(bào)數(shù)據(jù)存儲在數(shù)據(jù)庫中，程序如下：2.6.4爬取與存儲天氣預(yù)報(bào)數(shù)據(jù)frombs4importBeautifulSoupfrombs4importUnicodeDammitimporturllib.requestimportsqlite3

classWeatherDB:defopenDB(self):self.con=sqlite3.connect("weathers.db")self.cursor=self.con.cursor()try:self.cursor.execute("createtableweathers(wCityvarchar(16),wDatevarchar(16),wWeathervarchar(64),wTempvarchar(32),constraintpk_weatherprimarykey(wCity,wDate))")except:self.cursor.execute("deletefromweathers")

defcloseDB(self):mit()self.con.close()frombs4importBeautifulSoupdefinsert(self,city,date,weather,temp):try:self.cursor.execute("insertintoweathers(wCity,wDate,wWeather,wTemp)values(?,?,?,?)",(city,date,weather,temp))exceptExceptionaserr:print(err)

defshow(self):self.cursor.execute("select*fromweathers")rows=self.cursor.fetchall()print("%-16s%-16s%-32s%-16s"%("city","date","weather","temp"))forrowinrows:print("%-16s%-16s%-32s%-16s"%(row[0],row[1],row[2],row[3]))

classWeatherForecast:def__init__(self):self.headers={"User-Agent":"Mozilla/5.0(Windows;U;WindowsNT6.0x64;en-US;rv:1.9pre)Gecko/2008072421Minefield/3.0.2pre"}self.cityCode={"北京":"101010100","上海":"101020100","廣州":"101280101","深圳":"101280601"}definsert(self,city,date,weatdefforecastCity(self,city):ifcitynotinself.cityCode.keys():print(city+"codecannotbefound")return

url="/weather/"+self.cityCode[city]+".shtml"try:req=urllib.request.Request(url,headers=self.headers)data=urllib.request.urlopen(req)data=data.read()dammit=UnicodeDammit(data,["utf-8","gbk"])data=dammit.unicode_markupsoup=BeautifulSoup(data,"lxml")lis=soup.select("ul[class='tclearfix']li")defforecastCity(self,city):forliin