大數(shù)據(jù)采集與預(yù)處理課件：模擬登錄和數(shù)據(jù)采集綜合案例 -3

網(wǎng)站模擬登錄和數(shù)據(jù)采集案例 本章學(xué)習(xí)目標(biāo)●分析業(yè)務(wù)網(wǎng)站人才招聘的網(wǎng)頁結(jié)構(gòu)和內(nèi)容●使用selenium和chromedriver實現(xiàn)網(wǎng)站的模擬登錄●使用selenium和chromedriver實現(xiàn)編寫爬蟲代碼獲取指定的靜態(tài)和動態(tài)數(shù)據(jù)●使用pymysql庫實現(xiàn)數(shù)據(jù)的持久化8.7數(shù)據(jù)預(yù)處理1）導(dǎo)入指定的庫importpandasaspdimportpymysqlfromtabulateimporttabulate2）自定義方法main()，作為實現(xiàn)調(diào)用其他數(shù)據(jù)預(yù)處理自定義方法的入口。defmain():#1數(shù)據(jù)讀取data=read_data()#2探索清理數(shù)據(jù)check_data(data)#3轉(zhuǎn)換數(shù)據(jù)data_new=transform_data(data)#4規(guī)約數(shù)據(jù)defcustom_sampling(group):returngroup.sample(n=1)#采樣每個組內(nèi)的一個樣本result=data_new.groupby(['addr_names','industry_name']).apply(custom_sampling)print(tabulate(result,headers='keys',tablefmt='pretty'))8.7數(shù)據(jù)預(yù)處理3）自定義方法read_data()實現(xiàn)數(shù)據(jù)庫連接操作，并返回DataFrame格式數(shù)據(jù)。defread_data():db=pymysql.connect(host='localhost',user='root',password='12345678',port=3306,db='test')cursor=db.cursor()cursor.execute("SELECT*FROMzhitongdongtai")data=cursor.fetchall()columns=['com_name','salary','addr_names','industry_name']df=pd.DataFrame(list(data),columns=columns)db.close()returndf8.7數(shù)據(jù)預(yù)處理4）自定義方法check_data(data)實現(xiàn)數(shù)據(jù)的清理操作。defread_data():db=pymysql.connect(host='localhost',user='root',password='12345678',port=3306,db='test')cursor=db.cursor()cursor.execute("SELECT*FROMzhitongdongtai")data=cursor.fetchall()columns=['com_name','salary','addr_names','industry_name']df=pd.DataFrame(list(data),columns=columns)db.close()returndf8.7數(shù)據(jù)預(yù)處理5）自定義方法transform_data(data)實現(xiàn)數(shù)據(jù)轉(zhuǎn)換操作。deftransform_data(data):df=data#1轉(zhuǎn)換薪水字段salary_range_pattern=r'(?P<low_salary>\d+\.?\d*)-(?P<high_salary>\d+\.?\d*)K'salary_policy_pattern=r'(?P<salary_policy>\d*薪)?'final_pattern=rf'{salary_range_pattern}·?{salary_policy_pattern}'#使用逐步構(gòu)建的正則表達式拆分'salary'列并創(chuàng)建新的列extracted=df['salary'].str.extract(final_pattern,expand=True)#設(shè)置默認值為12薪extracted['salary_policy']=extracted['salary_policy'].fillna('12薪')#將提取的列賦值給DataFramedf[['low_salary','high_salary','salary_policy']]=extracted[['low_salary','high_salary','salary_policy']]#刪除原始'salary'列df.drop('salary',axis=1,inplace=True)#將字符串型的數(shù)字列轉(zhuǎn)為浮點型df[['low_salary','high_salary']]=df[['low_salary','high_salary']].astype(float)*1000#2轉(zhuǎn)換地址字段min_length=2max_length=4data_new=df[(df['addr_names'].str.len()>=min_length)&(df['addr_names'].str.len()<=max_length)]returndata_new6）設(shè)置程序入口if__name__=="__main__":main()8.8小結(jié)通過本綜合案例的學(xué)習(xí)，掌握了使用CHROME瀏覽器的“開發(fā)者工具”分析二手房的網(wǎng)頁結(jié)構(gòu)和內(nèi)容，使用requests庫和BeautifulSoup庫編寫和解析自定義的爬蟲代碼獲取字段為：地區(qū)(region)、戶型(house_type)、面積(area_list)、裝修類型(decorate_type)、樓層(floor_type)、房屋類型(building_type)、售價(total_price)、單價(avg_price)的數(shù)據(jù)。最后，使用pymysql庫在MYSQL數(shù)據(jù)