大數(shù)據(jù)采集與清洗課件

[數(shù)據(jù)采集與清洗]2019|02|15

周

樂[數(shù)據(jù)采集與清洗]2019|02|15周 樂什么是大數(shù)據(jù)大數(shù)據(jù)處理流程大數(shù)據(jù)的主要特征大數(shù)據(jù)采集的概念大數(shù)據(jù)采集應(yīng)用什么是大數(shù)據(jù)大數(shù)據(jù)處理流程大數(shù)據(jù)的主要特征大數(shù)據(jù)采集的概念大1什么是大數(shù)據(jù)1什么是大數(shù)據(jù)大數(shù)據(jù)采集與清洗課件大數(shù)據(jù)采集與清洗課件大數(shù)據(jù)采集與清洗課件淘寶推薦

依據(jù)購(gòu)物行為偏好引薦依據(jù)你最近的閱讀行為和消費(fèi)行為進(jìn)行引薦依據(jù)你用的設(shè)備往來不斷猜特征.依據(jù)時(shí)節(jié)改變進(jìn)行引薦淘寶推薦依據(jù)購(gòu)物行為偏好引薦依據(jù)你最近的閱讀行為和消費(fèi)行為2014-032015-082017-102016-032018大數(shù)據(jù)工作首先寫入政府工作報(bào)告『十三五規(guī)劃綱要』提出『實(shí)施國(guó)家大數(shù)據(jù)戰(zhàn)略』』2018年《政府工作報(bào)告》提出：實(shí)施大數(shù)據(jù)發(fā)展行動(dòng)，注重用互聯(lián)網(wǎng)、大數(shù)據(jù)等提升監(jiān)管效能國(guó)務(wù)院發(fā)布《促進(jìn)大數(shù)據(jù)發(fā)展的行動(dòng)綱要》十九大提出推動(dòng)大數(shù)據(jù)戰(zhàn)略，與實(shí)體經(jīng)濟(jì)深度融合行業(yè)現(xiàn)狀與前景2014-032015-082017-102016-0320大數(shù)據(jù)采集與清洗課件2019年人社部擬最新發(fā)布15項(xiàng)新職業(yè)1.大數(shù)據(jù)工程技術(shù)人員2.云計(jì)算工程技術(shù)人員3.人工智能工程技術(shù)人員4.物聯(lián)網(wǎng)工程技術(shù)人員5....2019年人社部擬最新發(fā)布15項(xiàng)新職業(yè)1.大數(shù)據(jù)工程技術(shù)人員大數(shù)據(jù)采集與清洗課件大數(shù)據(jù)采集與清洗課件什么是大數(shù)據(jù)大數(shù)據(jù)（BigData）是指無法使用傳統(tǒng)和常用的軟件技術(shù)和工具在一定時(shí)間內(nèi)完成獲取、管理和處理的數(shù)據(jù)集什么是大數(shù)據(jù)大數(shù)據(jù)（BigData）是指無法使用傳統(tǒng)和常用2大數(shù)據(jù)的主要特征2大數(shù)據(jù)的主要特征大數(shù)據(jù)主要特征

VolumeVelocityVarietyVeracity真實(shí)性（Veracity），即追求高質(zhì)量的數(shù)據(jù)。容量大（Volume）,指大規(guī)模的數(shù)據(jù)量，并且數(shù)據(jù)量呈持續(xù)增長(zhǎng)趨勢(shì)。速度快（Velocity）,指的是數(shù)據(jù)被創(chuàng)建和移動(dòng)的速度。種類多（Variety）,指數(shù)據(jù)來自多種數(shù)據(jù)源，數(shù)據(jù)種類和格式。Value價(jià)值密度低（Value），指隨著數(shù)據(jù)量的增長(zhǎng)，數(shù)據(jù)中有意義的信息卻沒有成相應(yīng)比例增長(zhǎng)。大數(shù)據(jù)主要特征VolumeVelocityVarietyV3大數(shù)據(jù)處理流程3大數(shù)據(jù)處理流程大數(shù)據(jù)處理流程

數(shù)據(jù)預(yù)處理就是將采集來的數(shù)據(jù)從多種數(shù)據(jù)庫(kù)導(dǎo)入到大型的分布式數(shù)據(jù)庫(kù)中（目前主要是hfds或hive）,并同時(shí)做一些簡(jiǎn)單的清洗和預(yù)處理工作。數(shù)據(jù)統(tǒng)計(jì)分析就是對(duì)上面已經(jīng)完成的存儲(chǔ)在大型分布式數(shù)據(jù)庫(kù)中的數(shù)據(jù)進(jìn)行歸類統(tǒng)計(jì)，可以滿足一般場(chǎng)景的分析需求。數(shù)據(jù)挖掘是對(duì)數(shù)據(jù)進(jìn)行基于各種算法的分析計(jì)算，從而起到預(yù)測(cè)的效果，實(shí)現(xiàn)一些高級(jí)別數(shù)據(jù)分析的需求。數(shù)據(jù)采集就是利用多種數(shù)據(jù)庫(kù)（關(guān)系型，NOSQL）去存儲(chǔ)不同來源的數(shù)據(jù)。數(shù)據(jù)展示就是對(duì)以上處理完的結(jié)果進(jìn)行分析，或者形成報(bào)表。大數(shù)據(jù)處理流程數(shù)據(jù)預(yù)處理就是將采集來的數(shù)據(jù)從多種數(shù)據(jù)庫(kù)導(dǎo)4大數(shù)據(jù)采集的概念4大數(shù)據(jù)采集的概念大數(shù)據(jù)采集的概念3、大數(shù)據(jù)采集技術(shù)方法大數(shù)據(jù)采集技術(shù)就是對(duì)數(shù)據(jù)進(jìn)行ETL操作，通過對(duì)數(shù)據(jù)進(jìn)行提取、轉(zhuǎn)換、加載，最終挖掘數(shù)據(jù)的潛在價(jià)值。ETL指的是Extract-Transform-Load，也就是抽取、轉(zhuǎn)換、加載。抽取->從各種數(shù)據(jù)源獲取數(shù)據(jù)轉(zhuǎn)換->按需求格式將源數(shù)據(jù)轉(zhuǎn)換為目標(biāo)數(shù)據(jù)加載->把目標(biāo)數(shù)據(jù)加載到數(shù)據(jù)倉(cāng)庫(kù)中2、數(shù)據(jù)采集與大數(shù)據(jù)采集的區(qū)別傳統(tǒng)數(shù)據(jù)采集：來源單一，數(shù)據(jù)量相當(dāng)??；結(jié)構(gòu)單一；關(guān)系數(shù)據(jù)庫(kù)和并行數(shù)據(jù)庫(kù)大數(shù)據(jù)的數(shù)據(jù)采集：來源廣泛，數(shù)量巨大；數(shù)據(jù)類型豐富；分布式數(shù)據(jù)庫(kù)1、什么是數(shù)據(jù)采集數(shù)據(jù)采集就是數(shù)據(jù)獲取，數(shù)據(jù)源主要分為線上數(shù)據(jù)和內(nèi)容數(shù)據(jù)大數(shù)據(jù)采集的概念3、大數(shù)據(jù)采集技術(shù)方法2、數(shù)據(jù)采集與大數(shù)據(jù)采大數(shù)據(jù)采集系統(tǒng)1.日志采集系統(tǒng)（ApacheFlume、Scribe）3.數(shù)據(jù)庫(kù)采集系統(tǒng)（關(guān)系型、nosql等各種數(shù)據(jù)庫(kù)）2.網(wǎng)絡(luò)數(shù)據(jù)采集系統(tǒng)（Scrapy框架、ApacheNutch）大數(shù)據(jù)采集系統(tǒng)1.日志采集系統(tǒng)（ApacheFlume、S5大數(shù)據(jù)采集應(yīng)用5大數(shù)據(jù)采集應(yīng)用大數(shù)據(jù)采集與清洗課件技能準(zhǔn)備Python基礎(chǔ)Linux操作系統(tǒng)基本操作數(shù)據(jù)庫(kù)基礎(chǔ)（SQL語(yǔ)句操作）技能準(zhǔn)備Python基礎(chǔ)Linux操作系統(tǒng)基本操作數(shù)據(jù)庫(kù)基礎(chǔ)環(huán)境準(zhǔn)備PythonJdk(java環(huán)境)數(shù)據(jù)庫(kù)（mysql）環(huán)境準(zhǔn)備PythonJdk(java環(huán)境)數(shù)據(jù)庫(kù)（mysqlThanks

ThanksYOURTITLE

Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.YOURTITLENothingisdifficulYOURTITLE

Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.YOURTITLENothingisdifficul2OKPPT工作室2OKPPT工作室YOURTITLE

Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.YOURTITLENothingisdifficulYOURTITLE

Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.YOURTITLENothingisdifficulYOURTITLE

21%9%28%42%YOURTITLE21%9%28%42%3OKPPT工作室3OKPPT工作室YOURTITLE

Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.YOURTITLENothingisdifficulYOURTITLE

Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.YOURTITLENothingisdifficulYOURTITLE

Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficul