課程九、大數(shù)據(jù)核心開(kāi)發(fā)技術(shù)-內(nèi)存計(jì)算框架spark精講課件0502rdd

1、大數(shù)據(jù)Hadoop高薪直通車課程SparkRDD)講師：軒宇（北風(fēng)網(wǎng)1Spark RDD 特性2Spark R3賴RDD ShuffleSpark 內(nèi)核分析課程大綱1Spark RDD 特性2Spark R3賴RDD ShuffleSpark 內(nèi)核分析課程大綱val rdd=sc.textFile(hdfs:/bigdata-:8020/user/beifeng/spark/wc.input)val wordsort=wordcount.map(x=(x._2,x._1).sortByKey(false).map(x=(x._2,x._1)wordsort.collect()val wordc

2、ount=rdd.flatMap(_.split( ).map(_,1).reduceByKey(_ + _) wordcount.collect()Spark WordCountHDFSsc.textFile(“path”)Memoryrdd.flatMap(line = line.split( )MemoryRDDStringRDDStringhdf :/:8020/user/hadoop/spark/wc.inputRDDA Resint Distributed Dataset (RDD), the basic abstraction in Spark.Represents an imm

3、utable, partitioned collection of elementst can be operated on in parallel.RDDRDD1Spark RDD 特性2Spark R3賴RDD ShuffleSpark 內(nèi)核分析課程大綱Parallelized CollectionsExternal DatasetsCreate RDDsParallelized CollectionsExternal DatasetsRDDResint Distributed Datasets13ActionsPersistenceTransformationsCreate a new

4、datasetfrom and existing one.Returns to the driver program a value orFor caching datasets emory for futureoperations.exports da system aft a compuo a storageerformingion.Lazy in nature. They are executed only when some action is performed.Option to store on disk or RAM or mixed (Storage Level).Examp

5、le:count() reduce(func) collect take()Example:Example :persist() cache()map(func) filter(func) distinct() RDDCreate new datasets from existing onesmap()ersection()carte()flatMap()distinct()pipe()filter()groupByKey()coalesce()mapPartitions()reduceByKey()repartition()mapPartitionsWithIndex()sortByKey(

6、)partitionBy()sample()join().lazyunion()cogroup.Transformations arent appd to an RDD until an action is executed;Spark remembers set of transformations appd to base dataset;Transformationreduce()takeOrdered()collect()saveAsTextFile()count()saveAsSequenceFile()()saveAsObjectFile()take()countByKey()ta

7、keSample()foreach()eagersaveToCassandra().Cause Spark to execute recipe to transform source;Cause dao be returned to driver or saved to output;ActionRDD Persistence1Spark RDD 特性2Spark R3賴RDD ShuffleSpark 內(nèi)核分析課程大綱RDDRDD Dependencies窄依賴（ narrow dependencies ）子 RDD 的每個(gè)分區(qū)依賴于常數(shù)個(gè)父分區(qū)（即與數(shù)據(jù)規(guī)模無(wú)關(guān)）輸入輸出一對(duì)一的算子，且結(jié)

8、果 RDD 的分區(qū)結(jié)構(gòu)不變，主要是 map 、 flatMap輸入輸出一對(duì)一，但結(jié)果 RDD 的分區(qū)結(jié)構(gòu)發(fā)生了變化，如 union 、 coalesce從輸入中選擇部分元素的算子，如 filter 、 distinct 、 subtract 、 sample寬依賴（ wide dependencies ）子 RDD 的每個(gè)分區(qū)依賴于所有父 RDD 分區(qū)對(duì)單個(gè) RDD 基于 key 進(jìn)行重組和 reduce ，如 groupByKey 、 reduceByKey ；對(duì)兩個(gè) RDD 基于 key 進(jìn)行 join 和重組，如 joinRDD Dependencies1Spark RDD 特性2Spa

9、rk R3賴RDD ShuffleSpark 內(nèi)核分析課程大綱Spark Shufflespark ShuffleThe shuffle is Sparks mechanism for re-distributing data那些操作會(huì)引起Shuffle？具有重新調(diào)整分區(qū)操作，eg：repartition,coalesce*BeyKeyeg: groupByKey ,reduceByKey關(guān)聯(lián)操作 eg： join ，cogroupRDD Shuffle1Spark RDD 特性2Spark R3賴RDD ShuffleSpark 內(nèi)核分析課程大綱Initializing SparkSpark

10、 SchedulerDAG Scheduler接收用戶提交的job構(gòu)建 Stage，哪個(gè)RDD 或者 Stage 輸出被物化重新提交 shuffle 輸出丟失的 stage將Taskset 傳給底層調(diào)度器DAG SchedulerTask Scheduler提交 taskset( 一組 task) 到集群運(yùn)行并為每一個(gè) TaskSet 構(gòu)建一個(gè) TaskSetManager 實(shí)例管理這個(gè) TaskSet的生命周期數(shù)據(jù)本地性決定每個(gè) Task 最佳位置 (pros-local, nocal,rack-local andthen any)推測(cè)執(zhí)行，碰到輸出 lost 要straggle 任務(wù)需要放到別的節(jié)點(diǎn)上重試出現(xiàn) shufflefetch failed 錯(cuò)誤Task SchedulerPartition & TaskTask是Executor中的執(zhí)行單元T