云計算技術及應用課件

云計算技術及應用大連理工大學計算機科學與技術學院2023年春季基本狀況申彥明B810助教：齊恒B812Officehour:Fri3:30-4:30PMCoursewebsite:教材內容Project論文教材內容分布式系統(tǒng)旳概況分布式與集群基本概念分布式數(shù)據(jù)庫分布式文獻系統(tǒng)GFS分布式編程MapReduce算法簡介搜索引擎與PageRank其他有關技術DataCenterBigTableAppEngineGradingHW：40%FinalProject:60%FinalprojectproposalProjectreports12teams,4-5studentsSyllabus(Subjecttochange)Week2Mar8:Lecture1:IntroductionMar10:Lecture2:Map/ReduceTheoryandImplementation,HadoopWeek3Mar15:Lecture3&4:GuestSpeaker(8:00AM-11:35AM研教樓102)Mar17:Lecture5:DistributedFileSystemandtheGoogleFileSystemWeek4Mar22:Lecture6&7:GuestSpeaker(8:00AM-11:35AM研教樓102)Mar24:Lecture8:DistributedGraphAlgorithmsandPageRankWeek5Mar29:Lecture9:IntroductiontoSomeProjectsMar31:Lecture10:DataCentersSyllabus(Subjecttochange)Week6Apr5:Lecture11:SomeGoogleTechnologiesApr7:Lecture12:VirtualizationWeek7Lecture13&14:ProjectPresentationWeek8:NoclassWeek9:Lecture15&16:ProjectPresentationGartnerReportTop10StrategicTechnologyAreas

for2023VirtualizationCloudComputingServers:BeyondBladesWeb-OrientedArchitecturesEnterpriseMashupsSpecializedSystemsSocialSoftwareandSocialNetworkingUnifiedCommunicationsBusinessIntelligenceGreenInformationTechnologyTop10StrategicTechnologyAreasfor2023CloudComputingAdvancedAnalyticsClientComputingITforGreenReshapingtheDataCenterSocialComputingSecurity–ActivityMonitoringFlashMemoryVirtualizationforAvailabilityMobileApplicationsFromDesktop/HPC/Gridsto

InternetCloudsin30YearsHPCmovingfromcentralizedsuperputerstogeographicallydistributeddesktops,clusters,andgridstocloudsoverlast30yearsR/DeffortsonHPC,clusters,Grids,P2P,andvirtualmachineshaslaidthefoundationofcloudputingthathasbeengreatlyadvocatedsince2023Locationofputinginfrastructureinareaswithlowercostsinhardware,software,datasets,space,andpowerrequirements–movingfromdesktopputingtodatacenter-basedcloudsWhatisCloudComputing?1.Web-scaleproblems2.Largedatacenters3.Differentmodelsofputing4.Highly-interactiveWebapplications1.“Web-Scale”Problems

Characteristics:Definitelydata-intensiveMayalsobeprocessingintensiveExamples:Crawling,indexing,searching,miningtheWebDatawarehousesSensornetworks“Post-genomics”lifesciencesresearchOtherscientificdata(physics,astronomy,etc.)Web2.0applications…Howmuchdata?Googleprocesses20PBaday(2023)“allwordseverspokenbyhumanbeings”~5EBCERN’sLHCwillgenerate10-15PBayear640K

oughttobeenoughforanybody.Whattodowithmoredata?AnsweringfactoidquestionsPatternmatchingontheWebWorksamazinglywellLearningrelationsStartwithseedinstancesSearchforpatternsontheWebUsingpatternstofindmoreinstancesHowdoImakemoney?Petabytesofvaluablecustomerdata…SittingidleinexistingdatawarehousesOverflowingoutofexistingdatawarehousesSimplybeingthrownawaySourceofdata:OLTPUserbehaviorlogsCall-centerlogsWebcrawls,publicdatasets…Structureddata(today)vs.unstructureddata(tomorrow)Howcananorganizationderivevaluefromallthisdata?2.LargeDataCentersWeb-scaleproblems?Throwmoremachinesatit!CentralizationofresourcesinlargedatacentersNecessaryingredients:fiber,juice,andlandWhatdoOregon,Iceland,andabandonedmineshaveinmon?ImportantIssues:EfficiencyRedundancyUtilizationSecurityManagementoverhead3.DifferentComputingModelsUtilityputingWhybuymachineswhenyoucanrentcycles?Examples:Amazon’sEC2PlatformasaService(PaaS)GivemeniceAPIandtakecareoftheimplementationExample:GoogleAppEngineSoftwareasaService(SaaS)Justrunitforme!Example:Gmail“Whydoityourselfifyoucanpaysomeonetodoitforyou?”4.WebApplicationsWhatisthenatureoffuturesoftwareapplications?FromthedesktoptothebrowserSaaS==Web-basedapplicationsExamples:GoogleMaps,FacebookHowdowedeliverhighly-interactiveWeb-basedapplications?AJAX(asynchronousJavaScriptandXML)Ahackontopofamistakebuiltonsand,allheldtogetherbyducttapeandchewinggum?SomeCloudDefinitionsIanFosteretaldefinedcloudputingasalarge-scaledistributedputingparadigm,thatisdrivenbyeconomicsofscale,inwhichapoolofabstractedvirtualized,dynamically-scalable,managedputingpower,storage,platforms,andservicesaredeliveredondemandtoexternalcustomersovertheinternet(云計算是一種商業(yè)計算模型。它將計算任務分布在大量計算機構成旳資源池上，使多種應用系統(tǒng)可以根據(jù)需要獲取計算力、存儲空間和多種軟件服務。)IBMexpertsconsidercloudsthatcan:Hostavarietyofdifferentworkloads,includingbatch-stylebackendinteractive,user-facingapplicationsAllowworkloadstobedeployedandscaled-outquicklythroughtherapidprovisioningofvirtualmachinesorphysicalmachinesSupportredundant,self-recovering,highlyscalableprogrammingmodelsthatallowworkloadstorecoverfromHW/SWfailuresMonitorresourceuseinrealtimetorebalanceallocationsondemandInternetCloudGoals

Sharingofpeak-loadcapacityamongalargepoolofusers,improvingoverallresourceutilizationSeparationofinfrastructuremaintenancedutiesfromdomain-specificapplicationdevelopmentMajorcloudapplicationsincludeupgradedwebservices,distributeddatastorage,rawsuperputing,andaccesstospecializedGrid,P2P,data-mining,andcontentnetworkingservicesThreeAspectsinHardwarethat

areNewinCloudComputingTheillusionofinfiniteputingresourcesavailableondemand,therebyeliminatingtheneedforclouduserstoplanfaraheadforprovisioningTheeliminationofanup-frontmitmentbycloudusers,therebyallowingpaniestostartsmallandincreasehardwareresourceswhenneededTheabilitytopayputingresourcesonashort-termbasisasneeded(e.g.,processorsbythehourandstoragebytheday)andreleasethemafterdoneandtherebyrewardingresourceconservationSomeInnovativeCloudServices

andApplicationOpportunitiesSmartandpervasivecloudapplicationsforindividuals,homes,munities,panies,andgovernments,etc.CoordinatedCalendar,Itinerary,jobmanagement,events,andconsumerrecordmanagement(CRM)servicesCoordinatedwordprocessing,on-linepresentations,web-baseddesktops,sharingon-linedocuments,datasets,photos,video,anddatabases,etcDeployconventionalcluster,grid,P2P,socialnetworkingapplicationsincloudenvironments,morecost-effectivelyEarthboundApplicationsthatDemandElasticityandParallelismratherdatamovementCostsOperationsinCloudComputingUsersinteractwiththecloudtorequestserviceProvisioningtoolcarvesoutthesystemsfromthecloud–configurationorreconfiguration,ordeprovisionTheserverscanbeeitherrealorvirtualmachinesSupportingresourcesincludedistributedstoragesystem,datacenters,securitydevices,etc.CloudComputingInstancesGoogleAmazonMicrosoftAzureIBMBlueCloudGoogleCloudInfrastructureSchedulerChubbyGFSmasterNodeNodeNode…UserApplicationSchedulerslaveGFSchunkserverLinuxNodeMapReduceJobBigTableServerGoogleCloudInfrastructureS3EBSEC2EBSEC2EBSEC2EBSEC2SimpleDBSQSUserDeveloperAmazonElasticComputingCloudSQS:SimpleQueueServiceEC2:RunningInstanceofVirtualMachinesEBS:ElasticBlockService,ProvidingtheBlockInterface,StoringVirtualMachineImagesS3:SimpleStorageService,SOAP,ObjectInterfaceSimpleDB:SimplifiedDatabase

Azure?ServicesPlatformMicrosoftAzurePlatformDeveloperMonitoringApplicationServerProvisioningManagerUserOpenSourceLinuxwithXenTivoliMonitoringAgent……IBMBlueCloudCostConsiderations:Power,Cooling,

PhysicalPlant,andOperationalCostsCosttechnologycostscostofsecurityetc.Benefitsavailabilityopportunityconsolidationetc.CostBreakdown+Storage($/MByte/year)+Computing($/CPUCycles)+Networking($/bit)ResearchChallengesServiceavailabilityS3outage:authenticationserviceoverloadleadingtounavailabilityAppEnginepartialoutageprogrammingerrorGmail:siteunavailableSolutions:ThemanagementofaCloudComputingservicebyasinglepanyresultsinasinglepointoffailure(SPF).IntheInternet,alargeISPusesmultiplenetworkproviderssothatfailurebyasinglepanywillnottakethemofftheair.Similarly,weneedmultipleCloudComputingproviderstosupporteachothertoeliminateSPF.ResearchChallengesDataSecurityCurrentcloudofferingsareessentiallypublicratherthanprivatenetworks,exposingthesystemtomoreattackssuchasDDoSattacks.Solutions:Therearemanywellunderstoodtechnologiessuchasencryptedstorage,virtuallocalareanetworks,andnetworkmiddleboxes.ResearchChallengesDataTransferBottlenecksApplicationscontinuetobeemoredata-intensive.Ifweassumeapplicationsmaybe“pulledapart”acrosstheboundariesofclouds,thismayplicatedataplacementandtransport.BothWANbandwidthandintra-cloudnetworkingtechnologyareperformancebottleneck.Industrialsolutions:Itisestimatedthat2/3ofthecostofWANbandwidthisconsumedbyhigh-endrouters,whereasonly1/3chargedbyfiberindustry.Wecanlowerthecostbyusingsimplerroutersbuiltfrommodityponentswithcentralizedcontrol,butresearchisheadingtowardsusinghigh-enddistributedrouters.ResearchChallengesSoftwareLicensingCurrentsoftwarelicensesmonlyrestricttheputersonwhichthesoftwarecanrun.Userspayforthesoftwareandthenpayanannualmaintenancefee.ManycloudputingprovidersoriginallyreliedonopensourcesoftwareinpartbecausethelicensingmodelformercialsoftwareisnotagoodmatchtoUtilityComputing.Someideas:WecanencouragesalesforcesofsoftwarepaniestosellproductsintoCloudComputing.Ortheycanimplementpay-per-usemodeltothesoftwaretoadapttoacloudenvironment.ResearchChallengesScalablestorageDifferencesbetweenmonstorageandcloudstorageThesystemisbuiltfrommanyinexpensivemodityponentsthatoftenfailThesystemstoresamodestnumberoflargefilesTheworkloadsprimarilyconsistbothlargestreamingreadsandsmallrandomreads.Theworkloadsmanylarge,sequentialwritesthatappenddatatofilesandoncewritten,filesareseldommodifiedagain.Thecloudstorage(file)systemneedstosharemanyofthesamegoalsaspreviousdistributedfilesystemssuchasperformance,scalability,reliability,andavailability.Inaddition,itsdesignneedstobedrivenbykeyobservationsofthespecificworkloadsandtechnologicalenvironment,bothcurrentandanticipated,thatreflectamarkeddeparturefromsomeearlierfilesystemdesignassumptions.GFSFilesaredividedintofixed-sizechunks,Chunksizeisoneofthekeydesignparameters.GFSchooses64MB,whichismuchlargerthantypicalfilesystemblocksizes.Themasterstoresthreemajortypesofmetadata:thefileandchunknamespaces,themappingfromfilestochunks,andthelocationsofeachchunk’sreplicas.GFSsupportstheusualoperationstocreate,delete,open,close,read,andwritefiles.ResearchChallengesTransparentProgrammingModelProgramswrittenforcloudimplementationneedtobeautomaticallyparallelizedandexecutedonalargeclusterofmoditymachines.Therun-timesystemshouldtakecareofthedetailsofpartitioningtheinputdata,schedulingtheprogram'sexecutionacrossasetofmachines,handlingmachinefailures,andmanagingtherequiredinter-machinemunication.Theprogrammingmodelshouldallowprogrammerswithoutmanyexperienceswithparallelanddistributedsystemstoeasilyutilizetheresourcesofalargedistributedsystem.MapReduceScalableDataProcessingonLargeClustersAwebprogrammingmodelimplementedforfastprocessinga