大語(yǔ)言模型綜述 A Survey of Large Language Models

yofLargeLanguageModelsWayneXinZhao,KunZhou*,JunyiLi*,TianyiTang,XiaoleiWang,YupengHou,YingqianMin,BeichenZhang,JunjieZhang,ZicanDong,YifanDu,ChenYang,YushuoChen,ZhipengChen,JinhaoJiang,RuiyangRen,YifanLi,XinyuTang,ZikangLiu,PeiyuLiu,Jian-YunNieandJi-RongWenAbstract—EversincetheTuringTestwasproposedinthe1950s,humanshaveexploredthemasteringoflanguageintelligencebymachine.Languageisessentiallyacomplex,intricatesystemofhumanexpressionsgovernedbygrammaticalrules.Itposesasigni?cantchallengetodevelopcapablearti?cialintelligence(AI)algorithmsforcomprehendingandgraspingalanguage.Asamajorapproach,languagemodelinghasbeenwidelystudiedforlanguageunderstandingandgenerationinthepasttwodecades,evolvinglargescalecorporashowingstrongcapabilitiesinsolvingvariousnaturallanguageprocessingNLPhatmodelscalingcanleadtoanimprovedmodelcapacitytheyfurtherinvestigatethescalingeffectbyincreasingtheparameterscaletoanevenlargersize.Interestingly,whentheparameterscaleexceedsacertainlevel,theseenlargedlanguagemodelsnotonlyachieveasigni?cantperformanceimprovement,butalsoexhibitsomespecialabilities(e.g.,in-contextlearning)thatarenotpresentinsmall-scalelanguagemodels(e.g.,BERT).Todiscriminatethelanguagemodelsindifferentparameterscales,theresearchcommunityhascoinedthetermlargelanguagemodels(LLM)forthePLMsofsigni?cantsize(e.g.,containingtensorhundredsofbillionsofparameters).Recently,theresearchonLLMshasbeenlargelyadvancedbybothacademiaandindustry,andaremarkableprogressisthelaunchofChatGPT(apowerfulAIchatbotdevelopedbasedonLLMs),whichhasattractedwidespreadattentionfromsociety.ThetechnicalevolutionofLLMshasbeenmakinganimportantimpactontheentireAItywhichwouldrevolutionizethewayhowwedevelopanduseAIalgorithmsConsideringthisrapidtechnicalprogressinthissurveywereviewtherecentadvancesofLLMsbyintroducingthebackground,key?ndings,andmainstreamtechniques.Inparticular,wefocusonfourmajoraspectsofLLMs,namelypre-training,adaptationtuning,utilization,andcapacityevaluation.Besides,wealsoleresourcesfordevelopingLLMsanddiscusstheremainingissuesforfuturedirectionsThissurveyprovidesanuptodatereviewoftheliteratureonLLMs,whichcanbeausefulresourceforbothresearchersandengineers.AlignmentCapacityEvaluation ◆L1INTRODUCTIONL ANGUAGEisaprominentabilityinhumanbeingsto expressandcommunicate,whichdevelopsinearlychildhoodandevolvesoveralifetime[1,2].Machines,however,cannotnaturallygrasptheabilitiesofunderstand-ingandcommunicatingintheformofhumanlanguage,unlessequippedwithpowerfularti?cialintelligence(AI)algorithms.Ithasbeenalongstandingresearchchallengetoachievethisgoal,toenablemachinestoread,write,andcommunicatelikehumans[3].Technically,languagemodeling(LM)isoneofthemajoresIngeneral,LMaimstomodelthegenerativelikelihoodofwordsequences,soastopredicttheprobabilitiesoffuture(ormissing)tokens.TheresearchofLMhasreceivedextensiveattentionintheliterature,whichcanbedividedintofourmajordevelopmentstages:lsSLMSLMsaredevelopedbasedonstatisticallearningmethodsthatroseinthe1990s.ThebasicideaistobuildthewordpredictionmodelbasedontheMarkovassumption,e.g.,predictingthenextwordbasedonthemostrecentcontextTheSLMswith●Version:v5(updateonApril16,2023).●GitHublink:/RUCAIBox/LLMSurvey●*K.ZhouandJ.Licontributeequallytothiswork.●TheauthorsaremainlywithGaolingSchoolofArti?cialIntelligenceandSchoolofInformation,RenminUniversityofChina,Beijing,China;Jian-versitedeMontrealCanadaa?xedcontextlengthnarealsocalledn-gramlanguagemodels,e.g.,bigramandtrigramlanguagemodels.SLMshavebeenwidelyappliedtoenhancetaskperformanceininformationretrieval(IR)[8,9]andnaturallanguageprocessing(NLP)[10–12].However,theyoftensufferfromthecurseofdimensionality:itisdif?culttoaccuratelyestimatehigh-orderlanguagemodelssinceanexponentialnumberoftransitionprobabilitiesneedtobeestimated.Thus,speciallydesignedsmoothingstrategiessuchasback-offestimation[13]andGood–Turingestimation[14]havebeenintroducedtoalleviatethedatasparsityproblem.NeurallanguagemodelsNLMNLMscharacterizetheprobabilityofwordsequencesbyneuralnetworks,e.g.,recurrentneuralnetworks(RNNs).Asaremarkablecontribution,theworkin[15]introducedtheconceptofdistributedrepresentationofwordsandbuiltthewordpredic-tionfunctionconditionedontheaggregatedcontextfeatures(i.e.,thedistributedwordvectors).Byextendingtheideaoflearningeffectivefeaturesforwordsorsentences,ageneralneuralnetworkapproachwasdevelopedtobuildauni?edsolutionforvariousNLPtasks[18].Further,word2vec[19,20]wasproposedtobuildasimpli?edshal-lowneuralnetworkforlearningdistributedwordrepresen-tations,whichweredemonstratedtobeveryeffectiveacrossavarietyofNLPtasks.Thesestudieshaveinitiatedtheuseoflanguagemodelsforrepresentationlearning(beyondwordsequencemodeling),havinganimportantimpactonthe?eldofNLP.tempt,ELMo[21]wasproposedtocapturecontext-awarewordrepresentationsby?rstpre-trainingabidirectionalLSTM(biLSTM)network(insteadoflearning?xedwordrepresentations)andthen?ne-tuningthebiLSTMnetworkaccordingtospeci?cdownstreamtasks.Further,basedonthehighlyparallelizableTransformerarchitecture[22]withanismsBERTwasproposedbypretrainingbidirectionallanguagemodelswithspeciallyde-signedpre-trainingtasksonlarge-scaleunlabeledcorpora.Thesepre-trainedcontext-awarewordrepresentationsareveryeffectiveasgeneral-purposesemanticfeatures,whichhavelargelyraisedtheperformancebarofNLPtasks.Thisstudyhasinspiredalargenumberoffollow-upwork,whichsetsthe“pre-trainingand?ne-tuning”learningparadigm.Followingthisparadigm,agreatnumberofstudiesonPLMshavebeendeveloped,introducingeitherdifferentarchitectures[24,25](e.g.,GPT-2[26]andBART[24])orimprovedpre-trainingstrategies[27–29].Inthisparadigm,itoftenrequires?ne-tuningthePLMforadaptingtodifferentdownstreamtasks.scalingPLM(e.g.,scalingmodelsizeordatasize)oftenleadstoanimprovedmodelcapacityondownstreamtasks(i.e.,followingthescalinglaw[30]).AnumberofstudieshaveexploredtheperformancelimitbytraininganeverlargerPLM(e.g.,the175B-parameterGPT-3andthe540B-parameterPaLM).Althoughscalingismainlyconductedinmodelsize(withsimilararchitecturesandpre-trainingtasks),theselarge-sizedPLMsdisplaydifferentbehaviorsfromsmallerPLMs(e.g.,330M-parameterBERTand1.5B-gentabilities[31])insolvingaseriesofcomplextasks.ForshottasksthroughincontextereasGPTcannotdowellThustheresearchtheselarge-sizedPLMs[32–35].AremarkableapplicationofLLMsisChatGPT2thatadaptstheLLMsfromtheGPTingconversationabilitywithhumans.Intheexistingliterature,PLMshavebeenwidelydis-cussedandsurveyed[36–39],whileLLMsareseldomre-viewedinasystematicway.Tomotivateoursurvey,we?rsthighlightthreemajordifferencesbetweenLLMsandPLMs.First,LLMsdisplaysomesurprisingemergentabilitiesthatmaynotbeobservedinprevioussmallerPLMs.Theseabili-rmanceoflanguagemodelsoncomplextasks,makingAIalgorithmsunprecedentlypowerfulandeffective.Second,LLMswouldrevolutionizethewaythathumansdevelopanduseAIalgorithms.UnlikesmallPLMs,themajorapproachtoaccessingLLMsisthroughthepromptinginterface(e.g.,GPT-4API).HumanshavetounderstandhowLLMsworkandformattheirtasksinawaythatLLMscanfollow.Third,thedevelopmentofLLMsnolongerdrawsacleardistinctionbetweenresearchanden-gineering.ThetrainingofLLMsrequiresextensivepracticalexperiencesinlarge-scaledataprocessinganddistributedanasmallPLMandemergentabilitiesmaynotoccurinsomeLLMs.2./blog/chatgpt/paralleltraining.TodevelopcapableLLMs,researchershavetosolvecomplicatedengineeringissues,workingwithineersNowadays,LLMsareposingasigni?cantimpactontheAIcommunity,andtheadventofChatGPTandGPT-4leadstotherethinkingofthepossibilitiesofarti?cialgeneralintelligence(AGI).OpenAIhaspublishedatechnicalarticleentitled“PlanningforAGIandbeyond”,whichdiscussestheshort-termandlong-termplanstoapproachAGI[40],andamorerecentpaperhasarguedthatGPT-4mightbeconsideredasanearlyversionofanAGIsystem[41].TheresearchareasofAIarebeingrevolutionizedbytherapidprogressofLLMs.Inthe?eldofNLP,LLMscanserveasageneral-purposelanguagetasksolver(tosomeextent),andtheresearchparadigmhasbeenshiftingtowardstheuseofLLMs.Inthe?eldofIR,traditionalsearchenginesarechallengedbythenewinformationseekingwaythroughAIchatbots(i.e.,ChatGPT),andNewBing3presentsaninitialattemptthatenhancesthesearchresultsbasedonLLMsInthe?eldofCV,theresearcherstrytodevelopChatGPT-likevision-languagemodelsthatcanbetterservemultimodaldialoguesandGPT-4[46]hassupportedmulti-modalinputbyintegratingthevisualinformation.Thisnewwaveoftechnologywouldpotentiallyleadtoaprosperousecosystemofreal-worldapplicationsbasedonLLMs.Forinstance,Microsoft365isbeingempoweredbyLLMs(i.e.,Copilot)toautomatetheof?cework,andOpenAIsupportstheuseofpluginsinChatGPTforimplementingspecialfunctions.Despitetheprogressandimpact,theunderlyingprin-ciplesofLLMsarestillnotwellexplored.Firstly,itismysteriouswhyemergentabilitiesoccurinLLMs,insteadofsmallerPLMs.Asamoregeneralissue,therelacksadeep,detailedinvestigationofthekeyfactorsthatcontributetothesuperiorabilitiesofLLMs.ItisimportanttostudywhenandhowLLMsobtainsuchabilities[47].Althoughtherearesomemeaningfuldiscussionsaboutthisproblem[31,47],moreprincipledinvestigationsareneededtouncoverthe“secrets“ofLLMs.Secondly,itisdif?cultfortheresearchcommunitytotraincapableLLMs.Duetothehugede-mandofcomputationresources,itisverycostlytocarryoutrepetitive,ablatingstudiesforinvestigatingtheeffectofvariousstrategiesfortrainingLLMs.Indeed,LLMsaremainlytrainedbyindustry,wheremanyimportanttrainingdetails(e.g.,datacollectionandcleaning)arenotrevealedtothepublic.Thirdly,itischallengingtoalignLLMswithhumanvaluesorpreferences.Despitethecapacities,LLMsarealsolikelytoproducetoxic,?ctitious,orharmfulcon-tents.Itrequireseffectiveandef?cientcontrolapproachestoeliminatingthepotentialriskoftheuseofLLMs[46].Facedwithbothopportunitiesandchallenges,itneedsmoreattentionontheresearchanddevelopmentofLLMs.InordertoprovideabasicunderstandingofLLMs,thissurveyconductsaliteraturereviewoftherecentadvancesinLLMsfromfourmajoraspects,includingpre-training(howtopre-trainacapableLLM),adaptationtuning(howtoeffectivelytunepre-trainedLLMsfromthetwoperspectivesofeffectivenessandsafety),utilization(howtouseLLMsforsolvingvariousdownstreamtasks)andcapabilityeval-3./newuation(howtoevaluatetheabilitiesofLLMsandexistingempirical?ndings).Wethoroughlycombtheliteratureandsummarizethekey?ndings,techniques,andmethodsofLLMs.Forthissurvey,wealsocreateaGitHubprojectwebsitebycollectingthesupportingresourcesforLLMs,atthelink/RUCAIBox/LLMSurvey.WearealsoawareofseveralrelatedreviewarticlesonPLMsorLLMs[32,36,38,39,43,48–54].ThesepaperseitherdiscussPLMsorsomespeci?c(orgeneral)aspectsofLLMs.Comparedwiththem,wefocusonthetechniquesandmethodstodevelopanduseLLMsandprovidearelativelycomprehensivereferencetoimportantaspectsofLLMs.Theremainderofthissurveyisorganizedasfollows:Section2introducesthebackgroundforLLMs,withtheterminology,settings,resources,andorganizationoutline,followedbythesummarizationofavailableresourcesfordevelopingLLMsinSection3.Sections4,5,6,and7reviewandsummarizetherecentprogressfromthefouraspectsofpre-training,adaptationtuning,utilization,andcapacityevaluation,respectively.Finally,weconcludethesurveyinSection8bysummarizingthemajor?ndingsanddiscusstheremainingissuesforfuturework.2OVERVIEWInthissection,weintroducethebackgroundofLLMswithkeyterminologies,abilitiesandtechniques,andthensum-marizethetechnicalevolutionoftheGPT-seriesmodels.2.1BackgroundforLLMsargelanguagemodelsLLMsrefertolanguagemodelsthatcontainhundredsofbillions(ormore)ofpa-rameterswhicharetrainedonmassivetextdata[32],suchasGPT-3[55],PaLM[56],Galactica[35],andLLaMA[57].Speci?cally,LLMsarebuiltupontheTransformerarchi-tecture[22],wheremulti-headattentionlayersarestackedinaverydeepneuralnetwork.ExistingLLMsmainlyadoptsimilarmodelarchitectures(i.e.,Transformer)andpre-trainingobjectives(i.e.,languagemodeling)assmalllanguagemodels.Asthemajordifference,LLMslargelyscalethemodelsize,pre-trainingdata,andtotalcompute(ordersofmagni?cation).Theycanbetterunderstandthenaturallanguageandgeneratehigh-qualitytextbasedonthegivencontext(i.e.,prompts).Suchacapacityimprove-mentcanbepartiallydescribedbythescalinglaw,wheretheperformanceroughlyfollowsasubstantialincreasewithoweversomeabilitiesegin-contextlearning[55])areunpredictableaccordingtothescalinglaw,whichcanbeobservedonlywhenthemodelabilitiesofLLMsareformallyde?nedas“theabilitiesthatarenotpresentinsmallmodelsbutariseinlargemodels”,whichisoneofthemostprominentfeaturesthatdistin-guishLLMsfrompreviousPLMs.ItfurtherintroducesaInexistingliterature,thereisnoformalconsensusontheminimumparameterscaleforLLMs,sincethemodelcapacityisalsorelatedtodatasizeandtotalcompute.Inthissurvey,wetakeaslightlyloosede?nitionofLLMs,andmainlyfocusondiscussinglanguagemodelswithamodelsizelargerthan10B.notablecharacteristicwhenemergentabilitiesoccur[31]:performancerisessigni?cantlyaboverandomwhenthelevelByanalogysuchanemergenttransitioninphysics[31,58].Inprinciple,emergentabilitiescanbede?nedinrelationtosomecomplextasks[31,59],whilewearemoreconcernedwithgeneralabilitiesthatcanbeappliedtosolveavarietyoftasks.Here,webrie?yintroducethreerepresentativeemergentabilitiesforLLMs,describedasfollows.ntroducedbyGPTassumingthatthelanguagemodelhasbeenprovidedwithanaturallanguageinstruc-tionand/orseveraltaskdemonstrations,itcangeneratetheexpectedoutputforthetestinstancesbycompletingthewordsequenceofinputtext,withoutrequiringadditionaltrainingorgradientupdate5.●Instructionfollowing.By?ne-tuningwithamixtureofmulti-taskdatasetsformattedvianaturallanguagedescrip-tions(calledinstructiontuning),LLMsareshowntoperformwellonunseentasksthatarealsodescribedintheformofinstructions[28,61,62].Withinstructiontuning,LLMsareenabledtofollowthetaskinstructionsfornewtaskswithoutusingexplicitexamples,thushavinganimprovedgeneralizationability.usuallydif?culttosolvecomplextasksthatinvolvemultiplereasoningsteps,e.g.,mathematicalwordproblems.While,withthechain-of-thoughtreasoningstrategy[33],LLMscansolvesuchtasksbyutilizingthepromptingmechanismthatinvolvesintermediatereasoningstepsforderivingthe?nalanswer.Thisabilityisspeculatedtobepotentiallyobtainedbytrainingoncode[33,47].KeyTechniquesforLLMs.Ithasbeenalongwaythatlearners.Inthedevelopmentprocess,anumberofimpor-tanttechniquesareproposed,whichlargelyimprovethecapacityofLLMs.Here,webrie?ylistseveralimportanttechniquesthat(potentially)leadtothesuccessofLLMs,asfollows.oincreasethemodelcapacityofLLMs.Astheinitialattempt,GPT-3?rstlyin-creasesthemodelsizetoanextremelylargescaleof175Bparameters.Lateron,PaLMfurtherraisestheparameterscaletoanewrecordof540B.Asdiscussedbefore,alargemodelsizeisessentialtoemergentabilities.While,scalingisnotonlyconductedonmodelsizebutalsorelatedtodatasizeandtotalcompute[34,63].Arecentstudy[34]ofmodelsize,datasize,andtotalcompute,givena?xedbudget.Further,thequalityofthepre-trainingdataplaysakeyroleinachievinggoodperformance,sothatdatacollectionandcleaningstrategiesareveryimportanttoconsiderwhenscalingthepre-trainingcorpora.●Training.Duetothehugemodelsize,itisverychal-lengingtosuccessfullytrainacapableLLM.Distributedtrainingalgorithmsareneededtolearnthenetworkparam-5.Insomerecentstudies[60],italsoshowsthatin-contextlearningimplicitlyperformsmeta-optimizationthroughtheattentionmecha-nism.etersofLLMs,inwhichvariousparallelstrategiesareoftenjointlyutilized.Tosupportdistributedtraining,severalopti-ationframeworkshavebeenreleasedtofacilitatetheimplementationanddeploymentofparallelalgorithms,suchmizationtricksarealsoimportantfortrainingstabilityandmodelperformance,e.g.,restarttoovercometraininglossspike[56]andmixedprecisiontraining[68].Morerecently,GPT-4[46]proposestodevelopspecialinfrastructureandoptimizationmethodsthatreliablypredicttheperformanceoflargemodelswithmuchsmallermodels.●Abilityeliciting.Afterbeingpre-trainedonlarge-scalecorpora,LLMsareendowedwithpotentialabilitiesasgeneral-purposetasksolvers.While,theseabilitiesmightnotbeexplicitlyexhibitedwhenLLMsperformsomespe-ci?ctasks.Asthetechnicalapproach,itisusefultode-signsuitabletaskinstructionsorspeci?cin-contextlearn-ingstrategiestoelicitsuchabilities.Forinstance,chain-of-thoughtpromptinghasbeenshowntobeusefultosolvecomplexreasoningtasksbyincludingintermediateepsBesideswecanfurtherperforminstructiontuningonLLMswithtaskdescriptionsexpressedinnaturallanguage,forimprovingthegeneralizabilityofLLMsonhilethesetechniquesmainlycorrespondtotheemergentabilitiesofLLMs,whichmaynotshowtheanguagemodels●Alignmenttuning.SinceLLMsaretrainedtocapturethedatacharacteristicsofpre-trainingcorpora(includingbothhigh-qualityandlow-qualitydata),theyarelikelytogeneratetoxic,biased,orevenharmfulcontentforhumans.ItisnecessarytoalignLLMswithhumanvalues,e.g.,helpful,honest,andharmless.Forthispurpose,InstructGPT[61]de-signsaneffectivetuningapproachthatenablesLLMstofol-lowtheexpectedinstructions,whichutilizesthetechniquegwithhumanfeedbackRLHFItincorporateshumaninthetrainingloopwithelaboratelydesignedlabelingstrategies.ChatGPTisindeeddevelopedonasimilartechniquetoInstructGPT,whichshowsastrongalignmentcapacityinproducinghigh-quality,harmlessre-sponses,e.g.,rejectingtoanswerinsultingquestions.●Toolsmanipulation.Inessence,LLMsaretrainedastextgeneratorsovermassiveplaintextcorpora,thusperforminglesswellonthetasksthatarenotbestexpressedintheformoftext(e.g.,numericalcomputation).Besides,theircapacitiesarealsolimitedtothepre-trainingdata,e.g.,theinabilitytocaptureup-to-dateinformation.Totackletheseissues,arecentlyproposedtechniqueistoemployexternaltoolstocompensateforthede?cienciesofLLMs[70,71].Forexample,LLMscanutilizethecalculatorforaccuratecomputation[70]andemploysearchenginestoretrieveunknowninformation[71].Morerecently,ChatGPThasenabledthemechanismofusingexternalplugins(existingornewlycreatedapps)6,whicharebyanalogywiththe“eyesandears”ofLLMs.SuchamechanismcanbroadlyexpandthescopeofcapacitiesforLLMs.Besides,manyotherfactors(e.g.,theupgradeofhard-ware)alsocontributetothesuccessofLLMs.While,welimitourdiscussiontothemajortechnicalapproachesandkey?ndingsfordevelopingLLMs.6./blog/chatgpt-pluginshnicalEvolutionofGPTseriesModelsDuetotheexcellentcapacityincommunicatingwithhu-mans,ChatGPThasignitedtheexcitementoftheAIcom-powerfulGPTmodelwithspeciallyoptimizedconversationcapacities.Consideringtheever-growinginterestinChat-GPTandGPTmodels,weaddaspecialdiscussionaboutthetechnicalevolutionoftheGPT-seriesmodels,tobrie?ysummarizetheprogresshowtheyhavebeendevelopedinberoughlydividedintothefollowingstages7.EarlyExplorations.AccordingtooneinterviewwithIlyaSutskever8(aco-founderandchiefscientistofOpenAI),theideaofapproachingintelligentsystemswithlanguagemodelswasalreadyexploredintheearlydaysofOpe-nAI,whileitwasattemptedwithrecurrentneuralnet-works(RNN)[104].WiththeadventofTransformer,OpenAIdevelopedtwoinitialGPTmodels,namelyGPT-1[105]andGPT-2[26],whichcanbeconsideredasthefoundationtomorepowerfulmodelssubsequentlyi.e.,GPT-3andGPT-4.●GPT-1.In2017,theTransformermodel[22]wasintro-ducedbyGoogle,andtheOpenAIteamquicklyadaptedtheirlanguagemodelingworktothisnewneuralnetworkarchitecture.Theyreleasedthe?rstGPTmodelin2018,i.e.,GPT-1[105],andcoinedtheabbreviationtermGPTasthemodelname,standingforGenerativePre-trainedTransformer.GPT-1wasdevelopedbasedonagenerative,decoder-onlyTransformerarchitecture,andadoptedatask-agnosticlearningapproachthatcombinesunsupervisedpretrainingandsupervised?ne-tuning.GPT-1hassetupthecorear-chitecturefortheGPT-seriesmodelsandestablishedtheunderlyingprincipletomodelnaturallanguagetext,i.e.,predictingthenextword.●GPT-2.FollowingasimilararchitectureofGPT-1,GPT-2[26]increasedtheparameterscaleto1.5B,whichwastrainedwithalargewebpagedatasetWebText.AsclaimedinthepaperofGPT-2,itsoughttoperformtasksviaun-supervisedlanguagemodeling,withoutexplicit?ne-tuningusinglabeleddata.Tomotivatetheapproach,theyintro-ducedaprobabilisticformulationformulti-tasklearning,i.e.,p(output|input,task)(similarformshavebeentakeninanearlierwork[106]),whichpredictstheoutputcondi-tionedontheinputandtaskinformation.Tomodelthiscon-ditionalprobability,naturallanguagetextcanbeemployedasauni?edwaytoformatinput,outputandtaskinforma-tion.Inthisway,theprocessofsolvingataskcanbecastasawordpredictionproblemforgeneratingthesolutiontext.Further,theyintroducedamoreformalclaimforthisidea:“Sincethe(task-speci?c)supervisedobjectiveisthesameastheunsupervised(languagemodeling)objectivebutonlyevaluatedonasubsetofthesequence,theglobalminimumoftheunsupervisedobjectiveisalsotheglobalminimumofthesupervisedobjective(forvarioustasks)”[26]9.Abasic7.Notethatthediscussionofthispartcanbesomewhatsubjective.Theoverallviewpointsandsummariesaremadebasedontheund