高級人工智能-第十章-強(qiáng)化學(xué)習(xí)

高級人工智能第十章史忠植中國科學(xué)院計(jì)算技術(shù)研究所強(qiáng)化學(xué)習(xí)2024/2/181內(nèi)容提要引言強(qiáng)化學(xué)習(xí)模型動態(tài)規(guī)劃蒙特卡羅方法時(shí)序差分學(xué)習(xí)Q學(xué)習(xí)強(qiáng)化學(xué)習(xí)中的函數(shù)估計(jì)應(yīng)用2024/2/182引言人類通常從與外界環(huán)境的交互中學(xué)習(xí)。所謂強(qiáng)化〔reinforcement〕學(xué)習(xí)是指從環(huán)境狀態(tài)到行為映射的學(xué)習(xí)，以使系統(tǒng)行為從環(huán)境中獲得的累積獎勵值最大。在強(qiáng)化學(xué)習(xí)中，我們設(shè)計(jì)算法來把外界環(huán)境轉(zhuǎn)化為最大化獎勵量的方式的動作。我們并沒有直接告訴主體要做什么或者要采取哪個(gè)動作,而是主體通過看哪個(gè)動作得到了最多的獎勵來自己發(fā)現(xiàn)。主體的動作的影響不只是立即得到的獎勵，而且還影響接下來的動作和最終的獎勵。試錯搜索(trial-and-errorsearch)和延期強(qiáng)化(delayedreinforcement)這兩個(gè)特性是強(qiáng)化學(xué)習(xí)中兩個(gè)最重要的特性。2024/2/183引言強(qiáng)化學(xué)習(xí)技術(shù)是從控制理論、統(tǒng)計(jì)學(xué)、心理學(xué)等相關(guān)學(xué)科開展而來，最早可以追溯到巴甫洛夫的條件反射實(shí)驗(yàn)。但直到上世紀(jì)八十年代末、九十年代初強(qiáng)化學(xué)習(xí)技術(shù)才在人工智能、機(jī)器學(xué)習(xí)和自動控制等領(lǐng)域中得到廣泛研究和應(yīng)用，并被認(rèn)為是設(shè)計(jì)智能系統(tǒng)的核心技術(shù)之一。特別是隨著強(qiáng)化學(xué)習(xí)的數(shù)學(xué)根底研究取得突破性進(jìn)展后，對強(qiáng)化學(xué)習(xí)的研究和應(yīng)用日益開展起來，成為目前機(jī)器學(xué)習(xí)領(lǐng)域的研究熱點(diǎn)之一。2024/2/184引言強(qiáng)化思想最先來源于心理學(xué)的研究。1911年Thorndike提出了效果律〔LawofEffect〕：一定情景下讓動物感到舒服的行為，就會與此情景增強(qiáng)聯(lián)系〔強(qiáng)化〕，當(dāng)此情景再現(xiàn)時(shí)，動物的這種行為也更易再現(xiàn)；相反，讓動物感覺不舒服的行為，會減弱與情景的聯(lián)系，此情景再現(xiàn)時(shí)，此行為將很難再現(xiàn)。換個(gè)說法，哪種行為會“記住”，會與刺激建立聯(lián)系，取決于行為產(chǎn)生的效果。動物的試錯學(xué)習(xí),包含兩個(gè)含義：選擇〔selectional〕和聯(lián)系〔associative〕，對應(yīng)計(jì)算上的搜索和記憶。所以，1954年，Minsky在他的博士論文中實(shí)現(xiàn)了計(jì)算上的試錯學(xué)習(xí)。同年，F(xiàn)arley和Clark也在計(jì)算上對它進(jìn)行了研究。強(qiáng)化學(xué)習(xí)一詞最早出現(xiàn)于科技文獻(xiàn)是1961年Minsky的論文“StepsTowardArtificialIntelligence”，此后開始廣泛使用。1969年，Minsky因在人工智能方面的奉獻(xiàn)而獲得計(jì)算機(jī)圖靈獎。2024/2/185引言1953到1957年，Bellman提出了求解最優(yōu)控制問題的一個(gè)有效方法：動態(tài)規(guī)劃〔dynamicprogramming〕Bellman于1957年還提出了最優(yōu)控制問題的隨機(jī)離散版本，就是著名的馬爾可夫決策過程〔MDP,Markovdecisionprocesse〕，1960年Howard提出馬爾可夫決策過程的策略迭代方法，這些都成為現(xiàn)代強(qiáng)化學(xué)習(xí)的理論根底。1972年，Klopf把試錯學(xué)習(xí)和時(shí)序差分結(jié)合在一起。1978年開始，Sutton、Barto、Moore，包括Klopf等對這兩者結(jié)合開始進(jìn)行深入研究。1989年Watkins提出了Q-學(xué)習(xí)[Watkins1989]，也把強(qiáng)化學(xué)習(xí)的三條主線扭在了一起。1992年，Tesauro用強(qiáng)化學(xué)習(xí)成功了應(yīng)用到西洋雙陸棋〔backgammon〕中，稱為TD-Gammon。2024/2/186內(nèi)容提要引言強(qiáng)化學(xué)習(xí)模型動態(tài)規(guī)劃蒙特卡羅方法時(shí)序差分學(xué)習(xí)Q學(xué)習(xí)強(qiáng)化學(xué)習(xí)中的函數(shù)估計(jì)應(yīng)用2024/2/187主體強(qiáng)化學(xué)習(xí)模型i:inputr:rewards:statea:action狀態(tài)sisi+1ri+1獎勵ri環(huán)境動作

aia0a1a2s0s1s2s32024/2/188描述一個(gè)環(huán)境〔問題〕Accessiblevs.inaccessibleDeterministicvs.non-deterministicEpisodicvs.non-episodicStaticvs.dynamicDiscretevs.continuousThemostcomplexgeneralclassofenvironmentsareinaccessible,non-deterministic,non-episodic,dynamic,andcontinuous.2024/2/189強(qiáng)化學(xué)習(xí)問題Agent-environmentinteractionStates,Actions,RewardsTodefineafiniteMDPstateandactionsets:SandAone-step“dynamics”definedbytransitionprobabilities(MarkovProperty):rewardprobabilities:EnvironmentactionstaterewardRLAgent2024/2/1810與監(jiān)督學(xué)習(xí)比照ReinforcementLearning–Learnfrominteractionlearnfromitsownexperience,andtheobjectiveistogetasmuchrewardaspossible.Thelearnerisnottoldwhichactionstotake,butinsteadmustdiscoverwhichactionsyieldthemostrewardbytryingthem.RLSystemInputsOutputs(“actions”)TrainingInfo=evaluations(“rewards”/“penalties”)SupervisedLearning–Learnfromexamplesprovidedbyaknowledgableexternalsupervisor.2024/2/1811強(qiáng)化學(xué)習(xí)要素Policy:stochasticruleforselectingactionsReturn/Reward:thefunctionoffuturerewardsagenttriestomaximizeValue:whatisgoodbecauseitpredictsrewardModel:whatfollowswhatPolicyRewardValueModelofenvironmentIsunknownIsmygoalIsIcangetIsmymethod2024/2/1812在策略Π下的Bellman公式Thebasicidea:So:

Or,withouttheexpectationoperator:isthediscountrate2024/2/1813Bellman最優(yōu)策略公式2024/2/1814MARKOVDECISIONPROCESS

k-armedbanditgivesimmediaterewardDELAYEDREWARD?CharacteristicsofMDP:asetofstates:Sasetofactions:Aarewardfunction:R:SxA

RAstatetransitionfunction:T:SxA

∏(S)

T(s,a,s’):probabilityoftransitionfromstos’usingactiona2024/2/1815MDPEXAMPLE:TransitionfunctionStatesandrewardsBellman

Equation:(Greedypolicyselection)2024/2/1816MDPGraphicalRepresentationβ,α:T(s,action,s’)SimilaritytoHiddenMarkovModels(HMMs)2024/2/1817動態(tài)規(guī)劃

DynamicProgramming-ProblemAdiscrete-timedynamicsystemStates{1,…,n}+terminationstate0ControlU(i)TransitionProbabilitypij(u)AccumulativecoststructurePolicies2024/2/1818FiniteHorizonProblemInfiniteHorizonProblemValueIteration動態(tài)規(guī)劃

DynamicProgramming–IterativeSolution

2024/2/1819動態(tài)規(guī)劃中的策略迭代/值迭代policyevaluationpolicyimprovement“greedification”PolicyIterationValueIteration2024/2/1820動態(tài)規(guī)劃方法TTTTTTTTTTTTT2024/2/1821自適應(yīng)動態(tài)規(guī)劃(ADP)Idea:usetheconstraints(statetransitionprobabilities)betweenstatestospeedlearning.Solve

=valuedetermination.Nomaximizationoveractionsbecauseagentispassiveunlikeinvalueiteration.usingDPLargestatespacee.g.Backgammon:1050equationsin1050variables2024/2/1822ValueIterationAlgorithmANALTERNATIVEITERATION:(Singh,1993)(Importantformodelfreelearning)StopIterationwhenV(s)differslessthan?.Policydifferenceratio=<2?γ/(1-γ)

(Williams&Baird1993b)2024/2/1823PolicyIterationAlgorithm

Policiesconvergefasterthanvalues.Whyfasterconvergence?

2024/2/1824ReinforcementLearning…DeterministictransitionsStochastictransitionsistheprobabilitytoreachingstatejwhentakingactionainstateistart3211234+1-1Asimpleenvironmentthatpresentstheagentwithasequentialdecisionproblem:Movecost=0.04(Temporal)creditassignmentproblemsparsereinforcementproblemOfflinealg:actionsequencesdeterminedexanteOnlinealg:actionsequencesisconditionalonobservationsalongtheway;Importantinstochasticenvironment(e.g.jetflying)2024/2/1825ReinforcementLearning…M=0.8indirectionyouwanttogo0.2inperpendicular0.1left0.1rightPolicy:mappingfromstatestoactions3211234+1-10.7053211234+1-1

0.8120.762

0.868

0.912

0.660

0.655

0.611

0.388Anoptimalpolicyforthestochasticenvironment:utilitiesofstates:EnvironmentObservable(accessible):perceptidentifiesthestatePartiallyobservableMarkovproperty:Transitionprobabilitiesdependonstateonly,notonthepathtothestate.Markovdecisionproblem(MDP).PartiallyobservableMDP(POMDP):perceptsdoesnothaveenoughinfotoidentifytransitionprobabilities.2024/2/1826ModelFreeMethodsModelsoftheenvironment:T:SxA

∏(S)

andR:SxARDoweknowthem?Dowehavetoknowthem?MonteCarloMethodsAdaptiveHeuristicCriticQLearning2024/2/1827MonteCarlo策略評價(jià)Goal:learnVp(s)

underPandRareunknowninadvanceGiven:

somenumberofepisodesunderpwhichcontainsIdea:AveragereturnsobservedaftervisitstosEvery-VisitMC:averagereturnsforeverytimesisvisitedinanepisodeFirst-visitMC:averagereturnsonlyforfirsttimesisvisitedinanepisodeBothconvergeasymptotically123452024/2/1828蒙特卡羅方法

MonteCarloMethodsIdea:HoldstatisticsaboutrewardsforeachstateTaketheaverageThisistheV(s)Basedonlyonexperience

Assumesepisodictasks(Experienceisdividedintoepisodesandallepisodeswillterminateregardlessoftheactionsselected.)Incrementalinepisode-by-episodesensenotstep-by-stepsense.2024/2/1829Problem:Unvisited<s,a>pairs(problemofmaintainingexploration)Forevery<s,a>makesurethat:P(<s,a>selectedasastartstateandaction)>0(Assumptionofexploringstarts)蒙特卡羅方法

2024/2/1830MonteCarlo方法TTTTTTTTTTTTTTTTTTTT2024/2/1831蒙特卡羅控制HowtoselectPolicies:(Similartopolicyevaluation)

MCpolicyiteration:PolicyevaluationusingMCmethodsfollowedbypolicyimprovement

Policyimprovementstep:greedifywithrespecttovalue(oraction-value)function2024/2/1832時(shí)序差分學(xué)習(xí)

Temporal-Differencetarget:theactualreturnaftertimettarget:anestimateofthereturn2024/2/1833時(shí)序差分學(xué)習(xí)

(TD)Idea:DoADPbackupsonapermovebasis,notforthewholestatespace.Theorem:AveragevalueofU(i)convergestothecorrectvalue.Theorem:Ifisappropriatelydecreasedasafunctionoftimesastateisvisited(=[N[i]]),thenU(i)itselfconvergestothecorrectvalue2024/2/1834時(shí)序差分學(xué)習(xí)

TDTTTTTTTTTTTTTTTTTTTT2024/2/1835TD(l)–AForwardViewTD(l)isamethodforaveragingalln-stepbackupsweightbyln-1(timesincevisitation)l-return:

Backupusingl-return:2024/2/1836時(shí)序差分學(xué)習(xí)算法

TD()

Idea:updatefromthewholeepoch,notjustonstatetransition.Specialcases: =1:Least-mean-square(LMS),MontCarlo =0:TDIntermediatechoiceof(between0and1)isbest.Interplaywith…2024/2/1837時(shí)序差分學(xué)習(xí)算法2024/2/1838時(shí)序差分學(xué)習(xí)算法收斂性TD(

)Theorem:Convergesw.p.1undercertainboundariesconditions.Decrease

i(t)s.t.Inpractice,oftenafixedisusedforalliandt.2024/2/1839時(shí)序差分學(xué)習(xí)

TD2024/2/1840Q-Learning

Watkins,1989EstimatetheQ-functionusingsomeapproximator(forexample,linearregressionorneuralnetworksordecisiontreesetc.).DerivetheestimatedpolicyasanargumentofthemaximumoftheestimatedQ-function.Allowdifferentparametervectorsatdifferenttimepoints.Letusillustratethealgorithmwithlinearregressionastheapproximator,andofcourse,squarederrorastheappropriatelossfunction.2024/2/1841Q-learningQ(a,i)Directapproach(ADP)wouldrequirelearningamodel.Q-learningdoesnot:Dothisupdateaftereachstatetransition:2024/2/1842ExplorationTradeoffbetweenexploitation(control)andexploration(identification)Extremes:greedyvs.randomacting (n-armedbanditmodels)Q-learningconvergestooptimalQ-valuesif*Everystateisvisitedinfinitelyoften(duetoexploration),*Theactionselectionbecomesgreedyastimeapproachesinfinity,and*Thelearningrateaisdecreasedfastenoughbutnottoofast (aswediscussedinTDlearning)2024/2/1843CommonexplorationmethodsInvalueiterationinanADPagent:OptimisticestimateofutilityU+(i)?-greedymethodNongreedyactionsGreedyactionBoltzmannexplorationExplorationfuncR+ifn<Nuo.w.2024/2/1844Q-LearningAlgorithmSetForTheestimatedpolicysatisfies2024/2/1845Whatistheintuition?BellmanequationgivesIfandthetrainingsetwereinfinite,thenQ-learningminimizeswhichisequivalenttominimizing2024/2/1846A-Learning

Murphy,2003andRobins,2004EstimatetheA-function(advantages)usingsomeapproximator,asinQ-learning.DerivetheestimatedpolicyasanargumentofthemaximumoftheestimatedA-function.Allowdifferentparametervectorsatdifferenttimepoints.Letusillustratethealgorithmwithlinearregressionastheapproximator,andofcourse,squarederrorastheappropriatelossfunction.2024/2/1847A-LearningAlgorithm

(InefficientVersion)ForTheestimatedpolicysatisfies2024/2/1848DifferencesbetweenQandA-learningQ-learningAttimetwemodelthemaineffectsofthehistory,(St,,At-1)andtheactionAtandtheirinteractionOurYt-1isaffectedbyhowwemodeledthemaineffectofthehistoryintimet,(St,,At-1)

A-learningAttimetweonlymodeltheeffectsofAtanditsinteractionwith(St,,At-1)OurYt-1doesnotdependonamodelofthemaineffectofthehistoryintimet,(St,,At-1)

2024/2/1849Q-LearningVs.A-LearningRelativemeritsanddemeritsarenotcompletelyknowntillnow.Q-learninghaslowvariancebuthighbias.A-learninghashighvariancebutlowbias.ComparisonofQ-learningwithA-learninginvolvesabias-variancetrade-off.2024/2/1850POMDP局部感知馬氏決策過程Ratherthanobservingthestateweobservesomefunctionofthestate.Ob–Observablefunction arandomvariableforeachstates.Problem:differentstatesmaylooksimilarTheoptimalstrategymightneedtoconsiderthehistory.2024/2/1851FrameworkofPOMDP

POMDP由六元組<S,A,R,P,Ω,О>定義。其中<S,A,P,R>定義了環(huán)境潛在的馬爾可夫決策模型上，Ω是觀察的集合，即系統(tǒng)可以感知的世界狀態(tài)集合，觀察函數(shù)О：S×A→PD〔Ω〕。系統(tǒng)在采取動作a轉(zhuǎn)移到狀態(tài)s′時(shí)，觀察函數(shù)О確定其在可能觀察上的概率分布。記為О〔s′,a,o〕。[1]

Ω可以是S的子集，也可以與S無關(guān)2024/2/1852POMDPsWhatifstateinformation(fromsensors)isnoisy?Mostlythecase!MDPtechniquesaresuboptimal!Twohallsarenotthesame.2024/2/1853POMDPs–ASolutionStrategySE:BeliefStateEstimator(CanbebasedonHMM)П:MDPTechniques2024/2/1854POMDP_信度狀態(tài)方法Idea:Givenahistoryofactionsandobservablevalue,wecomputeaposteriordistributionforthestatewearein(beliefstate)Thebelief-stateMDPStates:distributionoverS(statesofthePOMDP)Actions:asinPOMDPTransition:theposteriordistribution(giventheobservation)OpenProblem:Howtodealwiththecontinuousdistribution?2024/2/1855TheLearningProcessofBeliefMDP2024/2/1856MajorMethodstoSolvePOMDP算法名稱基本思想學(xué)習(xí)值函數(shù)Memorylesspolicies直接采用標(biāo)準(zhǔn)的強(qiáng)化學(xué)習(xí)算法Simplememorybasedapproaches使用k個(gè)歷史觀察表示當(dāng)前狀態(tài)UDM(UtileDistinctionMemory)分解狀態(tài)，構(gòu)建有限狀態(tài)機(jī)模型NSM(NearestSequenceMemory)存儲狀態(tài)歷史，進(jìn)行距離度量USM(UtileSuffixMemory)綜合UDM和NSM兩種方法Recurrent-Q使用循環(huán)神經(jīng)網(wǎng)絡(luò)進(jìn)行狀態(tài)預(yù)測策略搜索Evolutionaryalgorithms使用遺傳算法直接進(jìn)行策略搜索Gradientascentmethod使用梯度下降（上升）法搜索2024/2/1857強(qiáng)化學(xué)習(xí)中的函數(shù)估計(jì)RLFASubsetofstatesValueestimateastargetsV(s)GeneralizationofthevaluefunctiontotheentirestatespaceistheTDoperator.isthefunctionapproximationoperator.2024/2/1858并行兩個(gè)迭代過程值函數(shù)迭代過程值函數(shù)逼近過程HowtoconstructtheMfunction?Usingstatecluster,interpolation,decisiontreeorneuralnetwork?2024/2/1859FunctionApproximator:

V(s)=f(s,w)Update:Gradient-descentSarsa:

w

w+

a[rt+1+gQ(st+1,at+1)-Q(st,at)]

wf(st,at,w)weightvectorStandardgradienttargetvalueestimatedvalueOpenProblem:Howtodesignthenon-linerFAsystemwhichcanconvergewiththeincrementalinstances?2024/2/1860Semi-MDPDiscretetimeHomogeneousdiscountContinuoustimeDiscreteeventsInterval-dependentdiscountDiscretetimeDiscreteeventsInterval-dependentdiscountAdiscrete-timeSMDPoverlaidonanMDPCanbeanalyzedateitherlevel.OneapproachtoTemporalHierarchicalRL2024/2/1861Theequations2024/2/1862Multi-agentMDPDistributedRLMarkovGameBestResponseEnvironmentactionstaterewardRLAgentRLAgent2024/2/1863三種觀點(diǎn)問題空間主要方法算法準(zhǔn)則合作多agent強(qiáng)化學(xué)習(xí)分布、同構(gòu)、合作環(huán)境交換狀態(tài)提高學(xué)習(xí)收斂速度交換經(jīng)驗(yàn)交換策略交換建議基于平衡解多agent強(qiáng)化學(xué)習(xí)同構(gòu)或異構(gòu)、合作或競爭環(huán)境極小極大-Q理性和收斂性NASH-QCE-QWoLF最佳響應(yīng)多agent強(qiáng)化學(xué)習(xí)異構(gòu)、競爭環(huán)境PHC收斂性和不遺憾性IGAGIGAGIGA-WoLF2024/2/1864馬爾可夫?qū)Σ咴趎個(gè)agent的系統(tǒng)中，定義離散的狀態(tài)集S〔即對策集合G〕，agent動作集Ai的集合A,聯(lián)合獎賞函數(shù)Ri：S×A1×…×An→?和狀態(tài)轉(zhuǎn)移函數(shù)P：S×A1×…×An→PD〔S〕。2024/2/1865基于平衡解方法的強(qiáng)化學(xué)習(xí)OpenProblem:Nashequilibriumorotherequilibriumisenough?TheoptimalpolicyinsinglegameisNashequilibrium.2024/2/1866ApplicationsofRLChecker’s[Samuel59]TD-Gammon[Tesauro92]World’sbestdownpeakelevatordispatcher[Critesatal~95]Inventorymanagement[Bertsekasetal~95]10-15%betterthanindustrystandardDynamicchannelassignment[Singh&Bertsekas,Nie&Haykin~95]OutperformsbestheuristicsintheliteratureCart-pole[Michie&Chambers68-]withbang-bangcontrolRoboticmanipulation[Grupenetal.93-]PathplanningRobotdocking[Lin93]ParkingFootball[Stone98]TetrisMultiagentRL[Tan93,Sandholm&Crites95,Sen94-,Carmel&Markovitch95-,lotsofworksince]Combinatorialoptimization:maintenance&repairControlofreasoning[Zhang&DietterichIJCAI-95]2024/2/1867仿真機(jī)器人足球應(yīng)用Q學(xué)習(xí)算法進(jìn)行仿真機(jī)器人足球2對1訓(xùn)練，訓(xùn)練的目的是試圖使主體學(xué)習(xí)獲得到一種戰(zhàn)略上的意識，能夠在進(jìn)攻中進(jìn)行配合[宋志偉,2003]2024/2/1868仿真機(jī)器人足球前鋒A控球，并且在可射門的區(qū)域內(nèi)，但是A已經(jīng)沒有射門角度了；隊(duì)友B也處于射門區(qū)域，并且B具有良好的射門角度。A傳球給B，射門由B來完成，那么這次進(jìn)攻配合就會很成功。通過Q學(xué)習(xí)的方法來進(jìn)行2對1的射門訓(xùn)練，讓A掌握在這種狀態(tài)情況下傳球給B的動作是最優(yōu)的策略；主體通過大量的學(xué)習(xí)訓(xùn)練〔大數(shù)量級的狀態(tài)量和重復(fù)相同狀態(tài)〕來獲得策略，因此更具有適應(yīng)性。2024/2/1869仿真機(jī)器人足球狀態(tài)描述，將進(jìn)攻禁區(qū)劃分為個(gè)小區(qū)域，每個(gè)小區(qū)域是邊長為2m的正方形，一個(gè)二維數(shù)組()便可描述這個(gè)區(qū)域。使用三個(gè)Agent的位置來描述2對1進(jìn)攻時(shí)的環(huán)境狀態(tài)，利用圖10.11所示的劃分來泛化狀態(tài)?？烧J(rèn)為主體位于同一戰(zhàn)略區(qū)域?yàn)橄嗨茽顟B(tài)，這樣對狀態(tài)的描述雖然不精確，但設(shè)計(jì)所需的是一種戰(zhàn)略層次的描述，可認(rèn)為Agent在戰(zhàn)略區(qū)域內(nèi)是積極跑動的，這種方法滿足了需求。如此，便描述了一個(gè)特定的狀態(tài)；其中，是進(jìn)攻隊(duì)員A的區(qū)域編號，是進(jìn)攻隊(duì)員B的區(qū)域編號，是守門員的區(qū)域編號。區(qū)域編號計(jì)算公式為：。相應(yīng)的，所保存的狀態(tài)值為三個(gè)區(qū)域編號組成的對。前鋒A控球，并且在可射門的區(qū)域內(nèi)，但是A已經(jīng)沒有射門角度了；隊(duì)友B也處于射門區(qū)域，并且B具有良好的射門角度。A傳球給B，射門由B來完成，那么這次進(jìn)攻配合就會很成功。通過Q學(xué)習(xí)的方法來進(jìn)行2對1的射門訓(xùn)練，讓A掌握在這種狀態(tài)情況下傳球給B的動作是最優(yōu)的策略；主體通過大量的學(xué)習(xí)訓(xùn)練〔大數(shù)量級的狀態(tài)量和重復(fù)相同狀態(tài)〕來獲得策略，因此更具有適應(yīng)性。19圖10.11進(jìn)攻禁區(qū)內(nèi)的位置劃分072024/2/1870仿真機(jī)器人足球可選動作集確定為的策略通過基于概率的射門訓(xùn)練的學(xué)習(xí)來得到。的策略是，始終向受到威脅小，并且射門成功率高的區(qū)域帶球。為了實(shí)現(xiàn)這一策略目標(biāo)，可劃分進(jìn)攻區(qū)域?yàn)槎鄠€(gè)戰(zhàn)略區(qū)，在每個(gè)戰(zhàn)略區(qū)進(jìn)行射門評價(jià)，記