人工神經(jīng)網(wǎng)絡(luò)-PPT課件

1、人工神經(jīng)網(wǎng)絡(luò)中國科學(xué)院自動(dòng)化研究所吳高巍2016-11-29聯(lián)結(jié)主義學(xué)派 又稱仿生學(xué)派或生理學(xué)派認(rèn)為人的思維基元是神經(jīng)元，而不是符號(hào)處理過程認(rèn)為人腦不同于電腦核心：智能的本質(zhì)是聯(lián)接機(jī)制。原理：神經(jīng)網(wǎng)絡(luò)及神經(jīng)網(wǎng)絡(luò)間的連接機(jī)制和學(xué)習(xí)算法麥卡洛可（McCulloch）皮茨（Pitts)什么是神經(jīng)網(wǎng)絡(luò)所謂的人工神經(jīng)網(wǎng)絡(luò)就是基于模仿生物大腦的結(jié)構(gòu)和功能而構(gòu)成的一種信息處理系統(tǒng)（計(jì)算機(jī)）。個(gè)體單元相互連接形成多種類型結(jié)構(gòu)的圖循環(huán)、非循環(huán)有向、無向自底向上（Bottom-Up）AI起源于生物神經(jīng)系統(tǒng)從結(jié)構(gòu)模擬到功能模擬仿生人工神經(jīng)網(wǎng)絡(luò)內(nèi)容生物學(xué)啟示多層神經(jīng)網(wǎng)絡(luò)Hopfield網(wǎng)絡(luò)自組織網(wǎng)絡(luò)生物學(xué)啟示 神經(jīng)元

2、組成：細(xì)胞體，軸突，樹突，突觸 神經(jīng)元之間通過突觸兩兩相連。信息的傳遞發(fā)生在突觸。 突觸記錄了神經(jīng)元間聯(lián)系的強(qiáng)弱。 只有達(dá)到一定的興奮程度，神經(jīng)元才向外界傳輸信息。 生物神經(jīng)元神經(jīng)元神經(jīng)元特性信息以預(yù)知的確定方向傳遞一個(gè)神經(jīng)元的樹突細(xì)胞體軸突突觸另一個(gè)神經(jīng)元樹突時(shí)空整合性對(duì)不同時(shí)間通過同一突觸傳入的信息具有時(shí)間整合功能對(duì)同一時(shí)間通過不同突觸傳入的信息具有空間整合功能神經(jīng)元工作狀態(tài)興奮狀態(tài)，對(duì)輸入信息整合后使細(xì)胞膜電位升高，當(dāng)高于動(dòng)作電位的閾值時(shí)，產(chǎn)生神經(jīng)沖動(dòng)，并由軸突輸出。抑制狀態(tài)，對(duì)輸入信息整合后使細(xì)胞膜電位降低，當(dāng)?shù)陀趧?dòng)作電位的閾值時(shí)，無神經(jīng)沖動(dòng)產(chǎn)生。結(jié)構(gòu)的可塑性神經(jīng)元之間的柔性連接：突觸

3、的信息傳遞特性是可變的學(xué)習(xí)記憶的基礎(chǔ)神經(jīng)元模型從生物學(xué)結(jié)構(gòu)到數(shù)學(xué)模型人工神經(jīng)元M-P模型x1x2xny12nInputOutputThresholdMcClloch and Pitts, A logical calculus of the ideas immanent in nervous activity, 1943f: 激活函數(shù)(Activation Function)g: 組合函數(shù)(Combination Function)Weighted Sum Radial Distance組合函數(shù) (e) (f)ThresholdLinearSaturating LinearLogistic Si

4、gmoidHyperbolic tangent SigmoidGaussian激活函數(shù)人工神經(jīng)網(wǎng)絡(luò)多個(gè)人工神經(jīng)元按照特定的網(wǎng)絡(luò)結(jié)構(gòu)聯(lián)接在一起，就構(gòu)成了一個(gè)人工神經(jīng)網(wǎng)絡(luò)。神經(jīng)網(wǎng)絡(luò)的目標(biāo)就是將輸入轉(zhuǎn)換成有意義的輸出。生物系統(tǒng)中的學(xué)習(xí)自適應(yīng)學(xué)習(xí)適應(yīng)的目標(biāo)是基于對(duì)環(huán)境信息的響應(yīng)獲得更好的狀態(tài)在神經(jīng)層面上，通過突觸強(qiáng)度的改變實(shí)現(xiàn)學(xué)習(xí)消除某些突觸，建立一些新的突觸生物系統(tǒng)中的學(xué)習(xí)Hebb學(xué)習(xí)律神經(jīng)元同時(shí)激活，突觸強(qiáng)度增加異步激活，突觸強(qiáng)度減弱學(xué)習(xí)律符合能量最小原則保持突觸強(qiáng)度需要能量，所以在需要的地方保持，在不需要的地方不保持。ANN的學(xué)習(xí)規(guī)則能量最小 ENERGY MINIMIZATION對(duì)人工神經(jīng)網(wǎng)絡(luò)

5、，需要確定合適的能量定義；可以使用數(shù)學(xué)上的優(yōu)化技術(shù)來發(fā)現(xiàn)如何改變神經(jīng)元間的聯(lián)接權(quán)重。ENERGY = measure of task performance error兩個(gè)主要問題結(jié)構(gòu) How to interconnect individual units?學(xué)習(xí)方法 How to automatically determine the connection weights or even structure of ANN?Solutions to these two problems leads to a concrete ANN!人工神經(jīng)網(wǎng)絡(luò)前饋結(jié)構(gòu)(Feedforward Archite

6、cture) - without loops - static 反饋/循環(huán)結(jié)構(gòu)(Feedback/Recurrent Architecture) - with loops - dynamic (non-linear dynamical systems)ANN結(jié)構(gòu)General structures of feedforward networksGeneral structures of feedback networks通過神經(jīng)網(wǎng)絡(luò)所在環(huán)境的模擬過程，調(diào)整網(wǎng)絡(luò)中的自由參數(shù) Learning by data學(xué)習(xí)模型 Incremental vs. Batch兩種類型 Supervised vs.

7、 UnsupervisedANN的學(xué)習(xí)方法若兩端的神經(jīng)元同時(shí)激活，增強(qiáng)聯(lián)接權(quán)重Unsupervised Learning學(xué)習(xí)策略: Hebbrian Learning 最小化實(shí)際輸出與期望輸出之間的誤差(Supervised) - Delta Rule (LMS Rule, Widrow-Hoff) - B-P LearningObjective:Solution:學(xué)習(xí)策略: Error Correction采用隨機(jī)模式，跳出局部極小 - 如果網(wǎng)絡(luò)性能提高，新參數(shù)被接受. - 否則，新參數(shù)依概率接受Local MinimumGlobal Minimum學(xué)習(xí)策略: Stochastic Lear

8、ning“勝者為王”(Winner-take-all )UnsupervisedHow to compete? - Hard competition Only one neuron is activated - Soft competition Neurons neighboring the true winner are activated. 學(xué)習(xí)策略: Competitive Learning重要的人工神經(jīng)網(wǎng)絡(luò)模型多層神經(jīng)網(wǎng)絡(luò)徑向基網(wǎng)絡(luò)Hopfield網(wǎng)絡(luò)Boltzmann機(jī)自組織網(wǎng)絡(luò)多層感知機(jī)(MLP)感知機(jī)實(shí)質(zhì)上是一種神經(jīng)元模型閾值激活函數(shù)Rosenblatt, 1957感知機(jī)判別規(guī)則

9、輸入空間中樣本是空間中的一個(gè)點(diǎn)權(quán)向量是一個(gè)超平面超平面一邊對(duì)應(yīng) Y=1另一邊對(duì)應(yīng) Y=-1單層感知機(jī)學(xué)習(xí)調(diào)整權(quán)值，減少訓(xùn)練集上的誤差簡單的權(quán)值更新規(guī)則:初始化對(duì)每一個(gè)訓(xùn)練樣本:Classify with current weightsIf correct, no change!If wrong: adjust the weight vector30學(xué)習(xí): Binary Perceptron初始化對(duì)每一個(gè)訓(xùn)練樣本:Classify with current weightsIf correct (i.e., y=y*), no change!If wrong: adjust the weight

10、vector by adding or subtracting the feature vector. Subtract if y* is -1.多類判別情況If we have multiple classes:A weight vector for each class:Score (activation) of a class y:Prediction highest score wins學(xué)習(xí): Multiclass Perceptron初始化依次處理每個(gè)樣本Predict with current weightsIf correct, no change!If wrong: lower

11、 score of wrong answer, raise score of right answer感知機(jī)特性可分性: true if some parameters get the training set perfectly correctCan represent AND, OR, NOT, etc., but not XOR收斂性: if the training is separable, perceptron will eventually converge (binary case)SeparableNon-Separable感知機(jī)存在的問題噪聲（不可分情況）: if the

12、data isnt separable, weights might thrash泛化性: finds a “barely” separating solution改進(jìn)感知機(jī)線性可分情況Which of these linear separators is optimal? Support Vector MachinesMaximizing the margin: good according to intuition, theory, practiceOnly support vectors matter; other training examples are ignorable Supp

13、ort vector machines (SVMs) find the separator with max marginSVM優(yōu)化學(xué)習(xí)問題描述訓(xùn)練數(shù)據(jù)目標(biāo)：發(fā)現(xiàn)最好的權(quán)值，使得對(duì)每一個(gè)樣本x的輸出都符合類別標(biāo)簽樣本xi的標(biāo)簽可等價(jià)于標(biāo)簽向量采用不同的激活函數(shù)平方損失：單層感知機(jī)單層感知機(jī)單層感知機(jī)單層感知機(jī)采用線性激活函數(shù)，權(quán)值向量具有解析解批處理模式一次性更新權(quán)重缺點(diǎn)：收斂慢增量模式逐樣本更新權(quán)值隨機(jī)近似，但速度快并能保證收斂多層感知機(jī) (MLP)層間神經(jīng)元全連接MLPs表達(dá)能力3 layers: All continuous functions 4 layers: all functio

14、nsHow to learn the weights?waiting B-P algorithm until 1986B-P Network結(jié)構(gòu) A kind of multi-layer perceptron, in which the Sigmoid activation function is used.B-P 算法學(xué)習(xí)方法 - Input data was put forward from input layer to hidden layer, then to out layer - Error information was propagated backward from out

15、 layer to hidder layer, then to input layerRumelhart & Meclelland, Nature,1986B-P 算法Global Error Measuredesired outputgenerated outputsquared errorThe objective is to minimize the squared error, i.e. reach the Minimum Squared Error (MSE)B-P 算法Step1. Select a pattern from the training set and present

16、 it to the network.Step2. Compute activation of input, hidden and output neurons in that sequence.Step3. Compute the error over the output neurons by comparing the generated outputs with the desired outputs.Step4. Use the calculated error to update all weights in the network, such that a global erro

17、r measure gets reduced. Step5. Repeat Step1 through Step4 until the global error falls below a predefined threshold.梯度下降方法Optimization method for finding out the weight vector leading to the MSE learning rategradientvector form:element form:權(quán)值更新規(guī)則For output layer:權(quán)值更新規(guī)則For output layer:權(quán)值更新規(guī)則For hid

18、den layer權(quán)值更新規(guī)則For hidden layer應(yīng)用: Handwritten digit recognition3-nearest-neighbor = 2.4% error40030010 unit MLP = 1.6% errorLeNet: 768 192 30 10 unit MLP = 0.9% errorCurrent best (SVMs) 0.4% errorMLPs：討論實(shí)際應(yīng)用中Preprocessing is importantNormalize each dimension of data to -1, 1Adapting the learning ra

19、tet = 1/tMLPs：討論優(yōu)點(diǎn)：很強(qiáng)的表達(dá)能力容易執(zhí)行缺點(diǎn)：收斂速度慢過擬合（Over-fitting）局部極小采用Newton法加正則化項(xiàng)，約束權(quán)值的平滑性采用更少（但足夠數(shù)量）的隱層神經(jīng)元嘗試不同的初始化增加擾動(dòng) Hopfield 網(wǎng)絡(luò)反饋 結(jié)構(gòu)可用加權(quán)無向圖表示Dynamic System兩種類型 Discrete (1982) and Continuous (science, 1984), by HopfieldHopfield網(wǎng)絡(luò)Combination function：Weighted SumActivation function：Threshold吸引子與穩(wěn)定性How do

20、 we “program” the solutions of the problem into stable states (attractors) of the network?How do we ensure that the feedback system designed is stable? Lyapunovs modern stability theory allows us to investigate the stability problem by making use of a continuous scalar function of the state vector,

21、called a Lyapunov (Energy) Function.Hopfield網(wǎng)絡(luò)的能量函數(shù)With inputWithout inputHopfield 模型Hopfield證明了異步Hopfield網(wǎng)絡(luò)是穩(wěn)定的，其中權(quán)值定義為 Whatever be the initial state of the network, the energy decreases continuously with time until the system settles down into any local minimum of the energy surface.Hopfield 網(wǎng)絡(luò): 聯(lián)

22、想記憶Hopfield網(wǎng)絡(luò)的一個(gè)主要應(yīng)用基于與數(shù)據(jù)部分相似的輸入，可以回想起數(shù)據(jù)本身（attractor state）也稱作內(nèi)容尋址記憶(content-addressable memory).Stored PatternMemory Association虞臺(tái)文, Feedback Networksand Associative MemoriesHopfield 網(wǎng)絡(luò): Associative MemoriesStored PatternMemory Association虞臺(tái)文, Feedback Networksand Associative MemoriesHopfield網(wǎng)絡(luò)的一個(gè)主

23、要應(yīng)用基于與數(shù)據(jù)部分相似的輸入，可以回想起數(shù)據(jù)本身（attractor state）也稱作內(nèi)容尋址記憶(content-addressable memory).How to store patterns?=?How to store patterns?=?: Dimension of the stored pattern權(quán)值確定: 外積(Outer Product)Vector form: Element form:Why? Satisfy the Hopfield modelAn example of Hopfield memory 虞臺(tái)文, Feedback Networks and As

24、sociative Memories123422123422111111111111StableE=4E=0E=4Recall the first pattern (x1)123422111111111111StableE=4E=0E=4Recall the second pattern (x2)Hopfield 網(wǎng)絡(luò): 組合優(yōu)化(Combinatorial Optimization)Hopfield網(wǎng)絡(luò)的另一個(gè)主要應(yīng)用將優(yōu)化目標(biāo)函數(shù)轉(zhuǎn)換成能量函數(shù)(energy function)網(wǎng)絡(luò)的穩(wěn)定狀態(tài)是優(yōu)化問題的解例: Solve Traveling Salesman Problem (TSP)Gi

25、ven n cities with distances dij, what is the shortest tour?Illustration of TSP Graph1234567891011Hopfield Network for TSP=?Hopfield Network for TSP=City matrix Constraint 1. Each row can have only one neuron “on”. 2. Each column can have only one neuron “on”. 3. For a n-city problem, n neurons will

26、be on.Hopfield Network for TSP124351234512345TimeCityThe salesman reaches city 5 at time 3.Weight determination for TSP: Design Energy FunctionConstraint-1Constraint-2Constraint-3能量函數(shù)轉(zhuǎn)換為2DHopfield網(wǎng)絡(luò)形式Network is built!Hopfield網(wǎng)絡(luò)迭代（TSP ） The initial state generated randomly goes to the stable state (s

27、olution) with minimum energyA 4-city example 阮曉剛， 神經(jīng)計(jì)算科學(xué)，2006自組織特征映射 (SOFM) What is SOFM?Neural Network with Unsupervised LearningDimensionality reduction concomitant with preservation of topological information. Three principals - Self-reinforcing - Competition - CooperationStructure of SOFM競爭(Competition)Finding the best matching weight vector for the present input.Criterion for determining the winning neuron: Maximum Inner Product Minimum Euclidean Distance合作(Cooperation