《深度學(xué)習(xí)入門-基于Python的實現(xiàn)》課件 4 遞歸神經(jīng)網(wǎng)絡(luò)

October15,2024PAGE27/44張敏October15,2024PAGE27/44張敏《深度學(xué)習(xí)入門—基于Python的實現(xiàn)》(吳喜之張敏)遞歸神經(jīng)網(wǎng)絡(luò)張敏October15,2024遞歸神經(jīng)網(wǎng)絡(luò)(RNN)簡介基本思路 HiddenStateOutput1HiddenStateOutput2HiddenStateHiddenStateOutput1HiddenStateOutput2HiddenStateRNNCellFinalOutputFinalFinalInputInput1RNNCellInput2RNNCellInput1RNNCellInput2RNNCell前向傳播的數(shù)學(xué)描述 考慮一個具有輸入層、隱藏層及輸出層的神經(jīng)網(wǎng)絡(luò).xt為t次輸入向量,Whxt的權(quán)重矩陣(這里包含了常數(shù)項),htt次隱藏層輸出向量,ytt次輸出向量Uhht?1(Elman?形式y(tǒng)t1(Jordan形式的權(quán)重矩陣σh?和σy分別代表隱藏層和輸出層的激活函數(shù).Elman網(wǎng)絡(luò):Jordan網(wǎng)絡(luò):

ht=σh(Whxt+Uhht?1);yt=σy(Wyht).ht=σh(Whxt+Uhyt?1);yt=σy(Wyht).RNN的反向傳播 RNN的反向傳播和其他神經(jīng)網(wǎng)絡(luò)沒有什么區(qū)別.對于每條訓(xùn)練數(shù)據(jù)相應(yīng)得到一個對真實數(shù)據(jù)的猜測,在和準(zhǔn)確答案比較后可計算出該過程的損失,然后可以計算損耗函數(shù)的梯度并據(jù)此相應(yīng)地更新模型中的權(quán)重,以便將來使用輸入數(shù)據(jù)進(jìn)行的計算產(chǎn)生更準(zhǔn)確的結(jié)果.盡管看起來每個RNN單元格似乎都使用了不同的權(quán)重,但實際上所有權(quán)重實際上都與該RNN單元格在整個過程中被重復(fù)使用的權(quán)重相同.因此,在每個時間步長中,只有輸入數(shù)據(jù)和隱藏狀態(tài)是獨特的.RNN語言處理的一個微型例子 (naturallanguageprocessing,NLP)任務(wù)中,神經(jīng)網(wǎng)絡(luò)的文本數(shù)據(jù)通常會轉(zhuǎn)換成其他形式,一般會將文本數(shù)據(jù)預(yù)處理為簡單的字符級的獨熱編碼或一位有效編碼(one-hotencoding).所謂的獨熱編碼和統(tǒng)計中把分類變量的各(水平)啞元化是一樣的,0/1組成的向量聯(lián)系起來.首先輸入必要的模塊:importtorchimporttorchfromtorchimportnnimportnumpyasnp原始文字?jǐn)?shù)據(jù)及編碼 曹操的《短歌行》的前八句為“對酒當(dāng)歌,人生幾何!譬如朝露,去日苦多.慨當(dāng)以慷,憂思難忘.何以解憂?唯有杜康”.下面把每個字和一個整數(shù)做一一對應(yīng),并形成兩個方向的字典:TextText對酒當(dāng)歌人生幾何譬如朝露去日苦多','慨當(dāng)以慷','憂思難忘何以解憂唯有杜康']Chars=set(''.join(Text))Int2char=dict(enumerate(Chars))Char2int={c:ifori,cinInt2char.items()}print(Chars,'\n',Char2int,'\n',Int2char)把句子補(bǔ)成相同長度 由于句子長短不一,用空格把句子補(bǔ)成相同長度:Maxlen=len(max(Text,Maxlen=len(max(Text,key=len))foriinrange(len(Text)):whilelen(Text[i])<Maxlen:Text[i]+=''foriinText:print(i,'length=',len(i))確定輸入數(shù)據(jù)(并啞元化)及目標(biāo)數(shù)據(jù) 由于我們的目的是給了前面一個字來預(yù)測后面的,因此輸入序列去掉每句的最后一個字,而目標(biāo)序列去掉第一個字:Input=[]TargetInput=[]Target=[]foriinrange(len(Text)):Input.append(Text[i][:-1])#輸入去掉最后一個字Target.append(Text[i][1:])#輸出去掉第一個字print("Input:{};Target:{}".format(Input[i],Target[i]))輸出為(注意已經(jīng)把3個字的詞填補(bǔ)成4個):Input:對酒當(dāng)Target:酒當(dāng)歌Input:人生幾Target:生幾何Input:譬如朝Target:如朝露Input:去日苦Target:日苦多Input:慨當(dāng)以Target:當(dāng)以慷Input:憂思難Target:思難忘Input:何以解Target:以解憂Input:唯有杜Target:有杜康文字字符在計算機(jī)中不好用,轉(zhuǎn)換成整數(shù)代碼:foriinrange(len(Text)):foriinrange(len(Text)):Input[i]=[Char2int[c]forcinInput[i]]Target[i]=[Char2int[c]forcinTarget[i]]print(Input,'\n',Target)成為整數(shù)序列:[[27,12,25],[8,15,23],[5,21,19],[3,10,2],[6,25,1],[[27,12,25],[8,15,23],[5,21,19],[3,10,2],[6,25,1],[13,14,7],[0,1,4],[18,17,26]][[12,25,24],[15,23,0],[21,19,11],[10,2,22],[25,1,9],[14,7,20],[1,4,13],[17,26,16]]其中的輸入必須啞元化:Dict_size=len(Char2int)Seq_len=Maxlen-1Batch_size=len(Text)defGetDummies(sequence,dict_size,seq_len,batch_size):features=np.zeros((batch_size,seq_len,dict_size),dtype=np.float32)foriinrange(batch_size):foruinrange(seq_len):features[i,u,sequence[i][u]]=1returnfeaturesInput=GetDummies(Input,Dict_size,Seq_len,Batch_size)Input=torch.from_numpy(Input)Target=torch.Tensor(Target)Input[:2],Target上面代碼中的函數(shù)torch.from_numpy保持原來的對象的類型和存儲位置,而另一個函數(shù)torch.Tensor把數(shù)據(jù)類型轉(zhuǎn)換成浮點型,不占有原來對象的位置.這兩個函數(shù)都產(chǎn)生torch.Tensor對象.(Input3行)(Target為:(tensor([[[0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,1.],[0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,1.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.],[0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,1.,0.,0.]],[[0.,0.,0.,0.,0.,0.,0.,0.,1.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.],[0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,1.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.],[0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,0.,1.,0.,0.,0.,0.]]]),tensor([[12.,25.,24.],[15.,23.,0.],[21.,19.,11.],[10.,2.,22.],[25.,1.,9.],[14.,7.,20.],[1.,4.,13.],[17.,26.,16.]]))× ×注意輸入數(shù)據(jù)的維數(shù)為8 3 28(代碼Input.size()).這實際上代表了8句話,每句話3個字(輸入數(shù)據(jù)Input),而這× ××8個3 28矩陣就是先前整數(shù)化矩陣所對應(yīng)的8個啞元化矩陣:×[[27,12,25],[8,15,23],[5,21,19],[3,10,2],[6,25,1],[[27,12,25],[8,15,23],[5,21,19],[3,10,2],[6,25,1],[13,14,7],[0,1,4],[18,17,26]]這意味著在Input矩陣的第0行中的第27,12,25列的值分1,18,15,231,2行中第5,21,19列的值為1,等等.定義一個隱藏層的RNN模型 classModel(nn.Module):definit(self,input_size,output_size,hidden_dim,super(Model,self).init()self.hidden_dim=hidden_dimself.n_layers=n_layersself.rnn=nn.RNN(input_size,hidden_dim,n_layers,batch_first=True)self.fc=nn.Linear(hidden_dim,output_size)definit_hidden(self,batch_size):hidden=torch.zeros(self.n_layers,batch_size,self.hidden_dim).to(device)returnhiddendefforward(self,x):batch_size=x.size(0)hidden=self.init_hidden(batch_size)out,hidden=self.rnn(x,hidden)out=out.contiguous().view(-1,self.hidden_dim)out=self.fc(out)returnout,hiddendevice=torch.device("cuda"iftorch.cuda.is_available()else"cpu")net=Model(input_size=Dict_size,output_size=Dict_size,hidden_dim=12,n_layers=1)net=net.to(device)n_epochs=100lr=0.01criterion=nn.CrossEntropyLoss()optimizer=torch.optim.Adam(net.parameters(),lr=lr)訓(xùn)練及預(yù)測 Input=Input.to(device)forepochinrange(1,n_epochs+1):optimizer.zero_grad()#清理前一個紀(jì)元的梯度記錄output,hidden=net(Input)output=output.to(device)Target=Target.to(device)loss=criterion(output,Target.view(-1).long())loss.backward()#反向傳播計算梯度optimizer.step()#更新權(quán)重ifepoch%20==0:print('Epoch:{}/{}'.format(epoch,n_epochs),end='')print("Loss:{:.4f}".format(loss.item()))訓(xùn)練及預(yù)測 defpredict(model,character):character=np.array([[Char2int[c]forcincharacter]])character=GetDummies(character,Dict_size,character.shape[1],1)character=torch.from_numpy(character)character=character.to(device)out,hidden=model(character)prob=nn.functional.softmax(out[-1],dim=0).datachar_ind=torch.max(prob,dim=0)[1].item()returnInt2char[char_ind],hiddendefsample(model,out_len,start='對'):model.eval()#evalmodechars[chforchinstart]如果輸入字符串多于一個則成多元素listsize=out_lenlen(chars)foriiinrange(size):#對多個字循環(huán)每次chars增加一個預(yù)測值char,h=predict(model,chars)chars.append(char)return''.join(chars)最終,我們試著輸入一些字,看預(yù)測結(jié)果.z=['對z=['對','人','譬','去','何','唯']foriinz:print(sample(net,4,i))對酒當(dāng)歌人生幾何譬如朝露去日苦多何以解憂唯有杜康長短期記憶網(wǎng)絡(luò)(LSTM)LSTM原理 (longshorttermmemoryLSTM)是RNN,能夠?qū)W習(xí)長期依賴關(guān)系.長時間記住信息實際上是默認(rèn)行為,而不是要努力學(xué)習(xí)的東西.LSTM在處理各種各樣的問題上表現(xiàn)出色,已被廣泛使用.LSTMneural不同,LSTM(feedbackconnections).它不僅可以處理(如圖像),(如語音或視頻).例如,LSTM適用于未分段并連接的手寫識別、語音識別和網(wǎng)絡(luò)流量或入侵檢測系統(tǒng)中的異常檢測等任務(wù).在眾多應(yīng)用中,LSTMRNN、隱藏LSTM的公式和單元格示意圖 圖2:LSTMLSTM的機(jī)理 ?考慮從單元格舍棄的信息.S型激活函數(shù)層忘記門決定.ht1?? ?和xt之后,運算ft=σg(Wfxt+Ufht1)對每一個單元格狀態(tài)ct1輸出了一個在? ?0和1之間的數(shù)目,輸出1代表完全保留它,而輸出0則代表完全舍棄它.考慮從單元格狀態(tài)保留的信息并更新.這里有兩部分S型激itσg(WixtUiht?1),它確定哪些值需要更新,σh激活函數(shù)產(chǎn)生的候選向量t=σh(ct+Ucht?1),.我:t⊙t,并以此來更新舊狀態(tài)t?1,把舊狀態(tài)乘上t來忘掉決,然后加上it⊙t,:t=t⊙t?1+t⊙ct.需要確定輸出的信息.最后需要確定輸出什么信息這必須基于單元格的過濾后σh(ct),?11之間的值,otσg(WoxtUoht?1)相乘以輸出我們確定要輸出的:htotσh(ct).zt=σg(Wzxt+Ufzt=σg(Wzxt+Ufht1);October15,202428/44張敏LSTM的一些變化形式 LSTM.“窺視孔peehole),稱為窺視孔連接也就是各個門都可以查看單元格狀態(tài).下面是每個門都可查看的公式但實際上可能只有部分門有窺視孔.ft=σg(Wfcct?1+Wfxt+Ufht?1);it=σg(Wfict?1+Wixt+Uiht?1);ot=σg(Wfoct+Woxt+Uoht?1).耦合忘記門和輸入門.把忘記門和輸入門耦合可以使得兩者同時決定而不是分別決定要忘記什么以及應(yīng)該向其中添加什么新信息.我們只會在要輸入某些內(nèi)容時才忘記它,也只有在忘記較舊的內(nèi)容時才向狀態(tài)輸入新值:t=t⊙c?1+(1?t)⊙t.門控遞歸單元(gatedrecurrentunit，GRU)將忘記門和輸入門合“更新門(updategate).它還合并了單元狀態(tài)和隱藏狀態(tài)及其他一些更改.LSTM模型更簡單,并且越來越流行.October15,2024PAGE35/44張敏October15,2024PAGE35/44張敏LSTM的一個微型例子 考慮一個輸入維數(shù)為10、隱藏維數(shù)為20,一共2層的LSTM.5310(維數(shù)的數(shù)組2320(層數(shù)×序列長度×隱藏層維數(shù)),輸出為5×3×20(批次大小×序列長度×隱藏層維數(shù))的數(shù)組.rnn=nn.LSTM(10,20,2)#input_dim,hidden_dim,n_layerstorch.manual_seed(1010)Input=torch.randn(5,3,10)#batch_size,seq_len,input_dimh0=torch.randn(2,3,20)#n_layers,batch_size,hidden_dimc0=torch.randn(2,3,20)#n_layers,batch_size,hidden_dimoutput,(hn,cn)=rnn(Input,(h0,c0))print('output.size={},\nsizeof(hn,cn)=({},{})'.format(output.size(),hn.size(),cn.size()))LSTM模型的輸出有兩部分:output對于t,tLSTM最后一層的輸出特征的張量httuple((hn,cn)1個元素包tn(tseq_lenhn而第2tn(tseq_len的單元狀態(tài)的cn.它們的維度為:output.size=torch.Size([5,3,20]),sizeof(hn,cn)=(torch.Size([2,3,20]),torch.Size([2,3,20]×上面output的最后一個3 20子數(shù)組為output[-1],它等同于hn[-1],也可用等價的把高維數(shù)組降維的方式output.squeeze()[-1,:]來展示:×o=output.squeeze()[-1,o=output.squeeze()[-1,:]print(o.shape,'\n',o)torch.Size([3,20])tensor([[-1.0960e-01,4.5722e-02,1.0352e-01,-9.3727e-02,-1.4929e-01,1.4392e-01,3.4892e-02,-1.4209e-01,2.7806e-02,-9.8101e-03,-2.0786e-02,-1.3259e-01,1.1315e-01,-1.7163e-02,3.1604e-02,1.8512e-04,6.2479e-02,2.9081e-03,-5.4018e-02,7.5184e-02],[-1.3412e-01,1.2913e-02,1.7029e-01,-8.0695e-02,-7.2440e-02,9.7708e-02,-2.4272e-02,-8.1588e-02,6.6754e-02,3.6014e-02,1.0842e-02,-1.0665e-01,1.2948e-01,-4.2475e-02,1.5328e-02,7.3622e-03,7.5282e-02,-8.6665e-03,-5.2866e-02,8.9747e-02],[-8.0154e-02,4.6845e-02,1.0121e-01,-6.3563e-02,-1.0733e-01,1.4816e-01,5.4422e-02,-1.6079e-01,-3.9718e-02,2.6982e-02,-8.6131e-02,-1.9261e-01,1.0086e-01,-2.3770e-02,-1.3167e-02,-1.7888e-03,1.2150e-01