2024微軟人工智能系統(tǒng)深度神經(jīng)網(wǎng)絡(luò)計(jì)算框架基礎(chǔ)

1 lossdogloss

CatDogRaccoon????1

??error????2

????3

????4

??error????5

Error importnumpyasnpN,D=3,importnumpyasnpN,D=3,4x=np.random.randn(N,D)y=np.random.randn(N,D)z=np.random.randn(N,D)a=x*yb=a+zc=np.sum(b)

?? ?? ?? ????grad_c=grad_c=1.0grad_b=grad_c*np.ones((N,D))grad_a=grad_b.copy()grad_z=grad_b.copy()grad_x=grad_a*ygrad_y=grad_a*x

????????_??

????????_??

????????_??

Python-likeFlexibilityPython-like importxxlibimportxxlibx,y=load_data()y=xxlib.resnet152(x)libraryPython-likelibraryPython-like

Flexibility靈活 高效 Efficiency

library

Layer-based

Python-like

Flexibility ClassAttenionLayer<CPU>{voidforward(inputs..){}voidbackward(inputs,grad){}ClassAttenionLayer<CPU>{voidforward(inputs..){}voidbackward(inputs,grad){}ClassAttenionLayer<GPU>{…};REGISTER_LAYER(“Attention”,AttenionLayer); SGD:??←????????SGDwithmomentum:??←???(??????1+?????)?? ??\hhttps://ruder.io/optimizing-gradient-descent/ 前端編程語言和接口Python,Lua,R,C++自動(dòng)求導(dǎo)（AutoDifferentiation）統(tǒng)一模型表示：計(jì)算流圖前端編程語言和接口Python,Lua,R,C++自動(dòng)求導(dǎo)（AutoDifferentiation）統(tǒng)一模型表示：計(jì)算流圖xw*b+y圖的優(yōu)化與調(diào)度執(zhí)行Batching,Cache,Overlap內(nèi)核代碼優(yōu)化與編譯GPUkernel,autokernelgeneration 計(jì)算硬件計(jì)算硬件CPU,GPU,RDMAdevices AddLogWhileSubMatMulMergeMulConvBroadCastDivBatchNormReduceAddLogWhileSubMatMulMergeMulConvBroadCastDivBatchNormReduceReluLossMapTanhTransposeReshapeExpConcatenateSelectFloorSigmoid….. PAGE15PAGE15Numpyimportnumpyasnpnp.random.seed(importnumpyasnpnp.random.seed(0)N,D=3,4grad_c=1.0grad_b=grad_c*np.ones((N,D))grad_a=grad_b.copy()grad_z=grad_b.copy()grad_x=grad_a*ygrad_y=grad_a*x

?? ?? ?? ??x=np.random.randn(N,D)y=??x=np.random.randn(N,D)y=np.random.randn(N,D)z=np.random.randn(N,D)abc===x*ya+znp.sum(b)

????????_??

????????_??xyz??x??y*xyz??x??y*a*????z+bΣc+????a??bΣ??1717 L(??)=???????? ??(??,

),→????(??)????L?? =expexp ?? +exp?? 2 +sin(exp?? +exp?? 2)????(??)???? ?? ?? =exp exp?? +exp?? 2 +sin(exp?? +exp?? 2) xyz??x??y*a*xyz??x??y*a*????z+bΣc+????a??bΣ??圖的優(yōu)化與調(diào)度執(zhí)行Batching,Cache,Overlap內(nèi)核代碼優(yōu)化與編譯GPUkernel,autokernely+*bxw統(tǒng)一模型表示：計(jì)算流圖圖的優(yōu)化與調(diào)度執(zhí)行Batching,Cache,Overlap內(nèi)核代碼優(yōu)化與編譯GPUkernel,autokernely+*bxw統(tǒng)一模型表示：計(jì)算流圖前端編程語言和接口Python,Lua,R,C++自動(dòng)求導(dǎo)（AutoDifferentiation）計(jì)算硬件計(jì)算硬件CPU,GPU,RDMAdevicesxwxw*b+yxwxw*b+yPAGE29PAGE29Batchsame-typeoperatorsleverageGPUmassiveparallelism++×??+M??×????????+ +MMMMMRf Rzht-1xtData-flowgraphofaGRUcellWzWoWfhtBatchsame-typeoperatorsleverageGPUmassiveparallelism+×??+×??+M??×????????+ +MMMMMRf Rzht-1xtWzWoWfht+×??×??+M????????Mht-1RWxthtData-flowgraphofaGRUcellPAGEPAGE31xyz??x??y*a*????z+xyz??x??y*a*????z+bΣc+????a??bΣ??1xyzxyz??x*????y*a??z+bΣc+????bΣ????aGPU0顯式圖劃分GPU0??MatMul??Sigmoid ??????MatMul??Sigmoid ????MatMulGPU133DispatchpartitionsPartitiongraph??????*DispatchpartitionsPartitiongraph??????*????*???? ??tensortransmissionmechanism????Send*??*Recv??????????Server0ServerServer0Server136x y z

??x ??yCPUcodeGPUcode

* a+ +??b ??bΣ Σ??c

??a

??z計(jì)算硬件CPU,GPU,RDMAdevices前端編程語言和接口Python,Lua,R,C++自動(dòng)求導(dǎo)（AutoDifferentiation）統(tǒng)一模型表示：計(jì)算流圖* + yb圖的優(yōu)化與調(diào)度執(zhí)行Batching,Cache,Overlap內(nèi)核代碼優(yōu)化與編譯GPUkernel,autokernelgeneration計(jì)算硬件CPU,GPU,RDMAdevices前端編程語言和接口Python,Lua,R,C++自動(dòng)求導(dǎo)（AutoDifferentiation）統(tǒng)一模型表示：計(jì)算流圖* + yb圖的優(yōu)化與調(diào)度執(zhí)行Batching,Cache,Overlap內(nèi)核代碼優(yōu)化與編譯GPUkernel,autokernelgenerationLayer-basedStaticgraphLayer-basedStaticgraphPython-likePython,ScipyCannotleverageGPUNoprogrammingrestrictCNTK,Caffe2DeclarativeprogrammingGraphoptimizationCaffeProgramingwithconfigLargekernelgranularity

MoreFlexibilityimporttorchfromtorch.autogradimportVariableN,D=3,4x=Variable(torch.randn(N,D).cuda())y=Variable(torch.randn(N,D).cuda())z=Variable(torch.randn(N,D).cuda())foriinrange(10):importtorchfromtorch.autogradimportVariableN,D=3,4x=Variable(torch.randn(N,D).cuda())y=Variable(torch.randn(N,D).cuda())z=Variable(torch.randn(N,D).cuda())foriinrange(10):a=x*yb=a+zc=c+torch.sum(b)c.backward() 43Layer-basedStaticgraphLayer-basedStaticgraphDynamicgraphPython-likeDyNetImperativeprogramming(Define-by-run)NographoptimizationPython,ScipyCannotleverageGPUNoprogrammingrestrictCNTK,Caffe2DeclarativeprogrammingGraphoptimizationCaffeProgramingwithconfigLargekernelgranularity

MoreFlexibilityCompilerisusedtooptimizegeneralframeworktobemoreefficient,whilekeepingtheexistingflexibility!Compilerisusedtooptimizegeneralframeworktobemoreefficient,whilekeepingtheexistingflexibility!CustompurposemachinelearningalgorithmsTheanoCustompurposemachinelearningalgorithmsTheanoDisBeliefCaffeDeeplearningframeworksprovideeasierwaystoleveragevariouslibrariesMachineLearningLanguageandCompilerPowerfulCompilerInfrastructure:Codeoptimization,sparsityoptimization,hardwaretargeting