構建深度學習的理論基礎 -一種實證方法

HarvardJohnA.paulsonschoolofEngineeringandAppliedsciencesveexaminedadissertationentitled"BuildingtheTheoreticalFoundationsofDeepLearning:AnEmpiricalApproach"presentedby:YaminiBansalGraduateschoolofArtsandsciencesDIssERTATIoNACCEpTANCECERTIFICATEersignedappointedbythesignatureTypedname:professorB.Bara一signaturesignatureTypedname:professorD.CoxMarch022DeepLearning:AnEmpiricalApproachDroTHEscHooioFENciNEERiNcANDAppiiEDsciENcEsiNpARriAiFuiFiiiMENroFrHEREguiREMENrsDocroRoFpHiiosopHyiNrHEsuBJEcroFCoMpurERsciENcEiDcEMAssAcHusErrsAiiRicHrsREsERvED.ABsrRAcrContentsTiriEpAcEiCopyRicHriiABsrRAcriiiCoNrENrsivLisriNcoFFicuREsviiiAcKNowiEDcEMENrsxv1INrRoDucrioNring 1.2Deviationsfromclassicalstatistics:Thegeneralizationpuzzle 31.3ourapproach 61.4overviewofthethesis 8iblebigpicture IUnderstandingRepresentationsroANEMpiRicAiiNvEsricArioNoFDEEpREpREsENrArioNsr2.1Introduction 14summaryofResults 152.2Modelstitching 172.3stitchingvs.representationalsimilarity 192.4stitchingConnectivity 212.5AllRoadsLeadtoRome 222.6MoreisBetter 242.7ConclusionandFuturework 25vwirHsEiF.supERvisioN263.1Introduction 263.1.1Relatedwork 293.1.2organization 3o3.2Formalstatementofresults 3o3.2.1CompleⅩitymeasures 323.3proofofTheorem4 333.4Thethreegaps 353.5EmpiricalstudyoftheRRMbound 373.6positiverationalitygapleavesroomforimprovement 393.7Conclusionsandopenquestions 4opolatingClassifers4INwHArwAyDoiNrERpoiAriNcciAssiFiERscENERAiizE2434.1Introduction 434.1.1DistributionalGeneralization 454.1.2summaryofContributions 464.1.3Relatedworkandsignińcance 474.2preliminaries 494.2.1DistributionalCloseness 494.2.2FrameworkforIndistinguishability 5o4.3FeatureCalibration 514.3.1EⅩperiments 534.3.2Discussion 574.3.31.Nearest.NeighborsConnection 574.3.4pointwiseDensityEstimation 594.4Agreementproperty 59 4.4.2potentialMechanisms 614.5LimitationsandEnsembles 624.5.1Ensembles 624.6DistributionalGeneralization:BeyondInterpolatingMethods 634.7ConclusionandDiscussion 664.7.1openQuestions 67IIIUnderstandingsampleEfciency68DArAscAiiNciAwsiNNMT6o5.1Introduction 69 5.2Relatedworks 725.3DatascalingLaws 735.3.1BLEUscore 755.3.2out.of.DistributionGeneralization 755.4TheE#ectofArchitecture 765.5TheE#ectofNoise 775.5.1DataFiltering 785.5.2AddingNoise 795.6Conclusions 81 n A.1.2CIFAR.1o: 84A.1.3ImageNet: 85A.1.4stitcher 85A.2Additionalresults 86A.2.1Ablations 86A.2.2ComparisonwithCKA 86A.2.3Comparisonwithńne.tuning 88 B.2EⅩperimentaldetails 92 B.2.4AdditionalResults 95 B.3.1Largerobustnessgap 96B.3.2Largerationalitygap 98 B.4simplerobustnessbounds 98B.4.1Robustnessofleastsquaresclassińers 98B.4.2Robustnessofempiricalriskminimizer 99 AppENDixCINwHArwAyDoiNrERpoiAriNcciAssiFiERscENERAiizE2Io2CEperimentalDetails......................1o2C.1.1Datasets..................................1o2C.1.2Models..................................1o3C.2FeatureCalibration:AppendiⅩ...........................1o5C.2.1Aguidetoreadingtheplots........................1o5C.2.2EⅩperiment1...............................1o6C.2.3Constantpartition............................1o7C.2.4Classpartition..............................1o7C.2.5MultipleFeatures 1o8C.2.6Coarsepartition 1o9C.2.7pointwiseDensityEstimation 11oC.3Agreementproperty:AppendiⅩ 113C.3.1EⅩperimentalDetails 113C.3.2Additionalplots 113C.3.3AlternateMechanisms 113C.4NerpolatingClassińers:AppendiⅩ 116C.5Nearest.Neighborproofs 117C.5.1Agreementproperty 117AppENDixDDArAscAiiNcLAwsiNNMTI22D.1scalingLawFittingDetails 122D.1.1separateFitsandvariance 122D.1.2optimizingHyperparameters 123D.1.3variance.LimitedRegime 123D.2BLEUscoreBehavior 125D.3scalingLawsforDi#erentooDDatasets 126D.4DatascalingphaseTransition 126D.5scalingLawswithDi#erentArchitectures 127D.6ChangingLanguagepairs 127REFERENcEsI46y true 7 formoreinformation 28RobustnessRationalityandMemorizationforCIFAR.ro.Eachbluepointisa nessRationalityandMemorizationforImageNetEachpointrepresents Ⅹ 4.3FeatureCalibrationwithoriginalclassesonCIFAR.ro:wetrainawRN.28.1o 4.5FeatureCalibrationforDecisiontreesonUCI(molecularbiology).weaddlabelbyConjecture13 56ing 56curacy........................................6o 61 4.1oDistributionalGeneralizationforwideResNetonCIFAR.ro.weapplylabelnoise p respectively 73s ndCKA C.6MNIsTEnsemble 112 CLaplaceKernelonFashion.MNIsT 119 LsTMhybrids 128