短發(fā)女生穿衣搭配技巧

1、1Tracking Metadata and Lineageof the Data Processing Chainfor Mapping Snow Cover Propertieswith the NASA MODISJames Frew1, Thomas H. Painter2,Peter Slaughter1, Jeff Dozier11Donald Bren School of Environmental Science and Management, University of California, Santa Barbara2National Snow and Ice Data

2、Center,University of Colorado, Boulder女裝批發(fā) http:/2OutlineoMotivation Snow mapping product Implications for hydrologic modelingoLineage Capture Wrapping: the ESSW experience Instrumenting,overriding,monitoring: the (ongoing) ES3 experience3MODIS image Sierra NevadaEOS Terra MODIS07 March 2004MOD09 Su

3、rface Reflectance0.555 0.645 0.85844Snow-covered area and grain size5Hindu Kush2003 DOY 0706Colorado RockiesCLPX13 March 20027Model structure: MODIS snow-area / albedo8Lineage Capture, Take 1The ESSW experience9Using Existing Science ApplicationsoNo “standard”Earth science computing environment comm

4、ercial packages (ArcInfo, MATLAB, ) public packages/models (MM5, MODTRAN, ) locally-developed codes arbitrary combinations of oExample: SST from AVHRR commercial, standalone programs parameters highly customized for UCSBoHow do we get these programs to communicate cooperate with ESSW, without rewrit

5、ing them?Navigate(Manual/Automatic)ReceiveIngest and CalibrateRectifySea Surface Temp (SST)SSTMaps10Lineage: Current Best Practice11Earth System Science Workbench (ESSW)Producer and consumer issues can both be addressedby a laboratory metaphoroExperiment Network of models ingesting / synthesizing da

6、ta generating productsoLaboratory Experiment execution environment Computing + storage = accessibility + scalabilityoLab Notebook Persistent storage that can be queried Keeps track of all experiments Documentation + lineage = accountability12Wrap Your App: Scripts Talk to ESSWoNo changes,just additi

7、ons Wrapper scripts Make program (groups) look like ESSW experiments use Perl API Lab Notebook daemon Accepts API commands Creates XML documentsuSends to database ESSW database XML metadata & DTDs Tabular metadatauXML search termsuLineage linksNavigate(Manual/Automatic)ReceiveIngest and CalibrateRec

8、tifySea Surface Temp (SST)SSTMapsESSWDatabasePerl APILab NotebookdaemonXML + SQLMySQLJDBCJavaPerl13ESSW Metadata managementoLab Notebook daemon verifies XML metadata documentoExperiment step metadata stored for product lineage trackingoComplete metadata document stored in custom database table XML D

9、TD 1:1 database table (n+1)th column is document itselfoSome metadata values extracted into database tables DTD contains column names and types for some elements Always save all the XML,even if dont know how to “columnize” all of it14# SST experiment wrapper # $L1B is the input Level 1B AVHRR image

10、file# $SST is the output SST image file # run legacy command nitpix: creates SST image from L1B image$base_temp=5.0;$temp_step=0.1;.system(nitpixbase_temp=$base_temptemp_step=$temp_step.$L1B$SST);# start recording ESSW metadatabeginXMLBld($ENVUSER,PRODUCTION);# get metadata for input file$L1B_ID=fin

11、dSciObjFromFile($L1B); AHVRR Level 1BproductMulti-channelsea surfacetemperaturealgorithmSea surfacetemperature(SST)avhrr_sstModelavhrr_l1bavhrr_sstWrapper Example: Input Dataset15#createmetadataforSSTimage$SST_ID=createMetadata(avhrr_sst);addValue($SST_ID,avhrr_sst.scene_id.satellite,$satellite);add

12、Value($SST_ID,avhrr_sst.scene_id.pass_date,$pass_date);.saveToDB($SST_ID,avhrr_sst);closeMetadata($SST_ID);saveDigest($SST,$SST_ID);AHVRR Level 1BproductMulti-channelsea surfacetemperaturealgorithmSea surfacetemperature(SST)avhrr_sstModelavhrr_l1bavhrr_sstWrapper Example: Output Dataset16#createmeta

13、dataforSSTexperiment$exp=createExperimentMetadata(avhrr_sstModel);$exp_step=createExpStepMetadata($exp,avhrr_sstExpStp);addValue($exp_step,avhrr_sstExpStp.base_temp,$base_temp);addValue($exp_step,avhrr_sstExpStp.temp_step,$temp_step);.saveToDB($exp_step,avhrr_sstExpStp);closeMetadata($exp_step);#con

14、nectinputandoutputimagestoexperimentregisterExperimentInputs($exp,$L1B_ID);registerExperimentOutputs($exp,$SST_ID);#finishrecordingESSWmetadataendXMLBld();AHVRR Level 1BproductMulti-channelsea surfacetemperaturealgorithmSea surfacetemperature(SST)avhrr_sstModelavhrr_l1bavhrr_sstWrapper Example: Proc

15、ess17# create metadata for SST experiment$exp=createExperimentMetadata(avhrr_sstModel);$exp_step=createExpStepMetadata($exp,avhrr_sstExpStp);addValue($exp_step,avhrr_sstExpStp.base_temp,$base_temp);addValue($exp_step,avhrr_sstExpStp.temp_step,$temp_step);.saveToDB($exp_step,avhrr_sstExpStp);closeMet

16、adata($exp_step);# connect input and output images to experimentregisterExperimentInputs($exp,$L1B_ID);registerExperimentOutputs($exp,$SST_ID);# finish recording ESSW metadataendXMLBld();AHVRR Level 1BproductMulti-channelsea surfacetemperaturealgorithmSea surfacetemperature(SST)avhrr_sstModelavhrr_l

17、1bavhrr_sstWrapper Example: Lineage Links18Process graph reconstructedfrom ESSW database19ESSW LessonsoProviders are customersESIPs arent much good unless scientists are happy to put information in themoA light touch is the right touchWrapping is easier for scientists and their programmers to deal w

18、ith than complete re-engineeringoScientists do write scripts, but not necessarily PerlScripting (gluing stuff together) comes naturally to scientistsoScientists dont write DTDsoNobody calls metadata APIsESSW was automatic, but not automatic enough20Lineage Capture, Take 2The ES3 experience21ES3 : Ea

19、rth System Science Server ESSW+ data lineage trackingBUB data storageROCKS processing clustersAlexandria Digital LibraryMicrosoft TerraServerMODsterOpenDAPMODISCoronaAVHRRWatershed-scale snow productGlobal-scale snow product22From ESSW to ES3: SummaryoPerl wrappers “Probulators”oPerl API web service

20、s + XML messagesoMySQL XML database(s)23From Wrappers to ProbulatorsWrappers: Active Lineageo+ Complete control over what gets recorded Single language/API for all wrapped events Not tied to execution You can even lie about what happenedo Must explicitly script everything Scripts can drift from real

21、ity You can even lie about what happened24From Wrappers to ProbulatorsProbulators: Passive Lineageo+ Record what actually happened Not just what you think happened Not what didnt happen Automatic: dont have to write new scripts for everythingo Different flavors for different environments Cant just d

22、o everything in Perl25Probulator patternsoInstrumentation Insert lineage capture instructions directly into science codes e.g. “I just created file foo” Typical implementation: preprocessor/precompileroOverriding Replace standard routines/libraries with lineage-capturing versions e.g. open() snoopy_

23、open() Typical implementation: modify execution environment environment variables configuration filesoPassive monitoring Trace program execution e.g. “called open() with args foo, bar, ” Typical implementation: straced shell26ES3 Lineage Architectureprobulator1probulatornloggertransmitterES3 corelog

24、files27Probulating IDL: Instrumenting the code;editpromodscag_cleanse,prefix=prefix,ns=ns,nl=nlHELP, NAMES=*, OUTPUT=ES3_ENVIROMENT & ES3_LOG, $ ENTER=modscag_cleanse, ENVIROMENT=ES3_ENVIROMENT;cleanupunder,overflowofMODSCAGrun;Input:prefix=prefixforalloftheMODSCAGoutput;ns=numberofsamples;nl=number

25、oflines;Output:rewriteoftheMODSCAGfiles;t.h.painter/1.19.2005;opensnowfileES3_openr,1,string(prefix,snow.pic)snow=fltarr(ns,nl)readu,1,snow blah blah blah HELP, NAMES=*, OUTPUT=ES3_ENVIROMENT & ES3_LOG, LEAVE=modscag_cleanse, $ ENVIROMENT=ES3_ENVIROMENTEND ; modscag_cleanse28Probulating IDL: Results

26、variablename=!PATHvalue=/home/haavar/probulator/idl:/home/rsi/idl_6.1/lib/hook:/home/rsi/home/haavar/painter/data/tillsnow.pic29Probulating bash: Passive Monitoring cat /etc/passwd | grep haavar | sed -n s/(.*:)2(0-9+).*/2/p252321138336174.480079open(/etc/ld.so.cache,O_RDONLY)=3252321138336174.48021

27、5open(/lib/libm.so.6,O_RDONLY)=3252341138336178.887267dup2(3,255)=255252341138336178.887912pipe(3,4)=0252341138336178.888257clone(child_stack=0,child_tidptr=0 xb7f2e708)=25235252351138336178.889366dup2(4,1)=1252351138336178.889975pipe(3,4)=0252351138336178.890326clone(child_stack=0,child_tidptr=0 xb

28、7f2e708)=25236252351138336178.891260pipe(4,5)=0252351138336178.891756clone(child_stack=0,child_tidptr=0 xb7f2e708)=25237252351138336178.892753clone(child_stack=0,child_tidptr=0 xb7f2e708)=25238252381138336178.894266dup2(4,0)=0252361138336178.894726dup2(4,1)=1252371138336178.894763dup2(3,0)=025237113

29、8336178.895581dup2(5,1)=1252381138336178.897006execve(/bin/sed,sed,-n,s/(.*:)2(0-9+).*/2/p,HOSTNAME=,TERM=xterm-color,252361138336178.900117execve(/bin/cat,cat,/etc/passwd”,252371138336178.903342execve(/bin/grep,grep,haavar,30Probulating bash: Results same as IDL /etc/passwd

30、/etc/ld.so.cache/etc/passwdhaavar31Now What?oProbulator reports not universally unique Q: How hook separate reports together? A: Logger assigns UUIDs to Data streams Processes Jobs (workflows)oLineage not explicit Q: How publish lineage? A: ES3 Core builds serialized graph32Thanks to:CurrentoMike Co

31、leeoStephane MaritorenaoDominic MetzgeroKarl RittgeroDave SiegelFormeroAnurag AcharyaoRajendra BoseoScott DenningoDebbie DonahueoJim DuffoCalin DumaoErik FieldsoJim GrayoSteve MileyoJordan MorrisoMark PelletieroPete PetersonoWalter RosenthaloKlaus SchauseroHvar Valeur33To Probulate Further : Publicatio