GRID superscalar a programming paradigm for GRID ：GRID超標(biāo)量體系結(jié)構(gòu)為網(wǎng)格編程范式

1、GRID superscalar: a programming paradigm for GRID applicationsCEPBA-IBM Research InstituteRal Sirvent, Josep M. Prez, Rosa M. Badia, Jess Labarta* Ivan Rodero* Roman Roset, Rogeli Grima, Jose M. CelaOutlineObjectiveThe essenceApplications structure from developers point of viewAutomatic code generat

2、ionCurrent run-time featuresProgramming experiencesFuture workConclusionsObjectiveEase the programming of GRID applicationsBasic idea: Gridns seconds/minutes/hoursOutlineObjectiveThe essenceApplications structure from developers point of viewAutomatic code generationCurrent run-time featuresProgramm

3、ing experiencesFuture workConclusionsThe essenceAssembly language for the GRIDSimple sequential programming, well defined operations and operands C/C+, Perl, Automatic run time “parallelizationUse architectural concepts from microprocessor designInstruction window (DAG), Dependence analysis, schedul

4、ing, locality, renaming, forwarding, prediction, speculation,The essencefor (int i = 0; i MAXITER; i+) newBWd = GenerateRandom(); subst (referenceCFG, newBWd, newCFG); dimemas (newCFG, traceFile, DimemasOUT); post (newBWd, DimemasOUT, FinalOUT); if(i % 3 = 0) Display(FinalOUT);fd = GS_Open(FinalOUT,

5、 R);printf(Results file:n); present (fd);GS_Close(fd);The essenceSubstDIMEMASEXTRACTSubstDIMEMASEXTRACTGS_openSubstDIMEMASEXTRACTSubstDIMEMASEXTRACTSubstDIMEMASEXTRACTSubstDIMEMASEXTRACTSubstDIMEMASEXTRACTDisplayDisplayCIRI GridThe essenceSubstDIMEMASEXTRACTSubstDIMEMASEXTRACTGS_openSubstDIMEMASEXTR

6、ACTSubstDIMEMASEXTRACTSubstDIMEMASEXTRACTSubstDIMEMASEXTRACTSubstDIMEMASEXTRACTDisplayDisplayCIRI GridOutlineObjectiveThe essenceApplications structure from developers point of viewAutomatic code generationCurrent run-time featuresProgramming experiencesFuture workConclusionsThree components:Interfa

7、ceMain programTasksApplications structure from developers point of viewinterface MC void subst(in File referenceCFG, in double seed, out File newCFG);void dimemas(in File newCFG, in File traceFile, out File DimemasOUT);void post(in File newCFG, in File DimemasOUT, inout File FinalOUT);void display(i

8、n File toplot);Interface: CORBA-IDL LikeApplications structure from developers point of viewMain program:for (int i = 0; i MAXITER; i+) newBWd = GenerateRandom(); subst (referenceCFG, newBWd, newCFG); dimemas (newCFG, traceFile, DimemasOUT); post (newBWd, DimemasOUT, FinalOUT); if(i % 3 = 0) Display

9、(FinalOUT);fd = GS_Open(FinalOUT, R);printf(Results file:n); present (fd);GS_Close(fd);Applications structure from developers point of viewApplications structure from developers point of viewvoid display(in File toplot) char command500; sprintf(command, ./display.sh %s, toplot); GS_System(command);v

10、oid dimemas(in File newCFG, in File traceFile, out File DimemasOUT) char command200; putenv(DIMEMAS_HOME=/usr/local/cepba-tools); sprintf(command, /usr/local/cepba-tools/bin/Dimemas -o %s %s, DimemasOUT, newCFG ); GS_System(command); TasksOutlineObjectiveThe essenceApplications structure from develo

11、pers point of viewAutomatic code generationCurrent run-time featuresProgramming experiencesFuture workConclusionsAutomatic code generationapp.idlapp-worker.capp.capp-functions.cservergsstubgenapp.hclientapp-stubs.cAutomatic code generationapp.idlgsstubgenapp.iapp-worker.plswigapp_wrapper.capp.pmapp-

12、stubs.capp.soclientapp.plC compilerapp-functions.pmserverAutomatic code generation#include int gs_result; void Filter(file referenceCFG, double latency, double bandwidth, file newCFG) char buff_latencyGS_GENLENGTH; char buff_bandwidthGS_GENLENGTH; /* Parameter marshalling */ sprintf(buff_latency, %.

13、20g, latency); sprintf(buff_bandwidth, %.20g, bandwidth); Execute(FilterOp, 1, 2, 1, 0, referenceCFG, buff_latency, buff_bandwidth, newCFG); Automatic code generation#include int main(int argc, char *argv) enum operationCode opCod = (enum operationCode)atoi(argv2); IniWorker(argc, argv); switch(opCo

14、d) case FilterOp: double latency; double bandwidth; latency = strtod(argv4, NULL); bandwidth = strtod(argv5, NULL); Filter(argv3, latency, bandwidth, argv6); break; EndWorker(gs_result, argc, argv); return 0; Automatic code generationclientGRID superscalarruntimeserveriapp-functions.capp-worker.capp

15、-stubs.capp.cGT2.serveriapp-functions.capp-worker.cGT2 services: gsiftp, gramOutlineObjectiveThe essenceApplications structure from developers point of viewAutomatic code generationCurrent run-time featuresProgramming experiencesFuture workConclusionsCurrent run-time featuresData dependence analysis

16、 Renaming File forwardingShared disks management and file transfer policy Resource brokering Task scheduling Task Submission End of task notification Results collection Explicit task synchronization File management primitives Current run-time featuresData dependence analysisDetects RaW, WaR, WaW dep

17、endencies based on file parametersOriented to simulations, FET solvers, bioinformatic applicationsMain parameters are data filesTasks Directed Acyclic Graph is built based on these dependenciesSubstDIMEMASEXTRACTSubstDIMEMASEXTRACTSubstSubstDIMEMASEXTRACTDisplay“f1_2“f1_1Current run-time featuresFil

18、e renamingWaW and WaR dependencies are avoidable with renamingT1_1T2_1T3_1T1_2T2_2T3_2T1_NT1_NT1_N“f1“f1“f1While (!end_condition() T1 (, “f1); T2 (“f1, , ); T3 (,);WaRWaWCurrent run-time featuresT1T2f1T1T2f1 (by socket)File forwarding reduces the impact of RaW data dependenciesCurrent run-time featu

19、resShared disks management and file transfer policy clientserver1server2T1f1f4T6f4f7f1f7Working directoriesCurrent run-time featuresShared disks management and file transfer policy clientserver1server2f1f4f7f1f7T1T6Working directoriesCurrent run-time featuresShared disks management and file transfer

20、 policy clientserver1server2Input directoriesCurrent run-time featuresResource brokeringNot a project goal External broker preferredInterface between run-time and brokerMDS is not currently used:Machine LimitOfJobs SubmitQueue ExecutionPath 2 none /home/sc03/NAS/worker/ 4 short /user1/uni/upc/ac/ros

21、ab/NAS_dis/worker/ 1 none /scratch/rosab/NAS_dis/worker/ /home/sc03/NAS/client/Current run-time featuresLinkBW OpId0SimulatedTime OpId1SimulatedTime . (and MasterLinkBW)1310720 0.25 15.55 1.3 0. 1 0.2 0.3 0.2 0.21310720 0.5 5.0 0.3 0.01 0.1 0.1 0.1 0.11310720 0.15 7.5 0.5 0.07 0.15 0.2 0.15 0.151310

22、720Broker receives requests from the run-timeResources selected taking into account:FT = File transfer time to resource rET = Execution time of task t on resource rCurrent run-time featuresTask schedulingDistributed between the Execute call, the callback function and the GS_Barrier callPossibilities

23、 The task can be submitted immediately after being createdTask waiting for resourceTask waiting for data dependencyGS_Barrier primitive before ending the program that waits for all tasksCurrent run-time featuresTask submissionTask submitted for execution as soon as the data dependencies are solved i

24、f resources are availableComposed of File transferTask submissionAll specified in RSL Temporal directory created in the server working directory for each taskCalls to globus:globus_gram_client_job_request globus_gram_client_callback_allowglobus_poll_blockingCurrent run-time featuresEnd of task notif

25、icationAsynchronous state-change callbacks monitoring system globus_gram_client_callback_allow()callback_func functionData structures update in Execute function, GRID superscalar primitives and GS_BarrierCurrent run-time featuresResults collectionCollection of output parameters which are not filesPa

26、rtial barrier synchronizationSocket and file mechanisms providedCurrent run-time featuresImplicit task synchronization GS_BarrierInserted in the user main program when requiredMain program execution is blockedglobus_poll_blocking() called Once all tasks are finished the program may resumeCurrent run

27、-time featuresFile management primitivesGS_FOpen GS_FCloseGS_OpenGS_CloseMandatory for file management operations in main programOpening a file with write optionData dependence analysis Renaming is appliedOpening a file with read optionPartial barrier until the task that is generating that file as o

28、utput file finishesInternally file management functions are handled as local tasksTask node inserted Data-dependence analysis Function locally executedCurrent run-time featuresPrevious prototype over Condor and MWCurrent prototype over Globus 2.x, using the APIFile transfer, security, provided by Gl

29、obusRun-time implemented primitives GS_on, GS_offExecuteGS_Open, GS_Close, GS_FClose, GS_FOpenGS_BarrierWorker side: GS_SystemOutlineObjectiveThe essenceApplications structure from developers point of viewAutomatic code generationCurrent run-time featuresProgramming experiencesFuture workConclusions

30、Programming experiencesPerformance modelling (Dimemas, Paramedir)Algorithm flexibilityNAS Grid BenchmarksImproved component programs modularityReduced Grid level source code linesBioinformatics application (production)Improved portability (Globus vs just Loadleveler)Reduced Grid level source code li

31、nesPblade solution for bioinformaticsProgramming experiencesPerformance modellingProgramming experiencesNAS Grid BenchmarksLaunchReportBTSPLUBTSPLUBTSPLUMFMFMFMFMFMFMFMFLaunchLaunchReportReportBTBTSPSPLULUBTBTSPSPLULUBTBTSPSPLULUMFMFMFMFMFMFMFMFProgramming experiencesNAS Grid BenchmarksAll of them implemented with GRID superscalarRun with classes S, W, AResults scale as expectedWhen several servers are used, ASCII mode requiredProgramming experiencesPerformance analysisGRID superscalar run-time instrumentedParaver tr