分散式咚愫喗doc

1、Grid的介紹 B88901129戴松青 B88901162朱如瓏分散式運算簡介配合投影片第1-3張(節(jié)錄自EBSCOhost網(wǎng)站)Driven by increasingly complex problems and propelled by increasingly powerful technology, today s science is as much based on computation, data analysis, and collaboration as on the efforts of individual experimentalists and theorist

2、s. But even as computer power, data storage, and communication continue to improve exponentially, computational resources are failing to keep up with what scientists demand of them.The concept of sharing distributed resources is not new. In 1965, MIT's Fernando Corbato and the other designers of

3、 the Multics operating system envisioned a computer facility operating "like a power company or water company." And in their 1968 article "The Computer as a Communications Device," J. C. R. Licklider and Robert W. Taylor anticipated Grid-like scenarios. Since the late 1960s, much

4、 work has been devoted to developing distributed systems, but with mixed success.Now, however, a combination of technology trends and research advances makes it feasible to realize the Grid vision-to put in place a new international scientific infrastructure with tools that, together, can meet the c

5、hallenging demands of 21st-century science. Indeed, major science communities now accept that Grid technology is important for their future. Numerous government-funded R&D projects are variously developing core technologies, deploying production Grids, and applying Grid technologies to challengi

6、ng applications.電腦設(shè)備發(fā)展趨勢配合投影片第4張(節(jié)錄自EBSCOhost網(wǎng)站)A useful metric for the rate of technological change is the average period during which speed or capacity doubles or, more or less equivalently, halves in price. For storage, networks, and computing power, these periods are around 12, 9, and 18 months,

7、 respectively. The different time constants associated with these three exponentials have significant implications.Such large data volumes demand more from our analysis capabilities. Dramatic improvements in microprocessor performance mean that the lowly desktop or laptop is now a powerful computati

8、onal engine. Nevertheless, computer power is falling behind storage. By doubling "only" every 18 months or so, computer power takes five years to increase by a single order of magnitude. Assembling the computational resources needed for large-scale analysis at a single location is becoming

9、 infeasible.The solution to these problems lies in dramatic changes taking place in networking. In 1985, NSFnet's backbone operated at a then-unprecedented 56 Kb/s. This year, the centers will be connected by the 40 Gb/s TeraGrid network()-an improvement of six orders of magnitude in 17 years.Th

10、e doubling of network performance relative to computer speed every 18 months has already changed how we think about and undertake collaboration. If, as expected, networks outpace computers at this rate, communication becomes essentially free. To exploit this bandwidth bounty, we must imagine new way

11、s of working that are communication intensive, such as pooling computational resources, streaming large amounts of data from databases or instruments to remote computers, linking sensors with each other and with computers and archives, and connecting people, computing, and storage in collaborative e

12、nvironments that avoid the need for costly travel.If communication is unlimited and free, then we are not restricted to using local resources to solve problems. When running a colleague's simulation code, I do not need to install the code locally. Instead, I can run it remotely on my colleague&#

13、39;s computer. When applying the code to datasets maintained at other locations, I do not need to get copies of those datasets myself (not so long ago, I would have requested tapes). Instead, I can have the remote code access those datasets directly. If I wish to repeat the analysis many hundreds of

14、 times on different datasets, I can call on the collective computing power of my research collaboration or buy the power from a provider. And when I obtain interesting results, my geographically dispersed colleagues and I can look at and discuss large output datasets by using sophisticated collabora

15、tion and visualization tools.SETIhome簡介配合投影片第6張(節(jié)錄自截至目前為止，SETI 專案計劃絕大部份，是以美國加州柏克萊大學(xué)，為接收無線電天文望遠(yuǎn)鏡之即時資料所建構(gòu)的大型計算機組為主體。雖然擁有這些大型計算機組，但其運算能力在面對這麼龐大且信號微弱的天文無線電資料時，其分析能力也顯得力不從心。建構(gòu)一臺具有分析微弱訊號及強大運算能力的計算機是絕對必要的，最好的解決方案就是購置一臺鉅量超級電腦，但對非商業(yè)性質(zhì)的 SETI 專案計劃而言，一直都沒有足夠的經(jīng)費可購買如此高價位的電腦設(shè)備。與其使用一臺超級電腦來執(zhí)行這項作業(yè)，不如以次級計算機並花費較長時間來運作；

16、但如此一來，其不斷湧入的待分析訊號會因無法即時處理而堆積如山。是否能使用一批小型計算機，讓訊號的不同部份，同時在這批小型計算機作分析運算？而 SETI 小組又要去那裏找出上千部電腦，用以分析由 Arecibo 望遠(yuǎn)鏡持續(xù)湧入的無線電串流訊號？ 柏克萊大學(xué) SETI 小組發(fā)現(xiàn)，其實在週遭早已經(jīng)有成千的電腦可供他們利用。這些電腦大部份的時間，都處於螢?zāi)槐Ｗo(hù)模式的閒置狀態(tài)，浪費不少寶貴的電腦資源；這也使SETIhome的構(gòu)想得以成型。SETIhome的構(gòu)想是希望能借用您電腦的閒置時間，幫他們發(fā)現(xiàn)新生命、新文明，而這項工作則是藉由螢?zāi)槐Ｗo(hù)程式，經(jīng)網(wǎng)際網(wǎng)路，從 SETI 伺服器下載一小段訊號資料，進(jìn)行分

17、析，完成後將報告回傳給 SETI。當(dāng)你需要使用電腦作業(yè)時，SETI 螢?zāi)槐Ｗo(hù)程式會立刻退出，完全不影響你的正常作業(yè)；當(dāng)電腦閒置時，SETI 螢?zāi)槐Ｗo(hù)程式會自行啟動並接續(xù)未完成的分析作業(yè)。 Grid的特徵配合投影片第8張(節(jié)錄自WHAT IS THE GRID? A THREE POINT CHECKLIST,written by Ian Foster)1.coordinates resources that are not subject to centralized control - (A Grid integrates and coordinates resources and user

18、s that live within different control domains for example, the users desktop vs. central computing; different administrative units of the same company; or different companies; and addresses the issues of security, policy, payment, membership, and so forth that arise in these settings. Otherwise, we a

19、re dealing with a local management system.) 2.using standard, open, general-purpose protocols and interfaces - (A Grid is built from multi-purpose protocols and interfaces that address such fundamental issues as authentication, authorization, resource discovery, and resource access. As I discuss fur

20、ther below, it is important that these protocols and interfaces be standard and open. Otherwise, we are dealing with an application-specific system.) 3.to deliver nontrivial qualities of service - (A Grid allows its constituent resources to be used in a coordinated fashion to deliver various qualiti

21、es of service, relating for example to response time, throughput, availability, and security, and/or co-allocation of multiple resource types to meet complex user demands, so that the utility of the combined system is significantly greater than that of the sum of its parts.)虛擬組織簡介配合投影片第10張(節(jié)錄自The An

22、atomy of grid ,by Ian Foster, Carl Kesselman ,Steven Tuecke)The real and specific problem that underlies the Grid concept is coordinated resource sharingand problem solving in dynamic, multi-institutional virtual organizations. The sharing that weare concerned with is not primarily file exchange but

23、 rather direct access to computers, software,data, and other resources, as is required by a range of collaborative problem-solving and resource-brokeringstrategies emerging in industry, science, and engineering. This sharing is, necessarily,highly controlled, with resource providers and consumers de

24、fining clearly and carefully just whatis shared, who is allowed to share, and the conditions under which sharing occurs. A set ofindividuals and/or institutions defined by such sharing rules form what we call a virtuaorganization (VO).The following are examples of VOs: the application service provid

25、ers, storage service providers,cycle providers, and consultants engaged by a car manufacturer to perform scenario evaluationduring planning for a new factory; members of an industrial consortium bidding on a newaircraft; a crisis management team and the databases and simulation systems that they use

26、 to plana response to an emergency situation; and members of a large, international, multiyear high-energy physics collaboration. Each of these examples represents an approach to computing andproblem solving based on collaboration in computation- and data-rich environments.Grid的架構(gòu)配合投影片第12、13張(節(jié)錄自EBS

27、COhost網(wǎng)站) As figure 2 shows schematically, protocols and APIs can be categorized according to the role they play in a Grid system. At the lowest level, the fabric, we have the physical devices or resources that Grid users want to share and access, including computers, storage systems, catalogs, netw

28、orks, and various forms of sensors.Above the fabric are the connectivity and resource layers. The protocols in these layers must be implemented everywhere and, therefore, must be relatively small in number. The connectivity layer contains the core communication and authentication protocols required

29、for Grid-specific network transactions. Communication protocols enable the exchange of data between resources, whereas authentication protocols build on communication services to provide cryptographically secure mechanisms for verifying the identity of users and resources.The resource layer contains

30、 protocols that exploit communication and authentication protocols to enable the secure initiation, monitoring, and control of resource-sharing operations. Running the same program on different computer systems depends on resource-layer protocols. The Globus Toolkit (which is described in box 2 abov

31、e) is a commonly used source of connectivity and resource protocols and APIs.The collective layer contains protocols, services, and APIs that implement interactions across collections of resources. Because they combine and exploit components from the relatively narrower resource and connectivity lay

32、ers, the components of the collective layer can implement a wide variety of tasks without requiring new resource-layer components. Examples of collective services include directory and brokering services for resource discovery and allocation; monitoring and diagnostic services; data replication serv

33、ices; and membership and policy services for keeping track of who in a community is allowed to access resources.At the top of any Grid system are the user applications, which are constructed in terms of, and call on, the components in any other layer.The Globus ToolkitThe Globus toolkit comprises a

34、set of modules. Each module defines an interface, which higher-level services use to invoke that module's mechanisms, and provides an implementation, which uses appropriate low-level operations to implement these mechanisms in different environments. Currently identified toolkit modules are as f

35、ollows. l Resource location and allocation. This component provides mechanisms for expressing application resource requirements, for identifying resources that meet these requirements, and for scheduling resources once they have been located. Resource location mechanisms are required because applica

36、tions cannot, in general, be expected to know the exact location of required resources, particularly when load and resource availability can vary. Resource allocation involves scheduling the resource and performing any initialization required for subsequent process creation, data access, etc. In som

37、e situationsfor example, on some supercomputerslocation and allocation must be performed in a single step.l Communications. This component provides basic communication mechanisms. These mechanisms must permit the efficient implementation of a wide range of communication methods, including message pa

38、ssing, remote procedure call, distributed shared memory, stream-based, and multicast. Mechanisms must be cognizant of network quality of service parameters such as jitter, reliability, latency, and bandwidth.l Unified resource information service. This component provides a uniform mechanism for obta

39、ining real-time information about metasystem structure and status. The mechanism must allow components to post as well as receive information. Support for scoping and access control is also required.l Authentication interface. This component provides basic authentication mechanisms that can be used

40、to validate the identity of both users and resources. These mechanisms provide building blocks for other security services such as authorization and data security that need to know the identity of parties engaged in an operation.l Process creation. This component is used to initiate computation on a

41、 resource once it has been located and allocated. This task includes setting up executables, creating an execution environment, starting an executable, passing arguments, integrating the new process into the rest of the computation, and managing termination and process shutdown.l Data access. This c

42、omponent is responsible for providing high-speed remote access to persistent storage such as files. Some data resources such as databases may be accessed via distributed database technology or the Common Object Request Broker Architecture (CORBA). The Globus data access module addresses the problem

43、of chieving high erformance when accessing parallel _le systems and network-enabled I/O devices such as the High Performance Storage System (HPSS).Together, the various Globus toolkit modules can be thought of as defining a metacomputing virtual machine. The definition of this virtual machine simpli

44、fies application development and enhances portability by allowing programmers to think of geographically distributed, heterogeneous collections of resources as unified entities.The Global Grid Forum (GGF)The Global Grid Forum (GGF) is a group formed by individuals from within the community of resear

45、chers and practitioners engaged in research, development, deployment, and support activities related to high-capability distributed software systems, or “grids.” The scope of the applications that motivate these activities is quite broad, including for example high performance processing application

46、s, distributed collaborative environments, distributed data analysis, and remote instrument control. A defining characteristic is a perceived need for services beyond those provided by todays commodity Internet.The GGF working groups are investigating a range of research topics related to distribute

47、d systems, best practices for the design and interoperation of distributed systems, and the development of recommendations regarding the implementation of grid software. Some GGF working groups have evolved to function as sets of related sub-groups, each addressing a particular topic within the scop

48、e of the working group. Other GGF working groups have operated with a wider scope in terms of topic, surveying a broad range of related topics and focusing on long-term research issues. This has resulted in a different set of objectives, appropriate expectations, and operating styles across the vari

49、ous GGF working groups. Working groups are defined into three categories: Areas, Working groups, and Research groups.Working GroupA working group generally has a finite lifespan and is focused on a particular, specific problem, technology, or opportunity for which they will deliver a document or ser

50、ies of documents, after which they may disband or create a revised charter for further work. The completion of a working group charter and subsequent disbanding of the group is viewed as a sign of success. Expected lifespan would be anywhere from 4-24 months, but may in some cases be longer or short

51、er. In general, working groups and working group process are described in "IETF Working Group Guidelines and Procedures", RFC 2418, September 1998. Any and all types of GGF documents may come from GGF working groups. However, technical specifications and recommendations track documents wil

52、l generally be the focus of working groups.Research GroupA research group has an indefinite, longer-term lifespan, attempting to address a set of problems, technologies, or opportunities where a long term approach is appropriate or in some cases where it may be premature to develop technical specifi

53、cations or recommendations track documents. In general, research groups and research group process are described in “IRTF Research Group Guidelines and Procedures”, RFC 2014,October 1996. Technical specifications and recommendations track documents will not generally come from GGF research groups, h

54、owever informational, experimental, or community practice documents are expected to come from GGF research groups. It is expected, but not required, that research groups will either “spawn” a working group or will provide input to existing working groups where specifications or recommendations track

55、 documents are needed.AreaAn area is a collection of related research groups and working groups. Areas exist for management purposes and to provide a structure for interactions among related groups. Thus the set of areas at any given point in time is driven by existing or timely/desired group topics rather than an overall, top-down architecture statement