WP4: HPC Research GRIDWorkplan Overview  


WP5: Technology Development

Overview of WP5

Workpackage 5 of EuroGrid was designed to build on the funcionality of UNICORE and to extend it to provide the middleware necessary for the other workpackages. The spread of applications in Workpackages 1 to 4 is such that the middleware developed in Workpackage 5 is applicable to many applications in a computational Grid. Five areas of technology have been identified where support was improved in order to make EUROGRID suitable for use in the domain-specific GRIDs (WP 1 to WP4), and to improve the prospects of EUROGRID take-up and exploitation in science and industry. They are described below as tasks T5.1 to T5.5. There was clearly overlap with other European GRID projects and informal contacts were developed to provide a robust solution for European Grids.

T5.1: Efficient data transfer

Since EuroGrid technology is designed to be used over Internet routes, the issue of fast and secure data transfer is of great importance. Several considerations arise.
  • Optimisation of transfer bandwidth and cost. There may be more than one route for a point-to-point file transfer. There may be trade-offs between guarentees of bandwidth and cost. For example, ATM PVC routes provide guarenteed bandwidth and very low latency but are chargable. A decision may be made between an Internet route offering variable bandwidth or a dedicated link.
  • Fail-safe and encrypted transfer. If a network link goes down in a transfer, the files should not be corrupted without warning. It is also possible to investigate switching the transfer to alternative routes. Security of data over open networks is an important concern, especially for commercial use of the Grid.
  • Overlap of transfer and processing. We investigate this in terms of real-time processing. This form of file transfer needs to be "application aware". An example of a series of experiments investigating the use of such techniques over intercontinental networks can be found in the paper by Pickles et al. which was chosen the Best Paper of HPCN2000.
In general, application requirements on data transfer fall in two categories: burst transfers, where latency is critical and bulk transfer where the available bandwidth should be utilised. Both forms of transfer are important in a Computational Grid and both will be investigated in this workpackage.

T5.2: Resource broker

A resource broker in a computatational grid, where there are a variety of machines with specialist architectures, needs to be able to access and make decisions based on both static and dynamic information. By static information we mean resources such as the type of architecture, range of compilers and software, data storage facilities, that remain constant over relatively long periods of time. By dynamic information we mean the loading of machines, versions of compilers and software, current scheduling and charging strategies. These can change rapidly, in some cases continuously (e.g. loading of the machine). The resource broker needs this information to match a user's computational requirements, traditionally represented as a "job", to what is available on the GRID. Also the broker needs to be able to negotiate to balance the clients needs for low cost and fast turnaround, for example. The broker has to read and update information about the clients allowances and accounts on the machines across the GRID. For example, if the brokering process results in a job being submitted to Site A and Resources A, B, C are used in the job (say A is CPU time, B is disk space use, and C is archival space required for the results), then the broker needs to inform the client of cost and allow the site to have some means of recouping the cost. This could be via an account kept at the site, say a client has X amount of resource tokens representing Y euros worth of resource, then the account needs to be decremented by the amount used. Or the site might not hold a permanent account for the user but would bill him/her for the job. Thus a resource broker for a computatational Grid is a sophisticated software concept. This workpackage will be developed on the basis of a resource economy which has been running successfully at the CSAR WWW service for two years. The knowledge and account updating tools of this service provide a starting point for developing the full brokering functionality. More details are at CSAR WWW site

T5.3 ASP Infrastructure

The cost of a job submitted on the GRID may include the cost of the software licence necessary for the work. This particularly applies to commercial usage where the cost of the licence for powerful software may dominate the cost for the job. It may be that an organisation needs to use software on large machines but could not justify purchasing a machine and a software licence for its exclusive use. There is an opportunity for Application Service Provider (ASP) to make available such software on a pay-per-use basis. It can clearly be seen how such a model fits with the concept of the resource broker in task T5.2. The combination of middleware provided through T5.2 and T5.3 provides a basis for commercial and industrial use of a computational GRID, either as a company-wide Intranet or via independent ASPs over the Internet.

T5.4 Application coupling

Many large scale computational applications involve interacting sub-problems which may be best located on different machines. A good class of examples are systems where fluids interact with solid structures and differenent solvers are optimal for the fluid dynamics and structural mechanics. Another example are environmental simualtions which may involve solving equations for gaseous, fluid and solid behaviour along with chemical and perhaps biological reactions. In computational GRIDS with a range of specialist architectures, different submodels may be placed on machines where the solvers run particularly efficiently, this is know as "machine affinity". Task 5.4 will investigate integrating a communication middleware for such coupled applications. There needs to be some "weak" sychronicity for such sub-model coupling. Thus we will investigat interfaces to schedulers for co-scheduling of machines and other resources (archives and staging areas for file storage and transfer for example).

T5.5 Interactive access

Some computational work requires interactive access to resources. An example would be the running of a simulation with visualization of the results being provided as the simulation runs. This can be extended to situations where the simulation is steered according to the visualized output. This approach is becoming increasingly important to a range of industrial, scientific and civil users of computational facilities. Rapid virtual prototyping of new products needs design and engineering teams to be able to observe the behaviour of the proposed product and visual information is very important to this process. Civil emergency procedures, training simulations and environmental impact studies also rely on the interaction of skilled personnel with a simulation based on observed or hypothetical data. Most large scale computational facilities have traditionally operated their machines in batch mode and the current model of UNICORE is based on this approach. Workpackage T5.5 will add functionality to allow for the possibility of interactive use of computational and visualization facilities.

URL: <>