White Paper: A New Middleware and Networking 
Architecture to Support HEP Data Intensive Grid Applications

Howard J. Cohen (howard@cohensw.com)
Work performed at
Nortel Networks, Santa Clara, CA
January 2004

Abstract

The emergence of dynamic optical networks will begin to enable the scientific community to realize the full potential of the Grid in data intensive applications, where terabytes and even petabytes of data must be moved to their points of processing. New capabilties in the software and hardware that control the networks allow them to be treated as first class, schedulable resources, along with computation, storage, and instrumentation. In this paper we will discuss some of the data intensive issues of the High Energy Physics (HEP) community, as an archetypical motivation for the following brief introduction of an architecture, including OGSI-compliant middleware services, that allows the full potential of high speed optical networks to be realized in the service of eScience. The paper concludes with a summary of some of our achievements to date and plans for future development.

Introduction

The scientific community has driven the emergence of Grid computing over the past seven or so years as a way of applying distributed computational and storage resources to large problems in data analysis, data mining, simulation, and ab initio calculations. The marshalling of shared resources across Virtual Organizations is based upon distributed middleware which provides the facilities for scheduling, coordination, authorization and authentication, among other required web services. Heretofore, however, the network has been assumed to be either a necessary evil and bottleneck, or conversely as a entity that is always available and need not be treated as a first class schedulable resource along with computation, storage and instrumentation.

The emergence of very high capacity optical networks has potentially removed the network-as-bottleneck, but this is only a potential until it can be fully integrated into the Grid middleware as a schedulable service. The DWDM-RAM project has created an architecture for doing so and has demonstrated early successes using the OMNInet all-optical metro-area testbed for the validation of its concepts.  In this White Paper we will first lay the groundwork by reviewing some of the  pressing data intensive applications and issues facing the HEP community. Here our discussion will focus on High Energy Physics, but other areas of interest are included. We will next briefly describe the DWDM-RAM architecture and how it specifically addresses these issues, with a review of some of its achievements to date and a broad discussion of possible future directions. Finally, we will summarize the salient points we raised in this paper.

DoE and Grid Applications

The United States Department of Energy (DoE) and the HEP community in general support a vast portfolio of applied and basic research in the physical sciences, as well as ancillary research and development activities that support the primary scientific endeavors. The planning, execution and analysis of these research programs require the coordination of large, distributed teams of individuals and institutions. The archetypical enterprise is the High Energy Physics (HEP) program, although other DoE sponsored areas, such as those listed in Table 1, present many of the same issues.

	Estimated 2008 Data Generation [1]	
High energy physics		
   CEBAF (hadron structure experiments)	Several PB/year	
   RHIC (quark-gluon plasma experiments)	5 PB/year	
   CERN LHC (Higgs boson search)	10 PB/year	
   SLAC (BaBar experiments)	1 PB/year (2003)	
Nuclear physics		
Plasma physics		
Magnetic fusion	1 PB/year	
Computational fluid dynamics		
Climate studies	>10 PB/year	
Bioinformatics (genomics, proteomics, metabolomics)	>20 PB/year	
Astrophysics		
  Sloan Digital Sky Survey	15 TB [2]	
Chemistry	4 PB/year	
Materials Science (neutrons and photons)	0.35 PB/year	
Nanoscience		
Macro-molecular crystallography		
Visualization		

Table 1: DoE sponsored data intensive research areas (theoretical and experimental)

Consider the Compact Muon Solenoid (CMS) experiment being built as part of the Large Hadron Collider (LHC) at CERN. This project involves 31 nations, 150 institutions, and over 1800 scientists. The Atlas detector at the LHC, expected to come on line with the CMS in 2007, should generate 10 petabytes (1 PB = 1015 bytes) of data in its first year of operation, and grow to about 1 exabyte per year (1 EB = 1000 PB = 1018 bytes) by 2015.

This massive flow of raw data must be buffered at the experimental facility but shipped to a hierarchy of centers for analysis. The model is to have a number of distributed regional centers in the U.S, Europe and Asia for intermediate storage and some processing. These, in turn, feed large tier 2 centers, each of which feeds data to a large number of institutes, and these do local processing as well as send data to the workstations and compute farms of individual researchers. The same event data sets may be analyzed by many different teams. The results of the initial analyses may also be large derived data files which need to be transported to central locations for statistical and other analyses. In addition to this experimental raw run data and the large volumes of derived data are simulation data, estimated to run a factor of three greater in volume than the former sets. Simulations are crucial for understanding the nature of the detectors and modelling the underlying physics, to enable a comparison of what is seen with what is expected to be seen. Because these experiments will be searching for unknown physics buried at a few events of interest a year, old data will be reanalyzed a number of times as the algorithms improve and an understanding of the emerging physics materializes.

All this implies the need to move and manage enormous quantities of data, as well as the management and coordination of storage and computational resources. Since each LHC detector event is independent of all other events, the analyses can be performed in a highly parallel and distributed fashion, which fits in perfectly with the emerging concept and reality of Grid computing. The Grid is a way of marshalling and sharing resources for distributed processing, using divide-and-conquer to tackle computational problems too large for any one machine or even institution to deal with alone. The Grid provides a way of coordinating the use of these distributed and heterogeneous resources across multiple organizations by providing a uniform set of middleware and web services to allow authentication, authorization, accounting, resource discovery and scheduling, remote job execution, and feedback to the users.

Heretofore, the network has been a bottleneck in this scenario. Using a "fast" 100 Mbps network, it would take almost a day to move a 1 TB data set (or 2 1/2 years to move a Petabyte), assuming that the full bandwidth were available for this period. But the packet switched IP network is shared, has its own overhead, and cannot be scheduled. Advances in optical networking, however, are beginning to address this problem and will be discussed in the next section of this paper.

DWDM-RAM: Marrying Optical Networking to the Grid in the Service of Data Intensive Applications

  Architecture

The emergence of very high capacity optical networking, where a single optical fiber can transport a number of distinct 10 Gbps optical channels (lambdas), and multiple lambdas can be ganged together for even greater capacity, has changed the communications paradigm from one of a narrow bottleneck to that of a firehose. In theory, the problem is no longer the inability of the network to move massive data quantities in a timely manner but rather how to take advantage of this capacity and "fill the pipe".

Still, an optical connection between two nodes is one that can have a single user at a time, and so the concept of network scheduling comes into play. A scheduled, dedicated, circuit switched connection provides a guaranteed Quality of Service and avoids many of the issues of packet switched connections, such as collisions and recovery, so that the full bandwidth is available to the user. In practice, optical connections have been statically provisioned and dedicated to a single user. The lead time for getting such a connection is often several months, the cost quite high, and the efficiency of use low. On the other hand, a shared switchable dynamic optical network can be instantaneously available to a user who would get (and pay for) only the capacity needed for the time needed.

The DWDM-RAM project has designed an architecture to enable this, one that marries the Grid to a dynamic schedulable high capacity optical network in support of data intensive applications, and has created prototype software to demonstrate some of these features and capabilities. The architecture, described in more detail in several papers submitted to conferences and journals [3, 4, 5], provides two layers between the users and their applications and the underlying dynamical optical networks. The Application Middleware layer speaks for the requirements of the users and presents a uniform interface to the network, shielding the applications from all aspects of sharing and managing the required resources.

The Resource Middleware layer provides OGSI-compliant services that satisfy the resource requirements of the application, as specified or interpreted by the Application Middleware layer services. This layer contains interfaces and services that initiate and control sharing of the underlying resources, including scheduling and reservation services.

  Achievements

Using the OMNInet, a metro-area all optical dynamic switched and allocated testbed with multiple 10 Gbps channels between hosts, and its control software, ODIN, we developed early Java-based software to implement the services described above. We also developed client applications and some additional middleware web services to demonstrate the transfer of large files and data sets over the network, in an on-demand scenario, controlled through the middleware. These were demonstrated at GGF9 in Chicago in October 2003 and at SuperComputing 2003 in Phoenix in November 2003.

In addition, an early implementation of a network scheduler and optimizer, based on the use of under-constrained requests, was demonstrated at SuperComputing 2003.

The basic infrastructure of our architecture has had its initial implementation and we are continuing to gain experience in its behavior and utility, as well as an understanding of many of the areas in which it could profitably gain from extensions.

  Plans and Possibilities

We would like to continue fleshing out this architecture and implementation in the context of real-world data intensive Grid applications. Of the many possible areas to pursue in its ongoing development, just a few will be mentioned here.

We see the addition of other underlying file transfer protocols as one area to explore. More complex scheduling algorithms are another area, including the ability to query the network for its topology and the characteristics of its constituent segments and nodes, to be able to route over the topology and to do segment-level scheduling, allocation and deallocation.

More complex is developing cooperative protocols for interacting with other Grid resources (such as replica location services and local storage management services) and schedulers, both providing services to them and using them to provide inputs into the schedules and connectivity we provide.

But most of all, we believe that the greatest learning will be achieved by working with a user community with pressing needs, real networks, and large amounts of data, in order to be sure our system solves the right problems in ways that are immediately useful and transparent to the user community. To that end, we ask our potential users how they could use the Grid middleware services presented here, services which promote the network to a high capacity, reliable, schedulable entity, and how we could work together.

Summary

A brief discussion of DoE's data intensive applications, with an emphasis on High Energy Physics in the Grid environment, was followed by an introduction to the DWDM-RAM architecture, which provides middleware service layers enabling the potential of dynamic high speed optical networks to be realized as a first class service within the Grid context. It was pointed out how this can fundamentally change the paradigm of the network in the Grid context from a bottleneck to a plentiful enabling resource.

This was followed by a summary of some of the project results, early proof-of-concept demonstrations, and a short list of possible and interesting directions in which to continue our research and development.

Acknowledgements

OMNInet research is conducted under partnership between Nortel Networks, SBC, and the International Center for Advanced Internet Research (iCAIR) at Northwestern University.

ODIN research and development is supported by NSF.

DWDM-RAM research is supported by DARPA.

References

[1] Data from Table 2-1, “DOE Science Networking Challenge: Roadmap to 2008”, Roy Whitney and Larry Price, eds. Report of the June 3-5, 2003 DOE Science Networking Workshop. Available at http://www.es.net/hypertext/welcome/pr/Roadmap%20to%202008.pdf

[2]  http://www.sdss.org/background/index.html/

[3] "DWDM-RAM: Enabling Grid Services with Dynamic Optical Networks", S. Figueira, S. Naiksatam, H. Cohen, D. Cutrell, D. Gutierrez, D. B. Hoang, T. Lavian, J. Mambretti, S. Merrill, F. Travostino, paper submitted to CCGrid 2004 (4th IEEE/ACM International Symposium on Cluster Computing and the Grid), Chicago, IL, 19-22 April 2004.

[4] "DWDM-RAM: An Architecture for Data Intensive Service Enabled by Next Generation Dynamic Optical Networks", D.B. Hoang, H. Cohen, D. Cutrell, S. Figueira, T. Lavian, J. Mambretti, I. Monga, S. Naiksatam, F. Travostino, paper submitted to IEEE Journal on Selected Areas in Communications, special issue on Intelligent Services and Applications in Next Generation Networks, to be published Q1 2005.

[5] "A Grid Network Service Architecture for Dynamic Optical Networks", Tal Lavian, Steve Merrill, Howard Cohen, Doan Hoang, Joe Mambretti, Silvia Figueira, Doug Cutrell, Sumit Naiksatam, Franco Travostino, paper submitted to the Journal of Grid Computing, special issue on High Performance Networking, to be published mid-2004.