White Paper: Changing How Science is Pursued -- Paradigm Shifts From Optical Networking

Howard J. Cohen, Ph.D. (howard@cohensw.com)
Work performed at
Nortel Networks, Santa Clara, CA
January 2004

Abstract

The emergence of dynamic optical networks will begin to enable the scientific community to realize the full potential of the Grid in data intensive applications, where terabytes and even petabytes of data must be moved to their points of processing. New capabilities in the software and hardware that control the networks allow them to be treated as first class, schedulable resources, along with computation, storage, and instrumentation. In this paper we will discuss how these emergent properties of the network can change some of the paradigms of how science is done in the early 21st century, and use this discussion as motivation for the following brief introduction of an architecture, including OGSI-compliant middleware services, that allows the full potential of high speed optical networks to be realized in the service of eScience. The paper concludes with a summary of some of our achievements to date and plans for future development.

Introduction

The practice of science has seen a number of paradigm shifts in the 20th century, including the growth of large disbursed teams and the use of simulations and computational science as a third branch, complementing theory and laboratory experiment. The recent exponential growth in network capacity, brought about by the blossoming of optical networking, will provide another such shift as the 21st century begins.

The prospect of "unlimited bandwidth" propels, for example, the vision of the OptIPuter [1]:
The OptIPuter exploits a new world in which the central architectural element is optical networking, not computers - creating "supernetworks". This paradigm shift requires large-scale applications-driven, system experiments and a broad multidisciplinary team to understand and develop innovative solutions for a "LambdaGrid" world. The goal of this new architecture is to enable scientists who are generating terabytes and petabytes of data to interactively visualize, analyze, and correlate their data from multiple storage sites connected to optical networks.

In this White Paper we will present a few scenarios describing some of the possible consequences of this shift. We then follow with a brief, high level description of the DWDM-RAM project, one of the few prototype realizations of this underlying architecture which marries dynamic optical networking with the scientific Grid infrastructure in service of data intensive applications. We then summarize our discussion with a request for collaborators within the scientific community.

Science Examples

Much of science is experimental, with data being gathered using increasingly sophisticated and expensive instruments, which may range from electron microscopes to distributed arrays of seismometers to astronomical interferometric arrays of radio telescopes to high energy physics particle colliders and their detectors to space based remote sensing satellites to nuclear fusion facilities, and so on. Heretofore, a scientist wishing to use such an experimental facility would need to go to the facility, directly control the apparatus, gather his or her data, return to his home institution for the analysis phase, decide on the focus for follow on experiments or observations, and repeat the cycle, all thewhile competing for an allocation for the use of the scarce resources of machine time.

If network connectivity were "free", the experimental control could be done from the comfort of her home institution, obviating one requirement for travel and the personal and financial burden that often imposes. In addition, the data could be gathered in real time from the facility to the home institution where the emerging Grid paradigm of fast and distributed computation could facilitate early preliminary analysis andvisualization, thus guiding modifications to the experimental setup or parameters and greatly shortening the cycle time and increasing the efficiency with which the facility was used. This not only makes life easier for the individual user but frees more time for additional users, spreading the costs across a broader base.

Remote work could make one-of-a-kind facilities or instruments available to a far wider community of scientists than could afford to travel, for example, scientists from third world countries without the budgets to visit first world centers.

Non-experimental work could also benefit from very high capacity networking. Consider interlinked models used for climate simulation. There might be an atmospheric model that interacts with an oceanic model as well as with a solar model (radiation flux and solar storms affect the upper atmosphere) and even econometric models (how will climate affect land use patterns, agriculture, etc. and how might that feed back into atmospheric effects). Each simulation would be run at its own center of expertise, but all would need high speed data connections to communicate at each time step.

Experiment, theory and computation generate massive amounts of data. Astronomy, earth science, climatology, genomics and proteomics, medical imaging, and their kin can have data sets in the multi-terabyte range, and high energy particle physics and biology will soon be generating petabyte data sets (1 PB = 10^15 bytes = 1000 TB = 10^6 GB). Not only is there a need to move much of this data around from source to analysis  points, but also to understand it. And one of the primary means to gain appreciation and integration of data sets is through visualization, with the trend being to higher and higher resolution displays and real-time interaction with multi-dimensional data. These displays may be driven by arrays of special purpose rendering computers working in parallel and sending pixels to a set of display devices also working in concert to provide a large scale unified environment, such as a CAVE or wall sized display.

What are the data rates needed for this, even within a single laboratory? Consider a 100 Megapixel display, which will soon be available as a  prototype device.[2] Each pixel requires 3 bytes of data, one each for red, blue and green. So one frame involves
10^8 * 3 * 8 = 2.4 x 10^9 bits 

To fill the display with one frame on a fast 100 Mbps network would take
	2.4 * 10^9 / 10^8 = 24 seconds
which is a rather long latency.

If we wanted to drive video at 30 fames/sec, [NB, IBM's deepview page notes that their refresh rate is 41 Hz] we would need a bandwidth of
	2.4 x 10^9 * 30 = 72 * 10^9 bps = 72 Gbps

This is clearly beyond the range of today's standard 1 Gigabit ethernets but is achievable using the technologies we will describe in the next section. And these technologies can also allow the user to view all or parts of this display from a remote site, given the local display hardware. That is, the engines that analyze the data and create the visualization can be remote.

DWDM-RAM: Marrying Optical Networking to the Grid in the Service of Data Intensive Applications

  Architecture
  
 The emergence of very high capacity optical networking, where a single optical fiber can transport a number of distinct 10 Gbps optical channels (lambdas), and multiple lambdas can be ganged together for even greater capacity, has changed the communications paradigm from one of a narrow bottleneck to that of a firehose. In theory, the problem is no longer the inability of the network to move massive data quantities in a timely manner but rather how to take advantage of this capacity and "fill the pipe".
 
Still, an optical connection between two nodes is one that can have a single user at a time, and so the concept of network scheduling comes into play. A scheduled, dedicated, circuit switched connection provides a guaranteed Quality of Service and avoids many of the issues of packet switched connections, such as collisions and recovery, so that the full bandwidth is available to the user. In practice, optical connections have been statically provisioned and dedicated to a single user. The lead time for getting such a connection is often several months, the cost quite high, and the efficiency of use low. On the other hand, a shared switchable dynamic optical network can be instantaneously available to a user who would get (and pay for) only the capacity needed for the time needed.
 
The DWDM-RAM project has designed an architecture to enable this, one that marries the Grid to a dynamic schedulable high capacity optical network in support of data intensive applications, and has created prototype software to demonstrate some of these features and capabilities. The architecture, described in more detail in several papers submitted to conferences and journals [3, 4, 5], provides two layers between the users and their applications and the underlying dynamical optical networks. The Application Middleware layer speaks for the requirements of the users and presents a uniform interface to the network, shielding the applications from all aspects of sharing and managing the required resources.
 
The Resource Middleware layer provides OGSI-compliant services that satisfy the resource requirements of the application, as specified or interpreted by the Application Middleware layer services. This layer contains interfaces and services that initiate and control sharing of the underlying resources, including scheduling and reservation services. 

  Achievements
   
Using the OMNInet, a metro-area all optical dynamic switched and allocated testbed with multiple 10 Gbps channels between hosts, we developed early Java-based software to implement the services described above. We also developed client applications and some additional middleware web services to demonstrate the transfer of large files and data sets over the network, in an on-demand scenario, controlled through the middleware. These were demonstrated at GGF9 in Chicago in October 2003 and at SuperComputing 2003 in Phoenix in November 2003.
 
In addition, an early implementation of a network scheduler and optimizer, based on the use of under-constrained requests, was demonstrated at SuperComputing 2003.
 
The basic infrastructure of our architecture has had its initial implementation and we are continuing to gain experience in its behavior and utility, as well as an understanding of many of the areas in which it could profitably gain from extensions. 

  Plans and Possibilities

 We would like to continue fleshing out this architecture and implementation in the context of real-world data intensive Grid applications. Of the many possible areas to pursue in its ongoing development, just a few will be mentioned here.
 
We see the addition of other underlying file transfer protocols as one area to explore. More complex scheduling algorithms are another area, including the ability to query the network for its topology and the characteristics of its constituent segments and nodes, to be able to route over the topology and to do segment-level scheduling, allocation and deallocation.
 
More complex is developing cooperative protocols for interacting with other Grid resources (such as replica location services and local storage management services) and schedulers, both providing services to them and using them to provide inputs into the schedules and connectivity we provide.
 
But most of all, we believe that the greatest learning will be achieved by working with a user community with pressing needs, real networks, and large amounts of data, in order to be sure our system solves the right problems in ways that are immediately useful and transparent to the user community. To that end, we ask our potential users how they could use the Grid  middleware services presented here, services which promote the network to a high capacity, reliable, schedulable entity, and how we could work together.

Summary

A brief discussion of some scientific data intensive applications was followed by an introduction to the DWDM-RAM architecture, which provides middleware service layers enabling the potential of dynamic high speed optical networks to be realized as a first class service within the Grid context. It was pointed out how this can fundamentally change the paradigm of the network in the Grid context from a bottleneck to a plentiful enabling resource.
 
This was followed by a summary of some of the project results, early proof-of-concept demonstrations, and a short list of possible and interesting directions in which to continue our research and development.

Acknowledgements

OMNInet and ODIN have been supported by NSF.
 
DWDM-RAM has been supported by DARPA.

References

[1] From http://www.optiputer.net. See also "The OptIPuter", Larry L. Smarr, Andrew A. Chien., Tom DeFanti, Jason Leigh, Philip M. Papadapoulos, CACM vol 46, no. 1, pp 58-66 (Nov 2003)

[2] http://www.research.ibm.com/deepview descibes their current 9 Megapixel hardware.

[3] "DWDM-RAM: Enabling Grid Services with Dynamic Optical Networks", S. Figueira, S. Naiksatam, H. Cohen, D. Cutrell, D. Gutierrez, D. B. Hoang, T. Lavian, J. Mambretti, S. Merrill, F. Travostino, paper submitted to CCGrid 2004 (4th IEEE/ACM International Symposium on Cluster Computing and the Grid), Chicago, IL, 19-22 April 2004.
 
[4] "DWDM-RAM: An Architecture for Data Intensive Service Enabled by Next Generation Dynamic Optical Networks", D.B. Hoang, H. Cohen, D. Cutrell, S. Figueira, T. Lavian, J. Mambretti, I. Monga, S. Naiksatam, F. Travostino, paper submitted to IEEE Journal on Selected Areas in Communications, special issue on Intelligent Services and Applications in Next Generation Networks, to be published Q1 2005.
 
[5] "A Grid Network Service Architecture for Dynamic Optical Networks", Tal Lavian, Steve Merrill, Howard Cohen, Doan Hoang, Joe Mambretti, Silvia Figueira, Doug Cutrell, Sumit Naiksatam, Franco Travostino, paper submitted to the Journal of Grid Computing, special issue on High Performance Networking, to be published mid-2004.