Earth Observation Applications Approach to Data and Metadata Deployment on the European DataGrid Testbed

The European DataGrid (EDG) experience represents an important challenge for the current state-of-the-art in developing and deploying extended, large scale, Grid testbeds. The Earth Observation (EO) community has taken the EDG opportunity to examine the correspondence between EO application needs and actual and potential functionalities offered by Grids today. The application chosen for deployment is a typical one in the EO domain. It consists of elaborating low-level (e.g. raw) satellite data into higher-level products and validating them against similar products obtained from instruments using other platforms, such as groundbased instruments, balloon and other airborne observations, observations from ships, etc. The selected EO Application usecase involves processing and validating global atmospheric ozone observations made by the GOME instrument flying on board the European ERS satellite, throughout a 7-year mission. This requires processing and maintaining a large set of data files distributed on the Grid and the timely location and retrieval of small subsets of selected data files for the validation. This requires the creation and deployment of metadata catalogues that can be updated and accessed by partners with different roles in the collaboration, while providing secure access control and restricted accesses to both data and catalogues. The EO application partners involved are the European Space Agency (Frascati, Italy), the Netherlands Royal Institute of Meteorology (KNMI, de Bilt, Holland), and Institute Pierre Simon Laplace (IPSL, Paris France). The partners formed a virtual organization of more than fifteen scientists and engineers who collaborated on setting up the EO Grid infrastructure and deploying the application. The partners' main objectives were to (1) demonstrate how Grid can respond to the complexity and the constraints imposed by applications in EO domain and (2) identify the benefits of the technology and how it can improve the work of EO technical and scientific users. This paper aims to point out how these objectives are actually or potentially fulfilled by EDG middleware and services. An overview of the DataGrid middleware and infrastructure is followed by a description of the application. The problem of metadata handling and the use of different metadata catalogue services is then addressed and the three different solutions experimented with to finally deploy the application on the Grid are described.