DATA AND INFORMATION MANAGEMENT OVERVIEW

The COOA Program will establish an effective data management to successfully conduct the proposed research. To assist in this process, the project data management group will make available the basic common services, logistical support, and data management required by the COOA investigators as they conduct their research, analyze the data from individual efforts and integrate results, conduct modeling studies, and integrate all of these with education and outreach activities. The COOA data management server will be located in Woods Hole and will utilize the expertise developed and used by the U.S. GLOBEC Program. In particular, we will use the distributed information based on the Joint Global Ocean Flux Study (JGOFS) data management and data server software (Groman and Wiebe, 1998). The system is a distributed, object-based data system, with the data sets residing with the responsible scientists when possible. Core data from the COOA activities will be accessible to all the scientific investigators in a timely manner. The data include measurements of physical, chemical, and biological parameters from discrete samples and continuous probe measurements from sites that are generating them. Data that are not readily placed in standard digital format, such as the results of physiological experiments, will be indexed with information about their location and status for use by the investigators.

The data management system proposed here has the capability to automate data acquisition and subsequent data serving from realtime data acquisition systems. In this scheme, data files are routinely and automatically copied from the realtime computer systems at time intervals as little as an hour. The data in these files are automatically processed as necessary, and in many cases, averaged, before making them available on the server, and accessible by any standard browser, such as Internet Explorer or Netscape. Data quality control will be the responsibility of the contributing investigator(s), although initial quality control checks will be done prior to placing any data on-line.

We believe that it is important that data sets be made available to other researchers in the project as soon as they are useful to them. Because data can be remotely stored at the data contributor's site, we are assured that the most accurate and up to date data are being served over the Web. If it is necessary to restrict access to certain data sets for a certain length of time, we can require that a username and password be entered before data can be accessed, although we hope to avoid this requirement.

We will use an existing computer system to act as the central data directory: All requests for data are made through this central "data server", even though the data files themselves can be, and often are, stored on other computer systems. Before data will be served, the necessary metadata file(s) will be created to ensure that these data can be readily used by other researchers. In addition, we prepare standard metadata, Directory Interchange Format (DIF), records and submit them to the Global Change Master Directory (an FGDC clearinghouse) to ensure the widest possible distribution of our data sets.

Retrieval and display capabilities of the data management system are illustrated schematically in Figure 1. Open access to the data is accomplished via any standard browser. Web-based forms provide data selection and data projection capabilities, common to SQL type queries. The data projection capability permits people interested in the data to view, plot, and download data that is contained within specified geographic boundaries and satisfies any other retrieval conditions (such as air temperature less than 10 C, for instance). The data projection capability permits people to select for viewing and downloading only those parameters they want without having to deal with other data stored in the same files. The web-based system also supports data joins between two data sets that share a common data attribute, such as station number. We encourage our data contributors to follow our data field name thesaurus so that field names are used consistently and without ambiguity.

Communication among the investigators is an important factor in maximizing interdisciplinary synthesis. To ensure the rapid and extensive dissemination of data and results, following each scientific event/cruise, the scientist in charge (assisted by others in the scientific party) will prepare an initial assessment of the activities and acquired data (a report) so that the investigators will have a common record of position data, and general information and environmental data based on shipboard, satellite imagery, and other remote sensing observations.

Data from previous work (e.g. retrospective studies) will be brought on-line and as the modeling results and survey results develop, they will also be added to the data system. The reports, meeting minutes, and other forms of information will be made available on-line to foster information exchange and further analysis.

In summary, our approach to data management and data retrieval is flexible, extensible, and has been proven through use by existing large interdisciplinary projects (i.e., GLOBEC, JGOFS). Our personnel have extensive knowledge and experience in enabling web sharing of user contributed data, and it is our intention to thoroughly integrate into the COOA program's data access and data sharing approach.

Specifics of the data system

The JGOFS software uses an object-oriented approach by tying a method (software program) and the actual data together. While similar data types can share the same (or similar) software, the entity that the system deals with is the data object, which includes the necessary software to access and manipulate the data. Furthermore, since the system uses standard http protocol to communicate between multiple servers and multiple clients, we are able to take advantage of the implicit and explicit tools and techniques available for displaying information via a Web browser. For example, when a user requests to see the available data, a table is generated in real time, based on whatever data are currently available. There is no static version of a data directory; rather we use the Web's ability to run a common gateway interface (CGI) script to generate the directory list on demand. The directory listing is created with built in links so that the user can click on a data object (to look at the data) or click to another data category to see other available data objects.

Once a data object is chosen (by clicking on the object name) the actual data available are displayed, typically organized in a hierarchical fashion, with the slowest varying variables listed first. The data are formatted for display on the browser each time a request is made so that even if the data are changed, the display is always up to date. At any point during this process, the user can request an X-Y plot of the data using the "Plotting and other options" button at the top of the page. The options button offers several ways of displaying the data, again, all available by clicking on the appropriate choice. A user can chose to plot any parameter available within the object. The list of available choices is provided and is always based on the object being viewed. The resulting basic plot is useful to determine if further analysis is warranted. The user can then decide to download these data to his/her own machine in either a flat file (ASCII) or binary MATLAB format.

As the amount of data collected increases and as these data sets are added to the on-line data management system, there is increasing need for more and better Web-based display and analysis tools. Users can currently download data of interest (both their own and any other data set maintained in the system) to their local computers for detailed analysis, synthesis, and display. There is increased need for reviewing data to see if they are relevant or of potential interest for further study. We plan to enhance the retrieval, analysis, and display tools available through the Web to aid users in deciding what data are available, where data are located, and what the data fields look like, via two-dimensional and three-dimensional images.

The system is flexible and extensible, able to handle ASCII tables, binary data, and even images and movies. This is due to the inherent object based approach and the reliance on standard Web protocols. The data's method (computer program) is specific to the data. These programs are typically written in C, but any computer language is supported. These programs accommodate how the data are formatted (binary, ASCII, etc.). The programming task is simplified by requiring that only the read routines be specific to the data. These read routines are used by standard, pre-existing data manipulation and reformatting code, which takes care of converting the data into a form that the rest of the software can use.

Image and movie data are accommodated by creating "data files" of URL links to these data sets and serving these data files as if they were "regular" ASCII data. This allows objects composed of images and movies to be treated the same as any other data objects. The only requirement is that the user's browser supports the image format (e.g. gif, tiff, jpeg, etc.) or movie format (e.g. mov, quicktime, fli/flc, etc.) that is used in the object. Further power and flexibility is gained because of the ability of the remote data servers to reformat data as necessary. For example, we serve satellite images as gif images to the browser. However, these images are stored in an alternate format and may also be compressed. When a user requests to see one of these images, the original image is uncompressed (if necessary) and then reformatted as a gif image before being sent to the user's browser. This happens automatically, without the user being aware of these steps. By serving a contributor's original data, from their own system, we avoid the necessity of duplicating data and, more importantly, insure that the data viewed by others is always the most current version.

Reference:

Groman, R.C., and P.H. Wiebe. 1998. Data management in the U.S. GLOBEC Georges Bank Program. Ocean Community Conference 98 Proceedings, Marine Technology Society, Baltimore MD, pp. 807-812.

Figure 1: Retrieval and display capabilities of the data management system.