Placing Data on the JGOFS System

How to place data on your own JGOFS Server

Introduction

The JGOFS Data system was designed with the flexibility to serve data from the Investigator's own workstation. The U.S. GLOBEC Georges Bank Program encourages our scientific investigators to serve their data from their own computers, ensuring that the most up to date data are made available to others in the project.

Determine if your format can be read by the system

After data is ready to be shared with other Investigators in the program, check its format against the samples provided by the DMO. If the format of your data does not exactly match one of the existing sample formats, a decision needs to be made to either change the data format to conform to the most similar sample format or to request a modification to one of the formats. If the data does match one of the formats, note the name of the "method" or translation program that can read that format. Now you know most of the pieces required by our system: the method name, the name(s) of the files to be served, and where they will reside on your system.

Install a JGOFS server

The next step is to download the system software from the Distribution site. Follow the instructions offered with the software for downloading the JGOFS software as a tar file. We suggest that a new user be created on your system with the username, 'jgofs'. This user can be the one who installs and, therefore, owns the software. The user 'jgofs' should have its own group id and a user id that differs from any other user on your system. Install the JGOFS Data System according to the directions online. If you have problems while installing the software, contact the DMO for assistance.

Placing data on the server

Once you have a working server installed on your workstation, you are ready to tell it where your data are located. As user 'jgofs', make certain that you can read the data files [suggestion: 'more' one or more of the files]. This will ascertain that the system (run as jgofs) will have permission to read and serve the data.

There are two files that tell the JGOFS system where to find data and how to read it:

.remoteobjects
.objects

The .remoteobjects file contains a three line entry for every object available through the JGOFS data system in a given category. The three lines entered in the .remoteobjects file should be inserted alphabetically by the Principal Investigator's last name and will appear in this order on the system. The information given by the 3 line entry is:
1. Line 1:
  - the name of the data object to display, limited to 18 characters
  - the machine where the data is located, even if that machine is the local machine
  - the location of the .objects file on the serving machine
  - the name of the data object entry in the .objects file.
2. Line 2: the name of the Principal Investigator responsible for collecting the data
3. Line 3: a brief description of the contents of the data object
The structure of the 3-line entries in .remoteobjects is, where ^ indicates a required blank space:

display_object=//machine.domain/category-subdirectory-path/objname
-^P.Investigator
-^Brief description of the data

An example of an entry is:

wind_stress=//jgof.wh.whoi.edu/globec/gb/process/wind_stress
- James Manning
- Wind Stress
The .objects file contains a one (1) line entry for each object that it serves. The object name in the .objects file must exactly match the object name at the end of the URL-like entry beginning with //machine in the corresponding .remoteobjects file.
This one line entry provides the following information:
- the exact directory and filename(s) of the data on the local system
- which method, or translation program, to use to access it
- optionally, limits the parameters displayed by the system
Structure of .objects is as follows:
objname=method(/exact-absolute-path-on-local-system/filename)

An example of an entry is:
wind_stress=def(/data/manning/wind_stress.dat)

Optionally, one may restrict the parameters to be served. For example, an extended syntax allows:
objname=method(/path/file[,optional_list_parameters])

The existence of the item [,optional_list_parameters] indicated above, identifies the parameters which will be served by the system and may restrict which of these will be served, also. For example, a line such as
ctd=def(/data/nabe/a119.ctd,sta,press,sal,par,station<150)

displays only station, pressure, salinity, and PAR parameters, excluding any others which may reside in the data file a119.ctd. A further restriction to the parameters is made by specifying only stations numbered less than 150 e.g. station<150.
Contact the DMO for correct syntax for your object, if you are unsure.

These two files are located in the objects subdirectory tree of the JGOFS Data System. For example, if you chose to install the system at /home/machinename/globec, your data objects information will be located starting at /home/machinename/globec/objects. Below this level are the directories which are displayed by the system automatically.

The main U.S. GLOBEC Georges Bank server has as its top level, a /gb/ category, below which are the five categories, broadscale, modeling, mooring, process, and satellite. We further divide these categories by year. So, for example, CTD data from a 1996 process cruise would be /globec/gb/process/1996/ctd.

Assume that you are serving data from Georges Bank, 1997 process cruises. You would create a subdirectory structure like this:

/home/machinename/globec/objects/gb/process/1997

indicating that you want to display /gb/process/1997 as your category of data. The .remoteobjects and .objects files containing the information about the process data from 1997 would be located in this directory.

Documentation

Documents detailing protocols of scientific methodology related to sample or data collection, need to be available for proper utilization of the data. We ask that these documents be submitted to the DMO before you begin to serve your data.

Notes specific to a particular dataset, perhaps indicating anomalies at the time of collection or during the processing of data can be served with the information, documenting the parameter names used in the data. We do require that the parameter names be defined and that the units be clearly stated. The name of the object (from the .remoteobjects file) is used to provide the filename for another file that is also part of the system.

The objectname.info file is located in the same directory as the .remoteobjects and .objects files. It provides information related to data collection as well as the object's parameter names, their descriptions and units. Also included are the ship name, cruise identifier, the Principal Investigator(s)'s full name and links to any additional notes. Place four spaces between the longest parameter name and its description and keep lines to an 80 character maximum length to ensure the list will be visually aesthetic. A template for the objectname.info file is provided.

Spot-check the data

All the pieces required to display data on the system are now in place. Using a browser, visually inspect the data to insure proper alignment, etc. Use the "List all at next level" followed by the "Flat listing" features of the system and refer to the original file to insure that every record of the dataset is displayed.

Let the world know

The data is ready to be incorporated into the category collection. In other words, the data that you have prepared and are now serving needs to be made available to the other Investigators as part of the Process Study data set. To do this, send the .remoteobjects file to the Data Management Office to be added to its larger file of entries. The DMO will update the List of Available Objects.

Congratulations!

Note: If the above instructions were not clear, click here for another version of these instructions.

Last modified: August 29, 2005
Original text: C. Hammond