Date: Mon, 10 Jun 1996 18:18:00 -0400 Description of first .objects parameter for GLOBEC methods (10 Jun 96) (Describes methods generated from defgb v 2.4 and later) The first parameter (parameter 0) to a GLOBEC method will be a file specification, optionally preceded by an at-sign, or optionally enclosed in parentheses. If neither at sign nor parentheses are present, the parameter is the name of the level 0 data file. If the at-sign is present, the parameter is assumed to be the name of a file containing other file specifications, including that of the level 0 data file. This file is called the indirect input file. If enclosed in parentheses, the file is a command whose output will serve as the level 0 data file. In this case, the file specification may be followed by parameter strings. Piping and/or i/o redirection in the command is not allowed. --- NOTE: By default, any input line which begins with ERR_PREFIX (currently *** Error) is considered an error message by defgb. The line will be passed to the error processing routines (see documentation of diagnostics optional file for details about controlling error messages) and defgb will exit with an error status. ERR_PREFIX can be changed (or eliminated) by specifying it during compilation. See defgb_switches.doc --- The indirect input file The format of this file begins with optional comment lines, formatted the same way as comment lines in the level 0 data file for this method. The data in the file consist of lines holding one file specification apiece, optionally preceded by a file descriptor. The files named in the indirect input file are called optional files. An ordinary file specification is the name of a file containing information about a particular option of the method. The file specification may be preceded by a backslash character. If so, it indicates that what follows is not actually a file name. Instead, it is one data record formatted for that particular optional file. (Purpose of this "escape mechanism" is to allow small amounts of information to reside in the indirect file itself without requiring another file). The file specification may also be a keyword of the form .objects_parameter_N, to indicate that the file specification will come from parameter N of the .objects file that includes the definition of the object being accessed. N begins with 1, signifying the first parameter after the name of the indirect input file. The _N is optional. If not present, N is determined from the file descriptor or the position of the line within the indirect file (see below). This type of file specification may be useful if, for example, it's desirable to associate the same indirect input file with many different level 0 data files. The file descriptor, if present, is separated from the file it describes by spaces, tabs, and/or equal signs. The file descriptor tells the method what kind of file is being named. The descriptors are: datafile, transvar, inpwidths, removals, gmtparams, diagnostics, and latlonparams. If no descriptor is present, the file is assumed to be the type implied by its position within the indirect input file. The first file specified in the indirect input file is, if undescribed, the datafile. The second is the transvar file. The third is the inpwidths file. The fourth is the removals file. The fifth is the gmtparams file. The sixth is the diagnostics file. The seventh is the latlonparams file. The eighth is the datafieldopts file. It is possible to mix described and undescribed files. It is not possible to specify an undescribed file without specifying all preceding files. If the file specification is the keyword .objects_parameter, the "number" of the optional file is used to tell which parameter in the .objects file corresponds to the optional file. The datafile will be expected from parameter 1 (first parameter after the indirect input file), the transvar file from parameter 2, etc. The order may be changed by specifying the parameter number after the keyword (see above). The datafile specifies the level 0 data file. It may be enclosed in parentheses to indicate that the level 0 data will come from a script or executable program. The transvar file specifies the file containing variable name translations. The inpwidths file specifies the field widths of fixed-field-width input variables. The removals file contains the names of input variables that are to be logically removed from the dataset. The gmtparams file contains the names of input variables that are to be combined to produce a gmt time, and the name of the variable to contain that gmt time. The diagnostics file contains information about what output to produce (besides the data), and the file to which the output goes. See below for the details of contents and formatting of each optional file. The latlonparams file contains information allowing the calculation/reformatting of latitude and/or longitude data. The datafieldopts file contains information controlling the treatment of individual data fields relating to formatting, leading/trailing blanks, etc. File specifications must each include any necessary directory specifications. That is, there is no assumption made about the relationship between the directories holding the various files (eg, that they all come from the directory of the first file named). Optional files may contain information involving variables (eg, datafile, inpwidths and removals files). The variable names in these files may be either the names before transvar translation or after. A file descriptor may contain the prefix "pre-translation-" or the prefix "post-translation-" to indicate which mode is in effect. If neither prefix is supplied, "post-translation-" is assumed (except for datafiles, where "pre-translation-" is assumed). It is an error if "pre-translation-" is supplied but no transvar file is supplied in the indirect file, or if the prefixes are supplied for optional files that do not contain variable names, or if the prefixes are applied to the transvar descriptor. It may be that a method must or must not have a particular optional file. Inner can be compiled with switches to indicate that a particular file is not, in fact, optional. See documentation for details. The level 0 data file is not optional. Sample file: # Test indirect file for met data. # datafile = not required if data file on first line # removals & gmtparams data short enough to be included in here via \ # variable names specified to gmtparams require translation. HR variable # is not in transvar.dat, and therefore is the same pre- and post- # translation, so it doesn't care about the *-translation prefix. # WJS Mar 96 /export/home/rpayne/emet259.dat transvar=/export/home/rpayne/transvar.dat pre-translation-gmtparams= \ hrname=HR; mnname=MN; gmtname=MN removals = \ HR inpwidths = /export/home/rpayne/widths.dat latlonparams = \ inlonname=lon; inlonconvention=W --- The optional input files The format of the optional files may differ on a per-file basis. However there is a common format, used by several, that is described here. Files in this common format have two types of records; comment and data. Comment records that occur before the first data record are inserted in the data comment stream. Comment records that occur after the first data record are ignored. Comment records are specified the same way as comments for the level 0 data file. Data records consist of one or more pairs of strings. Pairs are separated by semicolons. Within a pair, the strings are separated by blanks, tabs, and/or equal signs. The first string of a pair must be unique among the first strings of all pairs. ------- datafile The format of the data file differs from method to method. The format for the default method, def, is described elsewhere. The datafile file descriptor accepts the optional pre-translation- and post-translation- prefixes. The datafile is required to be present. ------- transvar The presence of this file indicates that any variable names in the dataset and also in the translation file are to be changed by the method to the corresponding names in the translation file. There is no required correspondence between the variable names in the dataset and the variable names in the translation file. For example, you may have a single translation file for all the variables found in all your datasets. On the other hand, you may translate only a few of the names in a particular dataset. It is simplest if all the names, both before and after translation, are unique. It is not permitted for a single variable to be translated to more than one synonym. All the other combinations are allowed, but must be used carefully. For example, one dataset might have the variable LONG and another the variable name LONGITUDE. If you want the name of this variable to appear as lon, you might well build one translation file, and in it translate both LONG and LONGITUDE to lon. This will work unless the translation file is used on a dataset that has both the LONG and LONGITUDE variables in it. In this case, both will translate to lon, and the 2nd occurrence will be flagged as a "Duplicate variable name" error. Similarly, it is possible but tricky to translate A to X and X to Y in the same translation file. The transvar optional file is in common format. Its descriptor, however, does NOT accept the pre-translation or post-translation prefix. The first element of a pair is the "old" name, found in the datasets. The second element is the "new" synonym name to which the method is to translate the old name. Sample file: # defmet translation list from RCG via Email 29 Dec 95. WJS 23 Mar 96 # 25 Mar 96. Play with HR, MN translations to get gmt stuff to work. WJS YD = yrday_gmt LAT = lat LONG = lon SMG = speed_trim EDO_Z = depth_w # Will use defgb CREATE_GMT capability to make time_gmt variable # from HR & MN. To store it over MN, will translate MN into it. # Will use defgb variable removal capability to logically remove HR, # so it doesn't need translation MN = time_gmt ------- inpwidths The presence of this file indicates that some of the input data is to be read from fixed-width fields instead of via free-field input. Variables found in this file are read from fixed-width fields. Variables not in the file are read with free-field input. Variables at a particular level must be either all free-field or all fixed-field. It is not possible to specify the starting column for a field. The starting column is determined by the position of the variable within a data record, and the sum of the widths of the preceding variables (if any). The order of the variables is determined by the variable list for the level being read. Accordingly, the sum of the field widths of the variables present at a level must equal the width of the data records of that level (before any next-level file specification) The inpwidths optional file is in common format. Its descriptor accepts the optional pre-translation- and post-translation- prefixes. The first element of a pair is a variable name. The second element is the field width of that variable in the dataset. Sample file: # latitude & longitude recorded to 3 decimal places # Must read data in fixed field since latitude and longitude might # not be separated if longitude goes negative lat = 7 lon = 8 temp_air = 5 ------- removals The default method allows data selection by providing a list of variables to process. The presence of a removals optional file allows data selection by providing a list of variables to omit. If a variable is present in both a selection and omission list, the omission list takes priority. The removals optional file has the same comment record rules as a common format file. Each data record, however, consists of a list of variables separated by blanks, tabs, commas and/or semicolons. The removals descriptor accepts the optional pre-translation- and post-translation- prefixes. Removed variables are available to calculations performed by inner (such as gmt calculation). They are not available to outer. Sample file: # Remove HR variable, since its info will be in GMT variable HR ------- gmtparams The presence of this file indicates that GMT is to be calculated from variables in the dataset containing hour and minute information. The hour data is in 24-hour time format and the minute data is an integer from 0 to 59. The output GMT format is a 4 digit string HHMM. To perform this calculation, inner needs to know the locations of the hour and minute variables, and it needs to know where to store the result, Unfortunately, inner cannot easily create variables, so the result must be stored over some other data, destroying it. If you don't want to replace existing data, room for the gmt variable must be made in the dataset by adding a dummy name at the level where you want the gmt to appear (and, therefore, also adding the dummy name to the level 0 list). If you DO replace existing data, you may well wish to change the variable name of that data to reflect the fact that it will be gmt data instead of what it originally was. This is done via the transvar file. The gmt data may overlay its source hour or minute variables. The gmtparams optional file is in common format. Its descriptor accepts the optional pre-translation- and post-translation- prefixes. Its data consists of 3 pairs. The first element of each pair is a keyword and the second element of each pair is a variable name. The keywords are: hrname, mnname, and gmtname. hrname is the name of an existing variable containing the 2 digit hour. mnname is the name of an existing variable containing the 2 digit minute. gmtname is the name of the existing variable to be replaced by the calculated gmt data. Sample file: # Take hour information from variable HR and minute information from # variable MN. Store the calculated gmt time in the location where the # MN data came from, making the original MN data unavailable to users of # the method. # Note that the resulting gmt data will be labelled MN, unless MN is # translated to some other name (like time_gmt) in the transvar file hrname=HR; mnname=MN; gmtname=MN ------- diagnostics A method produces output of various types. At a minimum, it generates a stream of comments, a list of variables, and data from the data file described above. For a description of this output, see (?). If an error is detected, a message is sent to the standard output device, stdout. If the method is compiled with certain switches, debugging information is generated and written to the standard error device, stderr. When being run interactively, this output can be directed with standard shell output redirection techniques. When being run in an http environment, stdout output appears on a browser screen, while stderr output typically appears in the httpd server log file. A GLOBEC inner can produce other output. For example, there can be information related to the indirect and optional files, as well as comments contained in these files. The diagnostics file allows control of output other than the variable list and data. It is possible to control both quantity and destination of output. It is possible to produce output before the diagnostics file has been read. In this case, output quantity and destination may be determined by the compile-time values. If these differ from the run-time values, output can be split between the 2 places. Output may be "turned off" either by defining "quantities" to "none" or "destinations" to "nowhere". Therefore, it is not considered an error if output is requested but there is no destination provided. The diagnostics optional file is in common format. Its descriptor does not accept the optional pre-translation- and post-translation- prefixes because it does not contain variable names. All of the parameters have default values, determined at compile time. Some may be specified on the compilation line. See the "switches" documentation. Its data pairs consist of 5 types. In the first type, the first element is the name of an output sink and the second element is a file specification. The output sinks are: error_sink, data_comment_sink, other_comment_sink, and debug_sink. error_sink is the destination for inner's error messages. data_comment_sink is the destination for comment information from data files. other_comment_sink is the destination for all other comments. debug_sink is the destination for debugging information for inner's debug output. Each sink accepts the optional prefix addl-. Without this prefix, the appropriate output stream is redirected to the specified file. With this prefix, the stream is sent to stdout (or stderr for trace output) and also to the specified file. If addl- is specified, the usual stdout stream may not be redirected. Each sink or addl-sink also accepts the optional prefixes new- or unique-. By default, output sinks are opened in "append" mode (create new file if one doesn't exist; add to end if file exists). Consecutive uses of the data object will create a log-like file, in which it may be difficult to correlate data with individual uses of the sink. Simultaneous uses of the data object will result in intermixed entries in the sink. The new- prefix will cause a new file to be opened at each use. Information from a previous use of the object will be lost. Simultaneous uses of the data object will produce unknown results (to me, anyway). The unique- prefix will cause a time stamp of the form ".yymmmdd-hhmm-ss" to be appended to the sink name. The same stamp will be appended to any unique- sink in a single use of the data object. Since this name is only unique to the second, simultaneous use is still possible. unique- sinks are opened in "append" mode. Multiple sinks directed to the same file must all have the same new- or unique- prefixes (or no prefixes must be present). File specifications may be any file that may be written to, including /dev/stdout, /dev/stderr, and /dev/null. They must include any relevant directory information. The length of each file specification is approximately 40 characters (there is space for 40 character names for each sink; space unused by one name may be used by another). If 2 sinks are to be sent to the same file, the file must be specified the same way in each pair. For example, if you use an absolute directory specifier with one sink, do not use a relative directory specifier with the other. Errors opening the error sink are reported to stdout, along with the message(s) that were to be sent to the error sink. In the second type, the first element is the name of an output level and the second element is an integer specifying the level. The output level names are error_level and trace_level. error_level controls how much information is included with each error message. trace_level controls how much subroutine call information is sent to the debug sink. The addl- prefix can be used with the output level names in conjunction with the addl- prefix used with the output sinks. When used, it allows the level sent to the default sink differ from the level sent to the duplicate sink. The levels' values must be between 0 and 99, inclusive. Level 0 produces the least output; level 99 the most. Level n information includes all the information produced at lower levels. error_level = 0 produces no error messages 10 produces the error message 20 identifies the inner writing the error 25 identifies the remote connection (if possible) and the time 50 may produce information about the file being processed at the time of the error 60 if the file is a command, attempts to copy more information from from the process in which the command is running 80 produces information about all files open at the time of the error trace_level = 0 produces no trace information 10 traces calls to ioopen 20 traces calls to the optional file routines 40 traces calls to the file-level routines (scanheader*) 50 traces calls to the comment-processing routines 70 traces calls to the record-level routines (ioreadrec) In the third type, the first element is the name of a source of diagnostic lines and the second element is the number of lines from that source that are to be inserted into the error stream. The only name currently implemented is maxscriptdiags. A command being used as a data source might fail and issue diagnostics referring to that failure. max- scriptdiags refers to the maximum number of acceptable lines from such a source. These diagnostics are sent to the error sink. maxscriptdiags accepts a addl- prefix (see error_level, above). In the fourth type, the first element is the name of an output switch and the second element is the value of that switch. The output switches are: inserted_msgs, inserted_msg_tags, and comment_source_tag. The inserted_msgs switch controls whether defgb generates comment lines identifying the indirect and optional files in use. The inserted_msg_tag switch controls whether the inserted messages themselves are tagged with identification as to where they came from. The comment_source_tag switch controls whether each optional file comment is tagged with identification as to where it came from. Switch values are TRUE or FALSE. Synonyms for TRUE are true, T, t, YES, yes, Y, y and 1. Synonyms for FALSE are false, F, f, NO, no, N, n, and 0. In the fifth type, the first element is the name of a data trace type and the second element is an integer specifying the frequency of the data trace. The data trace types are print_iovalstr, print_iovalreal, and print_*_lines, where * represents one of the optional file descriptors (including "datafile"). The "diagnostics" descriptor may not be specified, however, since by the time it is read, information from the diagnostics file has already been processed. A frequency of 0 means no output. Any other frequency means print the first occurrence of the data trace type, and every 'frequency'th occurrence thereafter. For example, print_datafile_lines=10 means output lines 1, 11, 21, etc from each input data file. iovalstr=1 means output the result of every iovalstr call. The addl- prefix can be used with the data trace types in conjunction with the addl- prefix used with the output sinks. When used, it allows the amount of trace information sent to the default sink to differ from the amount sent to the duplicate sink. All data trace output is written to the debug sink. ------- latlonparams The presence of this file indicates that latitude and/or longitude are to be calculated or reformatted from variables in the dataset. The latlongparams optional file is in common format. Its descriptor accepts the optional pre-translation- and post-translation- prefixes. Its data consists of up to 12 pairs. The first element of each pair is a keyword. 6 pairs refer to latitude and 6 to longitude. Within the 6, 3 refer to input and 3 to output. The keywords are: inlatname, inlatconvention, inlatformat, outlatname, outlatconvention, outlatname, inlonname, inlonconvention, inlonformat, outlonname, outlonconvention, outlonformat. The second element of name keywords is a variable name, specifying which variable name(s) contain the input data and are to contain the output data. The variable names for input & output may be the same or may differ. Output variable names must be in the level 0 variable list on the same or deeper levels than input variable names. See gmtparams for more discussion of variable name rules. If an input variable name is specified, its output variable name defaults to the same variable. The second element of convention keywords is a character specifying which direction is considered positive. latitude keywords accept N or S, and default to N. longitude keywords accept E or W and default to E. If input and output conventions differ, defgb will place a negated copy of the input variable in the output variable. (Note: only the first character of the second element is considered; it is considered case-insensitively) The second element of format keywords is a string identifying the format of the corresponding latitude or longitude information. The allowable input formats are degdecmin and decdeg, representing data in whole degrees/ decimal minutes and decimal degrees formats, respectively. The only allowable output format is decdeg. The default is decdeg. If input and output formats differ, defgb will reformat the input into the output. If neither convention nor format differ between input and output, defgb will do nothing, even if the input and output variable names differ. Sample file: # Transform W longitude found in variable lon to E longitude, also in variable # lon. The longitude format is assumed to be decimal degrees. (The # negation will take place regardless of the actual format, however) inlonname = lon; inlonconvention = W ------- datafieldopts The datafieldopts optional file is in common format. Its descriptor does not accept the optional pre-translation- and post-translation- prefixes because it does not contain variable names. All of the parameters have default values, determined at compile time. Some may be specified on the compilation line. See the "switches" documentation. In its data pairs, the first element is the name of a switch and the second element is the value of that switch. The switches are significant_embedded_separators_in_fixed_field and significant_consecutive_ separators_in_free_field. The significant_embedded_separators_in_fixed_field switch controls whether a fixed field containing a separator between 2 non-separator strings should be considered 2 fields (hence an error), or 1 field. The significant_consecutive_separators_in_free_field controls whether 2 consecutive data field separators indicate a null data field, or whether they should be considered the same as one separator. Switch values are TRUE or FALSE. See the diagnostics optional file for acceptable synonyms.